Traffic Sign Recognition Device and Traffic Sign Recognition Method

Abstract

To improve accuracy of recognition of a traffic sign candidate detected by a camera.SOLUTION: A traffic sign recognition device includes a storage device that stores a camera image of a movable body and three-dimensional point group data, and a processor. The processor estimates, in a case where the camera image includes an image of a traffic sign candidate, a relative position of the traffic sign candidate to the movable body based on the camera image, specifies a set of three-dimensional point group data including a relative position to the movable body that is equal to that of the traffic sign candidate, out of the three-dimensional point group data, specifies, in the camera image, an image region of an object that corresponds to a set region indicative of a region where the set is specified and that includes the traffic sign candidate, calculates, based on a size of the image region of the object and color components constituting an image of the object, a percentage of a predetermined color component constituting a guide sign among the color components constituting the image of the object, and recognizes the object including the traffic sign candidate as the guide sign in a case where the percentage of the predetermined color component is equal to or greater than a threshold.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a system and method for recognizing traffic signs.

Background Art

[0002] Patent Document 1 discloses a drive recorder capable of alerting a driving vehicle to prevent reverse driving. In this technology, it is determined whether the traveling direction of the vehicle is the reverse direction of the direction in which it should travel, and if it is determined to be the reverse direction, an alarm is given to the driver.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, among traffic signs, there are traffic signs displayed on road signs or traffic signs displayed on guiding signs. Furthermore, traffic signs include traffic regulation signs (such as vehicle entry prohibition, temporary stop, etc.), traffic indication signs (such as direction indication, etc.). In the case of a traffic regulation sign indicating vehicle entry prohibition displayed on a "road sign", when it is determined that this is a traffic regulation sign based on the recognition of a traffic sign candidate detected by a camera, it can be said that the traffic sign candidate is correctly recognized.

[0005] However, in the case of a traffic regulation sign indicating vehicle entry prohibition displayed on a "guiding sign", when it is determined that this is a traffic regulation sign based on the recognition of a traffic sign candidate detected by a camera, it cannot always be said that the recognition of the traffic sign candidate is correct.

[0006] One object of the present disclosure is to provide a technique capable of improving the accuracy of recognition of traffic sign candidates detected by a camera. **Means for Solving the Problems**

[0007] A first aspect relates to a traffic sign recognition device including a camera image of a moving body, three-dimensional point cloud data, Information on the specifications of a camera, including information on the resolution of the camera and the mounting height of the camera, and information on the specified size of a traffic sign candidate a storage device in which the three-dimensional point cloud data is stored, and a processor. When an image of a traffic sign candidate is included in the camera image, the processor estimates the relative position of the traffic sign candidate with respect to the moving body based on the camera image. The processor identifies a set of three-dimensional point cloud data in which the relative position with respect to the moving body is equal to that of the traffic sign candidate among the three-dimensional point cloud data. The processor identifies, within the camera image, an image area of an object corresponding to the set area indicating the area where the set is identified and including the traffic sign candidate. The processor calculates, based on the size of the image area of the object and the color components constituting the image of the object, the ratio occupied by a predetermined color component constituting a guide sign among the color components constituting the image of the object. When the ratio occupied by the predetermined color component is equal to or greater than a threshold value, the processor recognizes that the object including the traffic sign candidate is a guide sign. The relative position includes the relative distance of the traffic sign candidate with respect to the moving object and the height of the traffic sign candidate obtained by adding the mounting height of the camera to the relative height of the traffic sign candidate with respect to the moving object. The information of the camera image includes the information of a plurality of camera images acquired by a plurality of cameras. In estimating the relative position, the processor estimates the relative distance by performing a calculation using a predetermined calculation formula with the resolution of the camera, the number of horizontal pixels constituting the image of the traffic sign candidate, and the specified size. In estimating the relative position, the processor estimates the relative height by performing a calculation using a predetermined calculation formula with the relative distance, the resolution of the camera, and the number of vertical pixels constituting the image of the traffic sign candidate.

[0008] A second aspect is Relates to a traffic sign recognition device including a storage device that stores a camera image of a moving object, three-dimensional point cloud data, information on the specifications of a camera including information on the resolution of the camera and the mounting height of the camera, and information on the specified size of a traffic sign candidate, and a processor. When the image of the traffic sign candidate is included in the camera image, the processor estimates the relative position of the traffic sign candidate with respect to the moving object based on the camera image. The processor identifies a set of three-dimensional point cloud data in which the relative position with respect to the moving object is equal to that of the traffic sign candidate among the three-dimensional point cloud data. The processor identifies, within the camera image, an image area of an object corresponding to the set area indicating the area where the set is identified and including the traffic sign candidate. Based on the size of the image area of the object and the color components constituting the image of the object, the processor calculates the ratio occupied by a predetermined color component constituting a guide sign among the color components constituting the image of the object. When the ratio occupied by the predetermined color component is equal to or greater than a threshold value, the processor recognizes that the object including the traffic sign candidate is a guide sign. The relative position includes the relative distance of the traffic sign candidate with respect to the moving body and the height of the traffic sign candidate obtained by adding the mounting height of the camera to the relative height of the traffic sign candidate with respect to the moving body. The camera image includes a plurality of camera images acquired by a plurality of cameras with different arrangements. In estimating the relative position, when the image of the traffic sign candidate is included in a plurality of camera images, the processor estimates the relative distance from the image of the traffic sign candidate based on the principle of triangulation. In estimating the relative position, the processor estimates the relative height by performing a calculation using a predetermined calculation formula with the relative distance, the resolution of the camera, and the number of vertical pixels constituting the image of the traffic sign candidate.

[0009] In addition to the first or second aspect, the third aspect further has the following features. The set area has a horizontal width and a vertical height calculated based on a set that satisfies a predetermined condition, and at least the horizontal width is equal to or greater than a predetermined size.

[0010] The fourth aspect is Third In addition to the aspect of, it further has the following features. The predetermined condition includes that the distance between three-dimensional point cloud data included in the set is less than a predetermined distance, and the point cloud density indicating the degree of density of the three-dimensional point cloud data included in the set is equal to or greater than a predetermined density.

[0011] The fifth aspect is Any one of the first to fourth viewpoints In addition to the aspect of, it further has the following features. The color components constituting the image of the object are represented by an integrated value obtained by integrating the pixel values of red pixels, an integrated value obtained by integrating the pixel values of green pixels, and an integrated value obtained by integrating the pixel values of blue pixels. The predetermined color component includes at least one of an integrated value obtained by integrating the pixel values of green pixels and an integrated value obtained by integrating the pixel values of blue pixels.

[0012] In addition to any one of the first to fifth aspects, the sixth aspect further has the following features. The traffic sign candidate includes at least a traffic regulation sign candidate.

[0013] The seventh aspect is Related to a traffic sign recognition method. The traffic sign recognition method is a step in which a processor acquires a camera image of a moving body, three-dimensional point cloud data, information on camera specifications including information on the resolution of the camera and information on the mounting height of the camera, and information on the specified size of the traffic sign candidate; a step in which, when an image of a traffic sign candidate is included in the camera image, the processor estimates the relative position of the traffic sign candidate with respect to the moving body based on the camera image; a step in which the processor identifies a set of three-dimensional point cloud data in which the relative position with respect to the moving body is equal to that of the traffic sign candidate among the three-dimensional point cloud data; a step in which the processor identifies, within the camera image, an image area of an object that corresponds to the set area where the set is identified and includes the traffic sign candidate; a step in which the processor calculates the ratio occupied by a predetermined color component constituting a guide sign among the color components constituting the image of the object based on the size of the image area of the object and the color components constituting the image of the object; a step in which the processor recognizes that the object including the traffic sign candidate is a guide sign when the ratio occupied by the predetermined color component is equal to or greater than a threshold value; including The relative position includes the relative distance of the traffic sign candidate with respect to the moving body and the height of the traffic sign candidate obtained by adding the mounting height of the camera to the relative height of the traffic sign candidate with respect to the moving body. The information of the camera image includes the information of a plurality of camera images acquired by a plurality of cameras. The step of estimating the relative position is A step of estimating a relative distance by using the resolution of a camera, the number of horizontal pixels constituting an image of a traffic sign candidate, and a specified size, and performing a calculation by a process using a predetermined calculation formula; A step of estimating a relative height by using the relative distance, the resolution of the camera, and the number of vertical pixels constituting the image of the traffic sign candidate, and performing a calculation by a processor using a predetermined calculation formula; including.

[0014] The eighth aspect is related to a traffic sign recognition method. The traffic sign recognition method includes a step of acquiring a camera image of a moving object and three-dimensional point cloud data and information on the specifications of the camera including information on the resolution of the camera and information on the mounting height of the camera, the processor and, when an image of a traffic sign candidate is included in the camera image, estimating a relative position of the traffic sign candidate with respect to the moving object based on the camera image , the processor and, identifying a set of three-dimensional point cloud data in which the relative position with respect to the moving object is equal to that of the traffic sign candidate among the three-dimensional point cloud data , the processor and, identifying, within the camera image, an image area of an object that corresponds to the identified set area and includes the traffic sign candidate , the processor and, calculating a ratio occupied by a predetermined color component constituting a guide sign among the color components constituting the object image based on the size of the object image area and the color components constituting the object image , the processor and, when the ratio occupied by the predetermined color component is equal to or greater than a threshold value, recognizing that the object including the traffic sign candidate is a guide sign , the processor and including . , the relative position includes the relative distance of the traffic sign candidate with respect to the moving body, and the height of the traffic sign candidate obtained by adding the mounting height of the camera to the relative height of the traffic sign candidate with respect to the moving body. The camera image includes a plurality of camera images acquired by a plurality of cameras with different arrangements. The step of estimating the relative position is When the image of the traffic sign candidate is included in a plurality of camera images, a step of estimating the relative distance by the processor based on the principle of triangulation from the image of the traffic sign candidate; A step of estimating a relative height by using the relative distance, the resolution of the camera, and the number of vertical pixels constituting the image of the traffic sign candidate, and performing a calculation by a processor using a predetermined calculation formula; including.

Advantages of the Invention

[0015] According to the first aspect, it is possible to improve the accuracy of recognizing a traffic sign candidate detected by a camera. Furthermore, it becomes possible to estimate the relative position of the traffic sign candidate with respect to the moving body.

[0016] According to the second aspect, It becomes possible to estimate the relative position of the traffic sign candidate with respect to the moving body without using the information on the specified size of the traffic sign candidate.

[0017] According to the third aspect, It becomes possible to improve the specific accuracy of the aggregation area.

[0018] According to the fourth aspect, It becomes possible to further improve the specific accuracy of the aggregation area.

[0019] According to the fifth aspect, It becomes possible to recognize whether an object including the traffic sign candidate is a guiding sign.

[0020] According to the sixth aspect, It can also be applied when the traffic sign candidate is a traffic regulation sign candidate.

[0021] According to the seventh aspect, The same effect as the first viewpoint can be obtained.

[0022] According to the eighth aspect, the same effects as those of the 2 aspect can be obtained.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Modes for Carrying Out the Invention

[0024] With reference to the accompanying drawings, a traffic sign recognition device and a traffic sign recognition method according to an embodiment of the present disclosure will be described. Note that the traffic sign recognition method according to the embodiment is realized by computer processing of the traffic sign recognition device according to the embodiment.

[0025] Embodiment 1. Overview The traffic sign recognition device according to the present embodiment is mounted on a moving body and is for recognizing traffic sign candidates. FIG. 1 shows a configuration example of the traffic sign recognition device 10 according to the present embodiment. The traffic sign recognition device 10 performs various information processes. The traffic sign recognition device 10 includes one or more processors 100 (hereinafter simply referred to as the processor 100) and one or more storage devices 110 (hereinafter simply referred to as the storage device 110). The processor 100 executes various processes. For example, the processor 100 includes a CPU, an ECU, and the like. The storage device 110 stores data of information 120 of a camera image, data of information 130 of a three-dimensional point cloud, data of information 140 of camera specifications, and data of information 150 of a specified size of a traffic sign candidate. Examples of the storage device 110 include a volatile memory, a non-volatile memory, an HDD, an SSD, and the like. By the processor 100 executing a traffic sign recognition program that is a computer program, the functions of the information processing device 20 are realized. The traffic sign recognition program is stored in the storage device 110. The traffic sign recognition program may be recorded on a computer-readable storage medium. The traffic sign recognition program may be provided via a network.

[0026] The information 120 of the camera image includes the information of an image obtained by imaging a traffic sign candidate with a camera mounted on a moving body. The information 130 of the three-dimensional point cloud includes the information of a point cloud indicating three-dimensional coordinates (lateral coordinate, longitudinal coordinate, and depth coordinate) generated based on data of reflected light of laser pulses reflected around the moving body using a three-dimensional measurement sensor mounted on the moving body. Examples of the three-dimensional measurement sensor include LIDAR (Laser Imaging Detection and Ranging). The data of the information 140 of the camera specifications includes the resolution indicating the instantaneous field of view angle per unit pixel in the camera image and the information of the mounting height of the camera. The data of the information 150 of the specified size of the traffic sign candidate includes the information of the specified size of the traffic sign candidate displayed on the guide sign.

[0027] Examples of the traffic sign candidates included in the information 120 of the camera image include traffic sign candidates displayed on road signs, traffic sign candidates displayed on guide signs, and the like. Examples of the types of traffic sign candidates include traffic regulation signs (such as vehicle entry prohibited, stop), traffic indication signs (such as direction indication), and the like.

[0028] In the traffic sign recognition device 10 according to the present embodiment, the processor 100 executes a traffic sign recognition program to perform various processes for recognizing whether a traffic sign candidate detected based on the information 120 of the camera image is a guiding sign. Specifically, there are differences in characteristics between the traffic sign candidates displayed on the guiding signs and the traffic sign candidates displayed on the road signs. By performing various processes using these differences in characteristics, it can be expected to improve the accuracy of recognition of traffic sign candidates. Examples of differences in characteristics include differences in height (guiding signs are installed at higher positions compared to road signs), differences in the areas of objects including traffic sign candidates (guiding signs are areas composed of a plurality of objects including the object of the traffic sign candidate and other objects, while road signs are areas composed of single objects that only include the object of the traffic sign candidate), and differences in the ratios of the color components of the areas of the objects (the color components of guiding signs occupy a larger ratio of the areas of other objects that are larger than the object of the traffic sign candidate, while the color components of road signs occupy a larger ratio of the areas composed of the object of the traffic sign candidate).

[0029] Hereinafter, the traffic sign recognition device 10 according to the present embodiment will be described in more detail.

[0030] 2. Details of Information Processing The traffic sign recognition device 10 recognizes whether an object including a traffic sign candidate is a guiding sign based on the information 120 of the camera image, the information 130 of the three-dimensional point cloud, the information 140 of the camera specifications, and the information 150 of the specified size of the traffic sign candidate. The traffic sign recognition device 10 according to the present embodiment includes characteristic processes as described below.

[0031] FIG. 2 is a block diagram showing a functional example of the traffic sign recognition device 10 according to the present embodiment. The traffic sign recognition device 10 includes, as functional blocks, an information input unit 200, a traffic sign recognition processing unit 300, and a processing result output unit 400. These functional blocks are realized by the processor 100 executing a traffic sign recognition program.

[0032] The information input unit 200 performs a process of inputting the information 120 of the camera image, the information 130 of the three-dimensional point cloud, the information 140 of the camera specifications, and the information 150 of the specified size of the traffic sign candidate recorded in the storage device 110. Then, the input information 120 of the camera image, the information 130 of the three-dimensional point cloud, the information 140 of the camera specifications, and the information 150 of the specified size of the traffic sign candidate are output to the traffic sign recognition processing unit 300.

[0033] The traffic sign recognition processing unit 300 further includes a traffic sign candidate detection unit 310, a relative position estimation unit 320, a set region specifying unit 330, an image region size specifying unit 340, a color component ratio determination unit 350, and a traffic sign candidate recognition unit 360. The traffic sign recognition processing unit 300 recognizes whether an object including a traffic sign candidate is a guiding sign based on the input information 120 of the camera image, the information 130 of the three-dimensional point cloud, the information 140 of the camera specifications, and the information 150 of the specified size of the traffic sign candidate. The details of each process of the traffic sign candidate detection unit 310, the relative position estimation unit 320, the set region specifying unit 330, the image region size specifying unit 340, the color component ratio determination unit 350, and the traffic sign candidate recognition unit 360 will be described later.

[0034] The traffic sign candidate detection unit 310 performs a process of detecting traffic sign candidates from the camera image based on the input information 120 of the camera image. In this process, it is determined whether the detected traffic sign candidate is a traffic regulation sign, a traffic indication sign, or the like. An example of the processing result when a traffic sign candidate is detected is shown as in FIG. 3. FIG. 3 shows an example when the detected traffic sign candidate is a traffic regulation sign candidate indicating prohibition of vehicle entry. Examples of the method for detecting traffic signs include template matching, Deep Learning, etc.

[0035] The relative position estimation unit 320 estimates the relative position of the traffic sign candidate with respect to the moving object based on the traffic sign candidate which is the processing result of the traffic sign candidate detection unit 310, the information 140 of the input camera specifications, and the information 150 of the specified size of the traffic sign candidate. The relative position of the traffic sign candidate includes the information of the relative distance of the traffic sign candidate with respect to the moving object and the information of the height of the traffic sign candidate obtained by adding the mounting height of the camera included in the information 140 of the camera specifications to the relative height of the traffic sign candidate with respect to the moving object.

[0036] An example of a method for estimating the relative position of a traffic sign candidate is shown in FIG. 4. When the resolution of the camera included in the information 140 of the camera specifications is α, the number of horizontal pixels constituting the image of the traffic sign candidate is Xp, and the specified size of the traffic sign candidate included in the information 150 of the specified size of the traffic sign candidate is W, the relative distance of the traffic sign candidate with respect to the moving object is represented by the following formula (1).

[0037]

Equation

[0038] When the relative distance of the traffic sign candidate with respect to the moving object represented by the above formula (1) is Z, the resolution of the camera is α, and the number of vertical pixels constituting the image of the traffic sign candidate is Yp, the information of the relative height of the traffic sign candidate with respect to the moving object is represented by the following formula (2).

[0039]

Equation

[0040] When the mounting height of the camera is Hc, the information of the height of the traffic sign candidate is represented by the following formula (3).

[0041]

Equation

[0042] When estimating the relative distance of the traffic sign candidate shown above with respect to the moving body, information 150 on the specified size of the traffic sign candidate is used, and it is premised that the size of the traffic sign candidate has been grasped in advance. If the information 150 on the specified size of the traffic sign candidate is not used, a plurality of cameras may be mounted on the moving body, and the relative distance of the traffic sign candidate with respect to the moving body may be estimated based on the principle of triangulation from the images of the traffic sign candidates detected by the plurality of cameras.

[0043] Based on the relative position of the traffic sign candidate estimated by the relative position estimation unit 320 and the information 130 of the three-dimensional point cloud, the set region specifying unit 330 specifies a set region indicating the region of the set of three-dimensional point cloud data that satisfies the conditions. Details of the processing of the set region specifying unit 330 will be described later.

[0044] The image area size specifying unit 340 specifies the size of the image area of an object including a traffic sign candidate corresponding to the set area based on the set area estimated by the set area specifying unit 330. Here, the coordinates indicated by the three-dimensional point cloud data included in the set area are represented by the width, height, and distance with respect to the mounting position of the three-dimensional measurement sensor. On the other hand, the coordinates indicated by the pixels included in the image area are not represented by the width and height with respect to the mounting position of the camera, but are represented by the positions of the pixels in the X direction and the Y direction when the object is imaged by the imaging element. Since the representation of the coordinates is different between the three-dimensional point cloud data and the camera image, it is necessary to associate the coordinates indicated by the three-dimensional point cloud data with the coordinates indicated by the pixels of the camera image. As a method for performing the coordinate association, for example, the coordinate association is performed using an edge portion (such as the edge of a building) indicating the feature of the target object. Specifically, in the three-dimensional point cloud data, since the presence or absence of data can be determined with the edge portion as a boundary, the coordinates of the edge portion can be grasped. On the other hand, in the camera image, since the edge portion can be detected by image processing, the coordinates indicating the positions of the pixels in the X direction and the Y direction constituting the image of the edge portion can be grasped. Thereby, the association between the specified coordinates can be performed, and it becomes possible to specify the size of the image area of the object including the traffic sign candidate corresponding to the set area. However, if there is an object with a similar shape near the target object, it is assumed that the coordinate association will be performed between different objects. On the other hand, if the coordinate association is correctly performed, it can be said that the coordinate association only needs to be performed once as long as there is no change in the arrangement of the camera and the three-dimensional measurement sensor mounted on the moving body. Therefore, the coordinate association may be performed as part of the calibration performed after mounting the camera and the three-dimensional measurement sensor on the moving body.

[0045] The color component ratio determination unit 350 performs threshold determination of the ratio occupied by a predetermined color component constituting a guide sign among the color components constituting the object image, based on the size of the image area of the object specified by the image area size specification unit 340 and the color components constituting the image area of the object. Specifically, the image area is composed of a plurality of pixels, and the color components constituted by a unit pixel are indicated by the pixel value of a red-based pixel R, the pixel value of a green-based pixel G, and the pixel value of a blue-based pixel B. That is, the color components constituted by the plurality of pixels in the image area are indicated by the integrated value obtained by integrating the pixel values of the red-based pixels R of the plurality of pixels, the integrated value obtained by integrating the pixel values of the green-based pixels G of the plurality of pixels, and the integrated value obtained by integrating the pixel values of the blue-based pixels B of the plurality of pixels. The color components constituting the guide sign include a lot of green, blue, or both colors. For this reason, the ratio occupied by the integrated value obtained by integrating the pixel values of the green-based pixel G, which is a predetermined color component constituting the guide sign, and the integrated value obtained by integrating the pixel values of the blue-based pixel B among the color components constituting the object image is calculated, and threshold determination is performed as to whether or not the ratio occupied by at least one of the integrated value obtained by integrating the pixel values of the green-based pixel G and the integrated value obtained by integrating the pixel values of the blue-based pixel B is equal to or greater than the threshold value. A specific example of the threshold determination is shown as in FIG. 5. Note that, instead of the RGB color space, the HSV color space obtained by converting the RGB color space into an HSV (hue, saturation, value) color space may be used for the color components.

[0046] When, in the result of the color component ratio determination unit 350, it is determined that the ratio occupied by one of the integrated value obtained by integrating the pixel values of the green-based pixel G, which is a predetermined color component constituting the guide sign, or the integrated value obtained by integrating the pixel values of the blue-based pixel B is equal to or greater than the threshold value, the traffic sign recognition unit 360 recognizes that the object including the traffic sign candidate is a guide sign. When it is determined that the ratio is not equal to or greater than the threshold value, the traffic sign recognition unit 360 recognizes that the object including the traffic sign candidate is not a guide sign.

[0047] The processing result output unit 400 performs processing of outputting the recognition result as to whether or not the object including the traffic sign candidate in the traffic sign recognition processing unit 300 is a guide sign to a processing unit different from the traffic sign recognition processing unit 300.

[0048] FIG. 6 is a flowchart showing an example of processing of the traffic sign recognition unit 300 of the traffic sign recognition device 10 according to the present embodiment.

[0049] In step S100, the traffic sign recognition unit 300 detects traffic sign candidates based on the information 120 of the input camera image. Then, the process proceeds to step S110.

[0050] In step S110, the traffic sign recognition unit 300 estimates the relative position of the traffic sign candidates. Then, the process proceeds to step S110.

[0051] In step S120, the traffic sign recognition unit 300 specifies a set region indicating a region of a set of three-dimensional point cloud data corresponding to the relative position of the traffic sign candidates. Then, the process proceeds to step S130.

[0052] In step S130, the traffic sign recognition unit 300 specifies the size of the image region of the object corresponding to the set region. Then, the process proceeds to step S140.

[0053] In step S140, the traffic sign recognition unit 300 determines whether or not the ratio of a predetermined color component constituting the guide sign among the color components constituting the image of the object is equal to or greater than a threshold value based on the image region of the object.

[0054] If it is determined that the ratio of the predetermined color component constituting the guide sign is equal to or greater than the threshold value (step S140; Yes), the process proceeds to step S150. Otherwise (step S140; No), the process proceeds to step S160.

[0055] In step S150, the traffic sign recognition unit 300 recognizes that the object including the traffic sign candidate is a guide sign.

[0056] In step S160, the traffic sign recognition unit 300 recognizes that the object including the traffic sign candidate is not a guide sign.

[0057] In step S170, the traffic sign recognition processing unit 300 outputs a recognition result.

[0058] FIG. 7 is a block diagram showing a functional example of the set area specifying unit 330 of the traffic sign recognition processing unit 300 according to the present embodiment. In the above-described set area specifying unit 330, it has been described that a set area indicating an area of a set of three-dimensional point group data satisfying conditions is specified based on the relative position of the traffic sign candidate estimated by the relative position estimating unit 320 and the information 130 of the three-dimensional point group. Here, the details of the processing of the set area specifying unit 330 will be described. The specific processing of the set area specifying unit 330 includes a first set specifying unit 331, a second set specifying unit 332, and a set area determining unit 333. The details of each processing unit will be described later.

[0059] The first set specifying unit 331 performs a process of specifying a set of three-dimensional point group data (hereinafter referred to as the first set) that is equal to the relative position of the traffic sign candidate estimated by the relative position estimating unit 320 among the data of the information 130 of the three-dimensional point group. Specifically, the difference between the relative distance included in the relative position estimated by the relative position estimating unit 320 and the distance included in the three-dimensional point group data is within a first range, and the difference between the height included in the relative position and the height included in the three-dimensional point group data is within a second range. A set of three-dimensional point group data that satisfies the above conditions is specified as the first set. The first and second ranges are set in advance as ranges that can be considered to exist at the same position in consideration of, for example, the measurement error based on the camera image and the measurement error based on the three-dimensional point group data. An example of the result before and after obtaining the first set is shown as in FIG. 8.

[0060] The second set specifying unit 332 performs a process of specifying a set (hereinafter referred to as the second set) that satisfies a predetermined condition among the first sets specified by the first set specifying unit 331. The predetermined condition includes that the distance between the three-dimensional point group data is less than a predetermined distance and the point group density indicating the degree of density of the three-dimensional point group data is equal to or greater than a predetermined density.

[0061] Here, the relationship between the distance between three-dimensional point cloud data and the point cloud density of the three-dimensional point cloud data is shown in FIG. 9. FIG. 9 gives an example when considering the three-dimensional point cloud data that can be obtained according to the relative distance of the moving object of the traffic sign candidate. Specifically, it can be said that as the relative distance becomes longer, the available three-dimensional point cloud data decreases, and as the relative distance becomes shorter, the available three-dimensional point cloud data increases. Furthermore, when the available three-dimensional point cloud data is small, the distance between the three-dimensional point cloud data becomes long, and when the available three-dimensional point cloud data is large, the distance between the three-dimensional point cloud data becomes short. Based on this, when the relative distance is long, it is assumed that sufficient three-dimensional point cloud data cannot be secured to recognize the area of the object including the traffic sign candidate. For this reason, the accuracy of specifying the set area decreases. On the other hand, when the relative distance is short, since the acquired three-dimensional point cloud data increases, it becomes possible to secure sufficient three-dimensional point cloud data to recognize the area of the object including the traffic sign candidate, and the accuracy of specifying the set area increases. From this, the above-mentioned predetermined conditions include that the distance between the three-dimensional point cloud data is less than a predetermined distance.

[0062] In addition, even if the second set is specified under the predetermined condition that the distance between the three-dimensional point cloud data is less than a predetermined distance, it is assumed that unnecessary three-dimensional point cloud data exists at the same position as the relative distance. For example, as shown in FIG. 9, this is the case where three-dimensional point cloud data exists on the support near the guide sign. If the unnecessary three-dimensional point cloud data existing on the support near the guide sign is specified as the set area, the accuracy of specifying the set area decreases, and there is a possibility that an object including a traffic sign candidate is misrecognized as not being a guide sign even though it is originally a guide sign. Therefore, the above-mentioned predetermined conditions include that the point cloud density indicating the degree of density of the three-dimensional point cloud data is equal to or higher than a predetermined density.

[0063] The set area determination unit 333 performs a process of determining a set area based on the second set specified by the second set specifying unit 332. Specifically, in the horizontal width and vertical height calculated from the second set, an area of the second set in which at least the horizontal width is equal to or greater than a predetermined size is determined as the set area. Since the size of the image area of the object including the traffic sign candidate is larger than the size of the image of the traffic sign candidate, a value larger than the size of the image of the traffic sign candidate is set as the predetermined size that is the determination criterion for specifying the set area. As a method for calculating the horizontal width and vertical height of the second set, for example, when considering the three-dimensional coordinates of the three-dimensional point cloud data included in the second set as two-dimensional coordinates of the X coordinate and the Y coordinate, when the lower left of the two-dimensional coordinates is the origin, the distance from the reference coordinate indicating the coordinate based on the three-dimensional point cloud data closest to the origin to the first coordinate (the coordinate with the maximum horizontal direction (X direction) coordinate and the minimum vertical direction (Y direction) coordinate) may be estimated as the horizontal width. Further, the distance from the reference coordinate to the second coordinate (the coordinate with the minimum horizontal direction (X direction) coordinate and the maximum vertical direction (Y direction) coordinate) may be estimated as the vertical height. An example of the processing result when the set area is determined is shown as in FIG. 10.

[0064] FIG. 11 is a flowchart showing a processing example of the set area specifying unit 330 of the traffic sign recognition processing unit 300 according to the present embodiment.

[0065] In step S200, the set area specifying unit 330 specifies a first set of three-dimensional point cloud data equal to the relative position of the traffic sign candidate with respect to the moving body. Then, the process proceeds to step S210.

[0066] In step S210, the set area specifying unit 330 determines whether there is a second set that satisfies a predetermined condition among the first sets.

[0067] When it is determined that there is a second set that satisfies the predetermined condition (step S210; Yes), the process proceeds to step S220. Otherwise (step S210; No), the process proceeds to step S260.

[0068] In step S220, the set region specifying unit 330 calculates the horizontal width and vertical height of the second set. Thereafter, the process proceeds to step S230.

[0069] In step S230, the set region specifying unit 330 determines whether at least the horizontal width is equal to or greater than a predetermined size among the horizontal width and vertical height calculated in step S220.

[0070] If it is determined that at least the horizontal width is equal to or greater than the predetermined size (step S230; Yes), the process proceeds to step S240. Otherwise (step S230; No), the process proceeds to step S250.

[0071] In step S240, the set region specifying unit 330 determines the region of the corresponding second set as the set region.

[0072] In step S250, the set region specifying unit 330 determines the region of the non - corresponding second set as not being the set region.

[0073] In step S260, the set region specifying unit 330 determines that there is no set region.

[0074] In step S270, the set region specifying unit 330 outputs the determination result.

Explanation of Signs

[0075] 10 Traffic sign recognition device 100 Processor 110 Storage device 120 Information of camera image 130 Information of three - dimensional point cloud 140 Information of camera specifications 150 Information of specified size of traffic sign candidate 200 Information input unit 300 Traffic sign recognition processing unit 310 Traffic sign candidate detection unit 320 Relative position estimation unit 330 Set Region Specific Part 331 First Set Specific Part 332 Second Set Specific Part 333 Set Region Determination Part 340 Image Region Size Specific Part 350 Color Component Ratio Judgment Part 360 Traffic Sign Candidate Recognition Part 400 Processing Result Output Part

Claims

1. A storage device storing a camera image of a moving body, three-dimensional point cloud data, information on camera specifications including information on the resolution of the camera and information on the mounting height of the camera, and information on the specified size of a traffic sign candidate; A processor, and the processor is configured to: when an image of the traffic sign candidate is included in the camera image, estimate the relative position of the traffic sign candidate with respect to the moving body based on the camera image; specify a set of three-dimensional point cloud data in the three-dimensional point cloud data, in which the relative position with respect to the moving body is equal to that of the traffic sign candidate; specify, within the camera image, an image area of an object corresponding to the set area indicating the area where the set is specified and including the traffic sign candidate; calculate, based on the size of the image area of the object and the color components constituting the image of the object, the ratio occupied by a predetermined color component constituting a guiding sign among the color components constituting the image of the object; when the ratio occupied by the predetermined color component is equal to or greater than a threshold value, recognize that the object including the traffic sign candidate is a guiding sign; the relative position includes the relative distance of the traffic sign candidate with respect to the moving body and the height of the traffic sign candidate obtained by adding the mounting height of the camera to the relative height of the traffic sign candidate with respect to the moving body; the information of the camera image includes information of a plurality of camera images acquired by a plurality of cameras; the processor, in the estimation of the relative position, estimates the relative distance by performing a calculation using a predetermined calculation formula with the resolution of the camera, the number of horizontal pixels constituting the image of the traffic sign candidate, and the specified size; is configured to estimate the relative height by performing a calculation using a predetermined calculation formula with the relative distance, the resolution of the camera, and the number of vertical pixels constituting the image of the traffic sign candidate; A traffic sign recognition device.

2. A storage device storing a camera image of a moving body, three-dimensional point cloud data, and information on camera specifications including information on the resolution of the camera and information on the mounting height of the camera; A processor, and the processor is configured to: when an image of a traffic sign candidate is included in the camera image, estimate the relative position of the traffic sign candidate with respect to the moving body based on the camera image; specify a set of three-dimensional point cloud data in the three-dimensional point cloud data, in which the relative position with respect to the moving body is equal to that of the traffic sign candidate; corresponding to a set region indicating the region where the set is specified, and specifying an image region of an object including the traffic sign candidate within the camera image, calculating, based on the size of the image region of the object and the color components constituting the image of the object, the ratio occupied by a predetermined color component constituting a guiding sign among the color components constituting the image of the object, configured to recognize that the object including the traffic sign candidate is a guiding sign when the ratio occupied by the predetermined color component is equal to or greater than a threshold value, The relative position includes the relative distance of the traffic sign candidate with respect to the moving body and the height of the traffic sign candidate obtained by adding the mounting height of the camera to the relative height of the traffic sign candidate with respect to the moving body, The camera image includes a plurality of camera images acquired by a plurality of cameras with different arrangements, The processor, in estimating the relative position, when the image of the traffic sign candidate is included in the plurality of camera images, estimating the relative distance from the image of the traffic sign candidate based on the principle of triangulation, configured to estimate the relative height by performing a calculation using a predetermined calculation formula with the relative distance, the resolution of the camera, and the number of vertical pixels constituting the image of the traffic sign candidate, Traffic sign recognition device.

3. The traffic sign recognition device according to claim 1 or 2, wherein the set region has a horizontal width of at least a predetermined size in the horizontal width and vertical height calculated based on a set that satisfies a predetermined condition among the sets. Traffic sign recognition device.

4. The traffic sign recognition device according to claim 3, wherein the predetermined condition includes that the distance between the three-dimensional point cloud data included in the set is less than a predetermined distance and the point cloud density indicating the degree of density of the three-dimensional point cloud data included in the set is equal to or greater than a predetermined density. Traffic sign recognition device.

5. The traffic sign recognition device according to any one of claims 1 to 4, wherein the color components constituting the image of the object are represented by an integrated value obtained by integrating the pixel values of red pixels, an integrated value obtained by integrating the pixel values of green pixels, and an integrated value obtained by integrating the pixel values of blue pixels, and the predetermined color component includes at least one of the integrated value obtained by integrating the pixel values of the green pixels and the integrated value obtained by integrating the pixel values of the blue pixels. Traffic sign recognition device.

6. The traffic sign recognition device according to any one of claims 1 to 5, wherein the traffic sign candidate includes at least a traffic regulation sign candidate. Traffic sign recognition device.

7. The steps of the processor acquiring the camera image of the moving body, the three-dimensional point cloud data, the information on the specifications of the camera including the information on the resolution of the camera and the mounting height of the camera, and the information on the specified size of the traffic sign candidate; When the image of the traffic sign candidate is included in the camera image, the step of the processor estimating the relative position of the traffic sign candidate with respect to the moving body based on the camera image; The step of the processor specifying a set of three-dimensional point cloud data in which the relative position with respect to the moving body is equal to that of the traffic sign candidate among the three-dimensional point cloud data; The step of the processor specifying, within the camera image, an image area of an object corresponding to the set area where the set is specified and including the traffic sign candidate; The step of the processor calculating the ratio occupied by a predetermined color component constituting a guiding sign among the color components constituting the image of the object based on the size of the image area of the object and the color components constituting the image of the object; When the ratio occupied by the predetermined color component is equal to or greater than a threshold value, the step of the processor recognizing that the object including the traffic sign candidate is a guiding sign; including The relative position includes the relative distance of the traffic sign candidate with respect to the moving body and the height of the traffic sign candidate obtained by adding the mounting height of the camera to the relative height of the traffic sign candidate with respect to the moving body; The information of the camera image includes the information of a plurality of camera images acquired by a plurality of cameras; The step of estimating the relative position is The step of the processor estimating the relative distance by performing a calculation with a predetermined calculation formula using the resolution of the camera, the number of horizontal pixels constituting the image of the traffic sign candidate, and the specified size; The step of the processor estimating the relative height by performing a calculation with a predetermined calculation formula using the relative distance, the resolution of the camera, and the number of vertical pixels constituting the image of the traffic sign candidate; including Traffic sign recognition method.

8. The steps of the processor acquiring the camera image of the moving body, the three-dimensional point cloud data, and the information on the specifications of the camera including the information on the resolution of the camera and the mounting height of the camera; When the image of the traffic sign candidate is included in the camera image, the step of the processor estimating the relative position of the traffic sign candidate with respect to the moving body based on the camera image; A step in which the processor identifies a set of three-dimensional point cloud data among the three-dimensional point cloud data, the relative position of which with respect to the moving object is equal to that of the traffic sign candidate. A step in which the processor identifies, within the camera image, an image region of an object that corresponds to the identified set region and includes the traffic sign candidate. A step in which the processor calculates a ratio occupied by a predetermined color component that constitutes a guiding sign among the color components that constitute the image of the object, based on the size of the image region of the object and the color components that constitute the image of the object. A step in which the processor recognizes that the object including the traffic sign candidate is a guiding sign when the ratio occupied by the predetermined color component is equal to or greater than a threshold value. Including The relative position includes the relative distance of the traffic sign candidate with respect to the moving object and the height of the traffic sign candidate obtained by adding the mounting height of the camera to the relative height of the traffic sign candidate with respect to the moving object. The camera image includes a plurality of camera images acquired by a plurality of cameras with different arrangements. The step of estimating the relative position is When the image of the traffic sign candidate is included in the plurality of camera images, a step in which the processor estimates the relative distance from the image of the traffic sign candidate based on the principle of triangulation. A step in which the processor estimates the relative height by performing calculations using a predetermined calculation formula with the relative distance, the resolution of the camera, and the number of vertical pixels that constitute the image of the traffic sign candidate. Including Traffic sign recognition method.

Images