Image recognition apparatus, image recognition method, and program

Abstract

To make it possible to recognize a mark in an image under various optical environments.SOLUTION: An image acquiring unit 41 acquires a first image as a distance image. The image acquiring unit 41 acquires a second image as a visible image. A region specifying unit 42 detects an object having a predetermined shape in the first image and specifies, as a region image, an image of a region corresponding to an object detected in the second image. An adjustment unit 43 adjusts contrast on the region image based on a comparison result of a dispersion value of the region image and a predetermined dispersion value.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image recognition device, an image recognition method, and a program. [Background technology]

[0002] There is known a device that acquires an image using a camera mounted on a vehicle and automatically recognizes signs contained in the image. As a related technique, Patent Document 1 discloses a method of notifying the result of speed limit sign recognition. In Patent Document 1, an electronic control unit (ECU: Electronic Control Unit) of an in-vehicle navigation acquires an image in front of the vehicle from a camera. The electronic control unit automatically recognizes a speed limit sign in the image in front of the vehicle. The electronic control unit displays a first sign image, which is an image of the speed sign portion in the image in front of the vehicle, and a second sign image, which is the result of the automatic recognition, side by side on a display. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-205160 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 describes that an electronic control device extracts a circular area from an image in front of the vehicle and adjusts the contrast of the extracted circular area. However, for example, in a lighting environment such as at night or in backlight, it may not be possible to extract a circular area corresponding to a sign from the image in front of the vehicle. In particular, when there are two signs with significantly different brightness in front of the vehicle, it may be possible to extract an area corresponding to one sign from the image in front of the vehicle, but not to extract an area corresponding to the other sign.

[0005] The present invention has been made to solve such problems, and aims to provide an image recognition device, an image recognition method, and a program that make it possible to recognize signs present in an image under various lighting environments. [Means for solving the problem]

[0006] Therefore, the present invention provides an image recognition device including an image acquisition unit that acquires a first image which is a distance image and a second image which is a visible image, an area identification unit that detects an object having a predetermined shape in the first image and identifies an image of an area in the second image corresponding to the detected object as a area image, and an adjustment unit that adjusts the contrast of the area image based on a comparison result between a variance value of the area image and a predetermined variance value.

[0007] The present invention also provides an image recognition method, which includes acquiring a first image which is a distance image, acquiring a second image from a second camera which is a visible image, detecting an object having a predetermined shape in the first image, identifying an image of a region in the second image which corresponds to the detected object as a region image, and adjusting the contrast of the region image based on a result of comparing a variance value of the region image with a predetermined variance value.

[0008] The present invention provides a program for causing a processor to execute a process of acquiring a first image which is a distance image, acquiring a second image which is a visible image, detecting an object having a predetermined shape in the first image, identifying an image of a region in the second image which corresponds to the detected object as a region image, and adjusting the contrast of the region image based on a result of comparing a variance value of the region image with a predetermined variance value. Effect of the Invention

[0009] According to the present invention, it is possible to recognize signs present in an image under various lighting environments. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a vehicle including an image recognition device according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing an example of the configuration of an image recognition device. [Diagram 3] FIG. 4 is a schematic diagram showing an example of a visible image captured by a visible camera. [Figure 4] FIG. 4 is a schematic diagram showing an example of an image taken at the same location as FIG. 3 at night. [Diagram 5] FIG. 5 is a schematic diagram showing an example of an image taken at the same location as in FIG. 3 at night after gain adjustment. [Figure 6] FIG. 2 is a schematic diagram showing an overview of a distance image. [Figure 7] FIG. 13 is a diagram showing an example of a region image. [Figure 8] FIG. 13 is a diagram showing another example of a region image. [Figure 9] FIG. 13 is a diagram showing another example of a region image. [Figure 10] FIG. 13 is a diagram showing an example of an image cut out from a region image. [Figure 11] FIG. 11 is a diagram showing another example of an image cut out from a region image. [Figure 12] 6A and 6B are diagrams showing an example of the relationship between pixel values ​​before and after contrast adjustment. [Figure 13] 4 is a flowchart showing an operation procedure of the image recognition device. [Figure 14] 11 is a flowchart showing an operation procedure of contrast adjustment. [Figure 15] FIG. 13 is a schematic diagram showing an example of a visible image after contrast adjustment. [Figure 16] FIG. 11 is a block diagram showing an image recognition device according to a second embodiment. [Figure 17] FIG. 11 is a block diagram showing an image recognition device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present technology will be described in detail with reference to the drawings. Note that the following description and drawings are omitted and simplified as appropriate for clarity of explanation. In addition, in each of the following drawings, the same elements and similar elements are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0012] FIG. 1 shows a vehicle including an image recognition device according to a first embodiment of the present invention. The vehicle 10 has a visible camera 20, a TOF (Time Of Flight) camera 30, and an image recognition device 40. The visible camera 20 is a camera that captures images of the area ahead of the vehicle 10. In the following description, the images captured by the visible camera 20 are also referred to as visible images. The TOF camera 30 is a camera that captures distance images of the area ahead of the vehicle 10. The visible camera 20 and the TOF camera 30 acquire images of the same or nearly the same capture area.

[0013] The visible camera 20 and the TOF camera 30 do not necessarily have to have the same angle of view as long as the coordinate relationship in each image is known and the coordinates can be converted between them. Although one visible camera 20 and one TOF camera 30 are illustrated in FIG. 1, the number of visible cameras 20 and TOF cameras 30 is not particularly limited to one. For example, the vehicle 10 may have at least one of the visible camera 20 and the TOF camera 30 that captures the area behind the vehicle, in addition to the visible camera 20 and the TOF camera 30 that capture the area in front of the vehicle.

[0014] The image recognition device 40 controls the visible camera 20 and the TOF camera 30. In addition, the image recognition device 40 performs image processing and image recognition processing on the images captured by the visible camera 20 and the TOF camera 30. The image recognition device 40 is also called a camera control unit (CCU).

[0015] 2 shows an example of the configuration of an image recognition device 40. The image recognition device 40 has an image acquisition unit 41, an area specification unit 42, an adjustment unit 43, and a sign recognition unit 44. The image acquisition unit 41 acquires a visible image captured by the visible camera 20. The image acquisition unit 41 also acquires a distance image captured by the TOF camera 30. The TOF camera 30 is also called a first camera, and the distance image is also called a first image. The visible camera 20 is also called a second camera, and the visible image is also called a second image.

[0016] In this embodiment, an example will be described in which the TOF camera 30 is used to acquire the distance image, but the camera used to acquire the distance image is not limited to a TOF camera. The image acquisition unit 41 may acquire the distance image from a device that acquires a three-dimensional point cloud image, such as a LiDAR (Light Detection And Ranging) device.

[0017] Fig. 3 shows an example of a typical visible image captured by the visible camera 20. In this example, there is a no parking sign 51 on the side of the road, and a sign 52 indicating a speed limit on the upper part of the road. Here, it is assumed that the sign 51 is a normal plate-shaped road sign. Meanwhile, the sign 52 is a sign that has lighting inside and emits light at night. In Fig. 3, the background and other vehicles such as a vehicle ahead and an oncoming vehicle are omitted for the sake of simplicity.

[0018] FIG. 4 shows an example of an image taken at night in the same location as FIG. 3. Here, it is assumed that the gain of the visible light camera 20 is not adjusted. At night, if the sign 51 is not illuminated by lighting such as the headlights of the vehicle 10, the sign 51 will appear dark in the image captured by the visible light camera 20, making it difficult to recognize the sign 51 from the image. On the other hand, the sign 52 will be illuminated at night. For this reason, the sign 52 will appear bright in the visible image captured by the visible light camera 20, and it is considered possible to recognize the sign 52 from the image.

[0019] Fig. 5 shows an example of an image taken at the same location as Fig. 3 at night after gain adjustment. Suppose the gain of the visible light camera 20 is increased to make objects that are difficult to see in a dark scene easier to see in order to photograph in a dark location. In this case, as shown in Fig. 5, a sign 51 that was dark and sunken in Fig. 4 appears bright enough to be recognized in the image. However, on the other hand, a sign 52 that appeared bright before the gain adjustment appears washed out as a side effect of increasing the gain, making it difficult to recognize in the image.

[0020] 6 shows an overview of an output image, i.e., a distance image, of the TOF camera 30. The TOF camera 30 measures the distance to an object by emitting infrared light to a shooting area and detecting the reflected light of the emitted infrared light. The distance image captured by the TOF camera 30 is not affected by the lighting environment or the brightness of the image. Therefore, the TOF camera 30 can stably capture distance images whether it is day or night.

[0021] The TOF camera 30 outputs a signal corresponding to the distance from the TOF camera 30 to an object for each pixel, for example. When the output signal of the TOF camera 30 is an 8-bit signal, the distance is represented by a signal of 256 levels. In general, a distance image is displayed by converting the signal of each pixel into a color that is easy for people to see. For the sake of simplicity of explanation, in FIG. 6, the distance is shown in four levels, from close to far, namely horizontal lines, diagonal lines, polka dots, and white.

[0022] In Fig. 6, a no parking sign 51 is displayed with horizontal lines indicating that it is the closest sign. Meanwhile, a speed limit sign 52 is displayed with polka dots indicating that it is a little further away. The visible image obtained from the visible camera 20 may not be an image suitable for recognition depending on the surrounding brightness, the gain of the signal processing, etc. Meanwhile, the distance image obtained from the TOF camera 30 is not easily affected by the surrounding brightness or the gain of the signal processing.

[0023] A distance image does not contain information about the color or brightness of an object. Therefore, it is not possible to recognize the patterns or numbers on a sign from a distance image. However, a distance image can be used to determine the position, size, shape, and whether or not an object is flat.

[0024] Returning to FIG. 2, the area identification unit 42 detects an object of a predetermined shape in the distance image acquired by the TOF camera 30. In this embodiment, the area identification unit 42 detects a road sign or an equivalent display object from the distance image. The area identification unit 42 calculates an index indicating the likelihood of the object being a road sign according to, for example, the shape, size, and position of the object, and detects the road sign or an equivalent display object based on the calculated index. The location of the object on the road can be determined, for example, by detecting the area of ​​the road in the visible image and calculating which position in the distance image the detected area of ​​the road corresponds to.

[0025] When a road sign or a similar display object is detected, the area identifying unit 42 identifies an area in the visible image captured by the visible camera 20 that corresponds to the area of ​​the object detected in the distance image. The area identifying unit 42 identifies a partial image of the identified area in the visible image as a area image. When multiple objects are detected in the distance image, the area identifying unit 42 identifies area images in the visible image for each of the detected multiple objects.

[0026] When a sign portion is crushed black or crushed white, such as sign 51 in Fig. 4 or sign 52 in Fig. 5, it is difficult to correctly recognize what kind of sign it is in the visible image. Adjustment unit 43 adjusts the contrast of the portion of the regional image corresponding to sign 51 or sign 52 so that the sign can be recognized in the visible image such as that in Fig. 4 or Fig. 5.

[0027] The adjustment unit 43 determines, for example, whether the region image is an image that can be used for sign recognition without contrast adjustment. In this embodiment, the adjustment unit 43 determines whether contrast adjustment is necessary using a variance value of pixel values. The adjustment unit 43 calculates the variance value of the region image identified by the region identification unit 42. The adjustment unit 43 compares the calculated variance value of the region image with a predetermined variance value, and determines whether to adjust the contrast of a portion of the region image in the visible image based on the comparison result.

[0028] The above-mentioned predetermined variance value is used as a threshold value for determining whether or not to perform contrast adjustment. When the variance value of the region image is smaller than the predetermined variance value, the adjustment unit 43 determines that contrast adjustment is to be performed on the region image. When the variance value of the region image is equal to or greater than the predetermined variance value, the adjustment unit 43 determines that contrast adjustment is not necessary for the region image. When multiple region images are identified by the region identification unit 42, the adjustment unit 43 determines whether or not to perform contrast adjustment on each of the multiple region images.

[0029] The predetermined variance value can be determined, for example, by preparing a plurality of visible images captured under different shooting conditions as learning data, and using the variance value of the sign region in images included in the learning data in which the sign can be recognized by the sign recognition unit 44 described below. The learning data includes images in which the region of the sign to be recognized is darkened and images in which the region of the sign is slightly washed out. The adjustment unit 43 determines the variance value of the central region of the sign region in the captured visible image. The adjustment unit 43 can determine the minimum value of the variance values ​​of the images in which the sign can be recognized by the sign recognition unit 44 as the predetermined variance value. In this case, the predetermined variance value corresponds to the lower limit of the variance values ​​of the images in which the sign can be recognized by the sign recognition unit 44.

[0030] FIG. 7 shows an example of a region image identified in a visible image captured in a certain situation. In this example, the road sign is a pedestrian crossing sign. In the following description, the visible image is assumed to be a monochrome 8-bit gradation image. The pixel value of the visible image corresponds to the luminance. FIG. 7 shows an example of a region image in which the visible image is captured in daytime in a situation where the sign can be clearly recognized and the background is a blue sky. When the variance value is calculated for the region image shown in FIG. 7, the variance value is assumed to be 6768.

[0031] Fig. 8 shows an example of a region image identified in a visible image captured in a different situation from that of Fig. 7. Fig. 8 shows an example of a region image obtained by capturing a visible image at night in which no lighting is shining on a sign. In this example, the pixel values ​​of the region image are distributed in areas close to black, and the variance value calculated for the region image shown in Fig. 8 is 33.

[0032] FIG. 9 shows an example of a region image identified in a visible image captured in yet another situation. FIG. 9 shows an example of a region image captured during the day, but in which a sign is in the shade and the background is blue sky. In this example, there is a difference in brightness between the sign portion and the background portion in the region image. When the variance value is calculated for the region image shown in FIG. 9, it is assumed that the variance value is 6953. A situation in which there is such a difference in brightness between the sign and the background can also occur in a case such as the self-luminous sign 52 at night shown in FIG. 4.

[0033] In calculating the variance value of the region image, the adjustment unit 43 may remove the background region from the region image and exclude the background portion from the calculation of the variance value in order to reduce the influence of the brightness difference between the sign and the background. For example, the adjustment unit 43 cuts out a certain region near the center of the region image. The adjustment unit 43 cuts out a region that is 50% of the center vertically and horizontally by removing, for example, 25% each on the top, bottom, left, and right from the region image. The adjustment unit 43 may calculate the variance value from the cut image near the center.

[0034] Fig. 10 shows an example of an image cut out from a region image. The adjustment unit 43 cuts out an image of a certain region near the center shown in Fig. 10 from the region image shown in Fig. 7, for example. The image shown in Fig. 10 is an image of a region corresponding to a part of a sign, and does not include a background region. When the variance value is calculated for the image shown in Fig. 10, the variance value is assumed to be 9020.

[0035] Fig. 11 shows another example of an image cut out from a region image. The adjustment unit 43 cuts out an image of a certain region near the center shown in Fig. 11 from the region image shown in Fig. 9, for example. The image shown in Fig. 11 is an image of a region corresponding to a part of a sign in the shade, and does not include a bright background region. When the variance value is calculated for the image shown in Fig. 11, the variance value is assumed to be 39.

[0036] When the variance value of the entire region image is used, as in the example shown in Fig. 9, if there is a difference in brightness between the background and the sign portion, the variance value may be small in the sign portion but large overall. On the other hand, the variance value of the image near the center of the region image is not affected by the background, so it will be small when the sign portion is dark and therefore not suitable for image recognition. In this way, by cutting out a certain area near the center that is not affected by the background from the region image specified using the distance image from the TOF camera 30 and calculating the variance value of that area, it is possible to determine whether the sign can be recognized in the visible image.

[0037] In contrast adjustment, the adjuster 43 adjusts the pixel values ​​of the regional image so that the variance of the pixel values ​​of the regional image becomes large. The adjuster 43 performs contrast adjustment in accordance with, for example, the bias of the distribution of the pixel values ​​of the regional image.

[0038] Fig. 12 shows an example of the relationship between pixel values ​​before and after contrast adjustment. In Fig. 12, the horizontal axis corresponds to input pixel values, i.e., pixel values ​​before contrast adjustment. The vertical axis corresponds to output pixel values, i.e., pixel values ​​after contrast adjustment. For example, when pixel values ​​of a region image are biased toward pixel values ​​close to white, adjustment unit 43 performs contrast adjustment using a conversion function shown by tone curve A in Fig. 12. In this case, adjustment unit 43 adjusts pixel values ​​so that minute differences in pixel values ​​are enlarged in a range RA of pixel values ​​close to white.

[0039] When the pixel values ​​of the region image are biased toward pixel values ​​close to black, the adjustment unit 43 performs contrast adjustment using, for example, a conversion function shown by a tone curve B in FIG. 12. In this case, the adjustment unit 43 adjusts the pixel values ​​so that minute differences in pixel values ​​are enlarged in a pixel value range RB close to black. When the pixel values ​​of the region image are biased toward a specific pixel value between white and black, the adjustment unit 43 performs contrast adjustment using, for example, a conversion function shown by a tone curve C in FIG. 12. In this case, the adjustment unit 43 adjusts the pixel values ​​so that minute differences in pixel values ​​are enlarged in a specific intermediate pixel value range RC.

[0040] In the contrast adjustment, the adjustment unit 43 may perform the contrast adjustment with an adjustment amount according to the variance value of the region near the center of the region image. When a plurality of region images are specified, the adjustment unit 43 performs the contrast adjustment for each region image.

[0041] The sign recognition unit 44 performs sign recognition on the visible image and recognizes what kind of sign the sign is. The sign recognition unit 44 stores, for example, a recognition model for recognizing signs. The recognition model is an inference model for recognizing signs, created, for example, by machine learning. The sign recognition unit 44 inputs an area image into the recognition model and calculates a score indicating the sign-likeness of each of a plurality of signs to be recognized. The sign recognition unit 44 may calculate the probability of each sign output by the recognition model as a score. When the contrast of the area image is not adjusted, the sign recognition unit 44 may calculate a higher score than when the contrast is adjusted. The sign recognition unit 44 recognizes a sign with a score equal to or greater than a predetermined threshold and with the highest score as the sign of the input image. When there is no sign with a score equal to or greater than a predetermined threshold, the sign recognition unit 44 fails to recognize the sign.

[0042] The above recognition model is created by machine learning using a plurality of learning data prepared from images of sign portions captured under different shooting conditions. For example, the learning data is teacher data for a plurality of signs, including images of sign portions contained in visible images captured under a plurality of shooting conditions and labels indicating which sign each is, as shown in Fig. 7 to Fig. 9. A learning device (not shown) learns the learning data and generates a recognition model that identifies a sign from an area image.

[0043] The learning device, for example, cuts out a certain area near the center of an image of a sign portion in the learning data, and calculates the variance of pixel values ​​in the cut-out area, in the same way as calculating the variance value in a region image. Specifically, the learning device selects, as an image of the sign portion, an image whose score is equal to or greater than a predetermined threshold and which is successfully recognized by the recognition model, and calculates the variance value. The sign recognition unit 44 stores the minimum value of the variance value calculated by the learning device, i.e., the lower limit value of the variance value for images in which a sign can be recognized. The lower limit value of the variance value for images in which a sign can be recognized is used as the predetermined variance value in the adjustment unit 43.

[0044] When contrast adjustment has been performed by the adjustment unit 43, the sign recognition unit 44 performs sign recognition on the visible image in which the contrast of the portion of the area image identified by the area identification unit 42 has been adjusted. The sign recognition unit 44 recognizes signs by inputting the visible image into a recognition model. The sign recognition unit 44 displays the result of the sign recognition on a display (not shown). The sign recognition unit 44 displays the result of the sign recognition on, for example, a screen of a car navigation device or a display audio.

[0045] Here, when learning data used to create a recognition model for recognizing signs exists for multiple types of signs, the minimum variance value for image recognition is considered to be different for each sign. The above-mentioned predetermined variance value, which is a threshold value, may be set to the maximum value among the minimum variance values ​​for image recognition that are contained in the learning data of each sign. The sign recognition unit 44 stores the predetermined variance value that is the maximum value among the minimum variance values ​​for image recognition that are contained in the learning data of each sign. The adjustment unit 43 selects the predetermined variance value stored in the sign recognition unit 44. The adjustment unit 43 may select the maximum value among the minimum variance values ​​for image recognition that are contained in the learning data of each sign as the predetermined variance value.

[0046] Here, to simplify the explanation, only two signs, "speed limit" and "no parking", are considered. For the learning data of "speed limit" signs, the minimum variance value for image recognition is assumed to be 40. Also, for the learning data of "no parking" signs, the minimum variance value for image recognition is assumed to be 50. In this case, if the predetermined variance value is set to "40", the "speed limit" signs will be able to be recognized as images, but the "no parking" signs will not be able to be recognized as images. For this reason, the predetermined variance value is set to a higher value, "50", so that the "no parking" signs will be able to be recognized as images.

[0047] In addition, when the learning data used to create the recognition model for recognizing a sign exists for a plurality of types of signs, the learning data for the plurality of types of signs may be grouped, for example, according to the shape of the sign. For example, the learning data for the plurality of types of signs is grouped into a group of triangular signs and a group of round signs. In this case, the predetermined variance value may be set for each group. The sign recognition unit 44 stores the predetermined variance values ​​grouped by shape. The area identification unit 42 can identify the shape of an object that seems to be a sign in the distance image. The adjustment unit 43 may compare the variance value of the area image with a predetermined variance value that is set according to the shape of the detected object that seems to be a road sign. For example, the adjustment unit 43 may select a predetermined variance value based on the shape of a sign including an object that seems to be a sign from the variance values ​​for each shape stored in the sign recognition unit 44.

[0048] For example, for a group of learning data of triangular signs, the minimum variance value for image recognition is assumed to be 80. Also, for a group of learning data of round signs, the minimum variance value for image recognition is assumed to be 50. When not grouping, the predetermined variance value is set to a higher value of "80" in order to recognize images of both triangular and round signs. When grouping, if the detected object that seems to be a road sign is triangular, the predetermined variance value is set to "80". Also, if the detected object that seems to be a road sign is round, the predetermined variance value is set to "50". In this case, the predetermined variance value can be lowered to "50" for round signs, which has the advantage of reducing the frequency of contrast adjustment.

[0049] Next, the operation procedure will be described. Fig. 13 shows the operation procedure of the image recognition device 40. The image acquisition unit 41 acquires a visible image acquired by photographing an area in front of the vehicle from the visible camera 20 (step S1). The image acquisition unit 41 also acquires a distance image acquired by photographing an area in front of the vehicle from the TOF camera 30 (step S2). Steps S1 and S2 may be performed in parallel. In step S2, the image acquisition unit 41 acquires, for example, a distance image with approximately the same angle of view as the visible image acquired in step S1.

[0050] The region identification unit 42 detects road signs or similar display objects from the distance image acquired in step S2 (step S3). In step S3, the region identification unit 42 detects objects that are likely to be road signs, for example, based on the object's position, height, distance, size, shape, and whether or not it is flat. For example, the region identification unit 42 calculates an index indicating the likelihood of an object being a road sign for each object that can be detected from the distance image, and detects an object whose index indicating the likelihood of an object being a road sign is equal to or greater than a predetermined value as a road sign or a similar display object.

[0051] For example, a roadside pole-type sign is located at a position of about 25 cm to 50 cm from the side of the road and at a height of about 1 m to 2 m. The area identification unit 42 can detect an object located at such a position as an object that resembles a road sign. Also, a cantilever-type or gate-type sign on the road is located at a position of about 4.5 m to 5 m above the road, has a size of about several tens of centimeters to 1.5 m, and is circular, rectangular, diamond, or triangular in shape. Also, a cantilever-type or gate-type sign on the road has a flat surface. The area identification unit 42 can detect an object whose position, size, and shape match those of a cantilever-type or gate-type sign on the road as an object that resembles a road sign.

[0052] The region identification unit 42 identifies, as a region image, an image of a region in the visible image that corresponds to the region detected from the distance image in step S3 (step S4). If multiple road signs are detected in step S3, the region identification unit 42 identifies, for the multiple detected road signs, images of the road sign portions in the visible image as region images.

[0053] The adjustment unit 43 calculates a variance value of pixel values ​​from the region image identified by the region identification unit 42 (step S5). In step S5, in order to reduce the influence of a background portion included in the region image, i.e., a portion other than the sign, the adjustment unit 43 may cut out a certain region near the center of the region image as a judgment region and calculate the variance value in the judgment region.

[0054] The adjustment unit 43 compares the variance value calculated in step S5 with a predetermined variance value used as a threshold value (step S6). The adjustment unit 43 judges whether the variance value is smaller than the threshold value (step S7). If the variance value calculated in step S5 is equal to or greater than the threshold value (step S7: NO), the adjustment unit 43 judges that the area image is an image in which signs can be recognized and that contrast adjustment is not necessary. If the variance value calculated in step S5 is smaller than the threshold value (step S7: YES), the adjustment unit 43 judges that the area image is an image in which signs cannot be recognized as it is. In that case, the adjustment unit 43 performs contrast adjustment on the area image identified in step S4 (step S8). After the contrast adjustment, the process returns to step S5, and the adjustment unit 43 calculates the variance value of the area image after the contrast adjustment.

[0055] Fig. 14 shows the operation procedure of contrast adjustment. The adjustment unit 43 calculates the average value of pixel values ​​of an image of a judgment region (see Fig. 10 or 11) cut out from a region image in which the variance is to be calculated (step S81). The adjustment unit 43 judges whether the average value calculated in step S81 is greater than a first threshold value set in advance (step S82). The first threshold value is set to a value slightly smaller than the pixel value corresponding to "white" in a gradation image, for example. If the average value is greater than the first threshold value, the image of the judgment region is considered to be whiteout.

[0056] If it is determined in step S82 that the average value is greater than the first threshold value (step S82: YES), adjustment unit 43 adjusts pixel values ​​so as to reduce the gain in the regional image (step S83). In step S83, adjustment unit 43 calculates pixel values ​​of each pixel after adjustment of the regional image using a conversion function indicated by tone curve A shown in Fig. 12, for example. Adjustment unit 43 adjusts the contrast of the regional image by reducing the gain, thereby increasing the contrast.

[0057] If the average value calculated in step S81 is equal to or less than the first threshold value (step S82: NO), the adjustment unit 43 judges whether the average value is smaller than a preset second threshold value (step S84). The second threshold value is set to a value smaller than the first threshold value, for example, a value slightly larger than the pixel value corresponding to "black" in a gradation image. If the average value is smaller than the second threshold value, the image in the judgment area is considered to be blacked out.

[0058] If the adjustment unit 43 determines in step S84 that the average value is smaller than the second threshold value (step S84: YES), it adjusts the pixel values ​​so as to increase the gain in the regional image (step S85). The adjustment unit 43 may calculate the pixel values ​​of each pixel in the regional image after adjustment, for example, using a conversion function indicated by the tone curve B shown in Fig. 12. The adjustment unit 43 adjusts the contrast of the regional image by increasing the gain, thereby increasing the contrast.

[0059] If it is determined in step S84 that the average value is equal to or greater than the second threshold value (step S84: NO), it is considered that the pixel values ​​in the region image are concentrated near a specific pixel value between white and black. In this case, the adjustment unit 43 performs a process of amplifying contrast in the specific pixel value range (step S86). In step S86, the adjustment unit 43 may calculate the pixel value of each pixel after adjustment of the region image using, for example, a conversion function indicated by the tone curve C shown in FIG. 12. In step S86, the adjustment unit 43 may amplify the contrast, for example, centering on the average value of the region image.

[0060] In steps S83, S85, and S86, the adjustment unit 43 may set a target variance value after contrast adjustment and adjust the contrast so that the variance value approaches the target variance value. The target variance value is set to a value greater than the minimum variance value among the variance values ​​of the area images in which the sign recognition is successful in the learning data. The adjustment unit 43 may set a predetermined variance value as the target variance value. As an example, the target variance value is set to a value obtained by squaring the average of the square root of the minimum variance value and the square root of the maximum variance value among the variance values ​​of the area images in which the sign recognition is successful. For example, when the variance value of the area image is 16, the square root is 4. When the target variance value is 100, that is, 10 squared, a region image with a desired contrast can be obtained by applying a gain of 10 / 4=2.5 times to the signal of the area image in the visible image. When the regional image is washed out white, the adjustment unit 43 may increase the contrast by first lowering the gain of the regional image to make the signal unsaturated, calculating the variance value of the regional image, and then increasing the gain according to the calculated variance value.

[0061] If it is determined in step S7 that the variance value is equal to or greater than the threshold value, the sign recognition unit 44 performs image recognition processing on the visible image to recognize signs contained in the visible image (step S9). The sign recognition unit 44 displays the recognized signs on the display and notifies the user of the vehicle 10 of the speed limit, etc. In the above description, an example has been described in which the processing returns to step S5 when contrast adjustment is performed in step S8. Alternatively, the processing may proceed from step S8 to step S9, and in step S9, image recognition processing is performed on the visible image on which contrast adjustment has been performed.

[0062] FIG. 15 shows an example of a visible image in which the contrast has been adjusted by the adjustment unit 43. The adjustment unit 43 increases the gain of the regional image of the portion of the sign 51 that is crushed black with respect to the image shown in FIG. 4. In this case, as shown in FIG. 15, a visible image in which the gain is amplified only for the portion of the sign 51 is obtained. In this case, even if a dark sign 51 and a bright sign 52 are mixed in the visible image, the adjustment unit 43 can provide the sign recognition unit 44 with a visible image in which both the signs 51 and 52 can be recognized. Therefore, the sign recognition unit 44 can recognize both the signs 51 and 52 from the visible image.

[0063] Recognition of road signs using the visible image captured by the visible camera 20 is greatly affected by the brightness or luminance of the subject. On the other hand, distance measuring sensors such as the TOF camera 30 can detect the distance to an object by irradiating infrared rays and detecting the reflection of the infrared rays, and can detect the object without depending on the brightness of the object. In this embodiment, the area identification unit 42 identifies an area of ​​an object that seems to be a road sign using the distance image captured by the TOF camera 30, and identifies an area image of the visible image corresponding to the detected area. The adjustment unit 43 judges whether the sign can be recognized without performing contrast adjustment on the area image. When the adjustment unit 43 judges that image recognition is difficult without performing contrast adjustment, it performs contrast adjustment on the area image so that image recognition is possible. In this way, the image recognition device 40 according to this embodiment can recognize the sign from the visible image without depending on the light environment or the brightness of the subject.

[0064] Next, a second embodiment of the present invention will be described. Fig. 16 shows an image recognition device according to the second embodiment. In this embodiment, an image recognition device 40a has a map information storage unit 45 and a selection unit 46 in addition to the configuration of the image recognition device 40 shown in Fig. 2. The operations of the image acquisition unit 41, the area identification unit 42, and the adjustment unit 43 in this embodiment may be similar to those described in the first embodiment.

[0065] In this embodiment, the sign recognition unit 44 outputs the sign calculated as the highest score from the scores for each sign calculated as a result of recognizing the sign using the recognition model as the recognized sign, and outputs the score of the recognized sign to the selection unit 46. When the recognized road sign indicates a speed limit, the selection unit 46 obtains the numerical value of the speed limit. Furthermore, the selection unit 46 obtains the score of the recognized sign as the accuracy of image recognition. The accuracy of image recognition is related to the accuracy of the recognized road sign.

[0066] The map information storage unit 45 stores map information. For example, the map information storage unit 45 stores information indicating the speed limit of the road on which the vehicle 10 (see FIG. 1) is traveling for each road. The selection unit 46 acquires the speed limit value of the road on which the vehicle 10 is traveling from the map information storage unit 45. For example, the selection unit 46 acquires the current position of the vehicle 10, and acquires the speed limit of the road corresponding to the current position from the map information storage unit 45. The speed limit acquired from the sign recognition unit 44 is also called the first speed limit. The speed limit acquired from the map information storage unit 45 is also called the second speed limit.

[0067] The selection unit 46 compares the first speed limit with the second speed limit. If the first speed limit and the second speed limit are different from each other, the selection unit 46 determines whether the accuracy of image recognition is lower than a predetermined value. If the accuracy of image recognition is lower than the predetermined value, the selection unit 46 selects the lower speed limit between the first speed limit and the second speed limit. If the accuracy of image recognition is equal to or higher than the predetermined value, the selection unit 46 selects the first speed limit. The selection unit 46 displays the selected speed limit on the display.

[0068] For example, suppose that the first speed limit acquired from image recognition is "40km / h" and the second speed limit acquired from map information is "50km / h". If the accuracy of image recognition is low, the first speed limit acquired from image recognition may be incorrect. However, the first speed limit may be correct. If the accuracy of the first speed limit is low and the second speed limit is displayed on the display with priority, the user may mistakenly believe that the speed limit is "50km / h" when actually traveling on a road with a speed limit of "40km / h". In the above case, if the accuracy of image recognition is low, that is, if the accuracy of the first speed limit is low, the selection unit 46 selects the low speed limit of "40km / h". In this case, there is an effect of suppressing the user from traveling beyond the actual speed limit.

[0069] On the other hand, when the accuracy of the image recognition is high, it is considered that the first speed limit acquired from the image recognition is likely to be correct. When the accuracy of the image recognition is high and the first speed limit and the second speed limit are different, the update frequency of the map information storage unit 45 may be low and an old speed limit may have been acquired as the second speed limit. Alternatively, the position measurement accuracy of the vehicle 10 may be low and the speed limit of a road different from the road on which the vehicle is actually traveling may have been acquired as the second speed limit. In the above case, when the accuracy of the image recognition is high, the selection unit 46 selects the first speed limit of "40km / h". In this case, the selection unit 46 can preferentially display on the display the speed limit of the road sign actually installed on the road. Other effects are similar to those described in the first embodiment.

[0070] In the second embodiment, the image recognition device 40a does not have to have the area identification unit 42 and the adjustment unit 43. Furthermore, the vehicle 10 (see FIG. 1) does not have to have the TOF camera 30. In that case, the sign recognition unit 44 may perform image recognition on the visible image acquired by the image acquisition unit 41. The selection unit 46 displays the lower of the first and second speed limits, or the first speed limit, on the display according to the accuracy of the image recognition.

[0071] Next, a third embodiment of the present invention will be described. FIG. 17 shows an image recognition device according to the third embodiment. The image recognition device 40b according to this embodiment has a learning data storage unit 47 and a learning unit 48 in addition to the configuration of the image recognition device 40 shown in FIG. 2. The operations of the image acquisition unit 41, the area identification unit 42, and the adjustment unit 43 in this embodiment may be similar to those described in the first embodiment. The image recognition device 40b may have a learning data storage unit 47 and a learning unit 48 in addition to the configuration of the image recognition device 40a shown in FIG. 16.

[0072] In this embodiment, the sign recognition unit 44 performs image recognition processing using the recognition model while updating the recognition model. The recognition model of the sign recognition unit 44 is the same as the recognition model described in the first embodiment before the update. The learning data storage unit 47 stores learning data. The learning unit 48 updates the recognition model of the sign recognition unit 44 by learning the learning data stored in the learning data storage unit 47. Nothing is stored in the learning data storage unit 47 before the recognition model is updated.

[0073] The sign recognition unit 44 outputs the sign calculated as the highest score from the scores calculated for each sign as a result of recognizing the sign using the recognition model as the recognized sign, and outputs the score of the recognized sign to the learning unit 48 as the accuracy of the image recognition processing. The learning unit 48 determines whether the accuracy of the image recognition processing in the sign recognition unit 44 is higher than a predetermined value for the area image after contrast adjustment. If the accuracy of the image recognition processing is higher than the predetermined value and the variance value of the area image before contrast adjustment is smaller than the predetermined variance value described in the first embodiment, the learning unit 48 stores the area image before contrast adjustment in the learning data storage unit 47. The learning unit 48 associates the sign recognition result with the area image before contrast adjustment and stores them in the learning data storage unit 47 as teacher data.

[0074] The learning unit 48 performs additional learning and updates the recognition model using the learning data stored in the learning data storage unit 47. The learning unit 48 can perform additional learning using known methods such as transfer learning and fine tuning. Here, the learning data storage unit 47 may include the learning data used to generate the recognition model before updating.

[0075] The learning unit 48 may perform the above-mentioned process, for example, every time image recognition is performed on the area image on which contrast adjustment has been performed. Alternatively, the learning unit 48 may perform the above-mentioned process at a predetermined timing. For example, the learning unit 48 may perform the above-mentioned process at a timing such as at night or while the vehicle is stopped. Alternatively, the learning unit 48 may transmit the area image before contrast adjustment to a server via a wireless network, have the server learn a recognition model, and receive the learned recognition model.

[0076] If image recognition processing can be performed with high accuracy on the area image after contrast adjustment, there is a possibility that image recognition processing can be performed with a certain degree of accuracy on the area image before contrast adjustment. In this embodiment, the learning unit 48 additionally learns a recognition model using such an area image. By performing additional learning, the learning unit 48 can generate a recognition model capable of image recognition even for an area image with a low variance value and difficult image recognition. In addition, the learning unit 48 can generate a recognition model according to a visible image captured at a location where the vehicle 10 is traveling. Other effects are similar to those described in the first or second embodiment.

[0077] In each of the above embodiments, the image recognition device 40 is configured as a device having, for example, one or more processors and one or more memories. The one or more memories store programs. At least a part of the functions of the image recognition device 40 can be realized by the processor executing processes according to the programs or instructions read from the memories.

[0078] The above-mentioned program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or the tangible storage medium includes Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Solid State Drive (SSD) or other memory technology, CD-ROM, Digital Versatile Disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium includes electrical, optical, acoustic, or other forms of propagating signals.

[0079] It should be noted that the image recognition device 40 does not necessarily have to be configured as a single device. The image recognition device 40 may be configured using a plurality of devices that are physically separated. Moreover, it is not necessarily required that all functions of the image recognition device 40 are mounted on the vehicle 10. For example, some of the functions of the image recognition device 40 may be mounted on the vehicle, and the rest may be disposed outside the vehicle.

[0080] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and the present invention also includes changes and modifications to the above-described embodiments that do not deviate from the spirit of the present invention. [Explanation of symbols]

[0081] 10: Vehicle 20: Visible camera 30: Time of Flight camera 40: Image recognition device 41: Image acquisition unit 42: Area identification part 43: Adjustment section 44: Sign recognition section

Claims

1. an image acquisition unit that acquires a first image which is a distance image and a second image which is a visible image; a region specifying unit that detects an object having a predetermined shape in the first image and specifies, as a region image, an image of a region in the second image corresponding to the detected object; and an adjustment unit that adjusts the contrast of the regional image based on a comparison result between the variance value of the regional image and a predetermined variance value.

2. A map information storage unit that stores map information; 2. The image recognition device according to claim 1, further comprising a selection unit that compares a first speed limit obtained by performing image recognition processing on the area image with a second speed limit obtained from the map information, and if the first speed limit and the second speed limit are different from each other, determines whether an accuracy of the image recognition processing is lower than a predetermined value, and selects the smaller speed limit of the first speed limit or the second speed limit if the accuracy is lower than the predetermined value, and selects the first speed limit if the accuracy is equal to or greater than the predetermined value.

3. The image recognition device according to claim 1, further comprising a learning unit that additionally learns the recognition model using the area image before contrast adjustment when the accuracy of image recognition processing performed using a recognition model for the area image after contrast adjustment is higher than a predetermined value and the variance value in the area image before contrast adjustment is smaller than the predetermined variance value.

4. 4. The image recognition device according to claim 1, wherein the predetermined shape includes a plurality of shapes, and the adjustment unit compares a variance value of the region image with a predetermined variance value that is set according to a shape of the object detected by the region identification unit.

5. acquiring a first image which is a range image and acquiring a second image from a second camera which is a visible image; Detecting an object having a predetermined shape in the first image, and identifying an image of a region in the second image corresponding to the detected object as a region image; The image recognition method includes adjusting the contrast of the region image based on a comparison result between the variance value of the region image and a predetermined variance value.

6. acquiring a first image which is a distance image and acquiring a second image which is a visible image; Detecting an object having a predetermined shape in the first image, and identifying an image of a region in the second image corresponding to the detected object as a region image; A program for causing a processor to execute a process of adjusting the contrast of the regional image based on a comparison result between the variance value of the regional image and a predetermined variance value.

Images