Obstacle detection device, obstacle detection method, and storage medium

Abstract

Provided are an obstacle detection device having a false detection prevention function, an obstacle detection method, and a storage medium. The obstacle detection device (100) of the present invention includes: a stereo camera (110L, 110R) that images left and right IR images; a point cloud data generation section (120) that generates point cloud data having 3-dimensional position information from the IR images imaged by the stereo camera (110L, 110R); an obstacle detection section (130) that detects an obstacle based on the point cloud data; an invalid region identification section (140) that identifies the presence or absence of an invalid region of the point cloud in the point cloud data; and an object recognition section (150) that recognizes a specific object in the invalid region of the point cloud based on the IR images. The obstacle detection section (130) does not determine the invalid region of the point cloud as an obstacle in a case where the specific object is recognized by the object recognition section (150).

Description

TECHNICAL FIELD

[0001] The present application relates to an obstacle detection device, and particularly to detection of an obstacle using a stereo camera. BACKGROUND

[0002] As a sensor that detects a distance to an object such as an obstacle, an infrared sensor or a millimeter wave radar is known. A stereo camera is one of such distance sensors that detects a distance to an object using a parallax of images captured by two cameras. For example, a vehicle obstacle detection device of Patent Literature 1 is provided with an infrared camera that captures an image of a front of a vehicle, and a stereo camera that captures an image of the front of the vehicle, determines a path based on an infrared image captured by the infrared camera, and detects an obstacle on the determined path based on a stereo image captured by the stereo camera.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2013-20543 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] In the conventional obstacle detection system, an infrared (IR) camera is sometimes used as a stereo camera. In an infrared (IR) image captured by the infrared camera, a region that reflects infrared light is captured as white, and a region that absorbs infrared light is captured as black. The region of white has high luminance, and the region of black has low luminance.

[0008] The obstacle detection system synthesizes left and right IR images captured by the stereo camera, and generates point cloud data that can represent a 3-dimensional position (or coordinates) of a captured object. A depth (a longitudinal depth) of the captured object is calculated using a parallax of the left and right IR images. The obstacle detection system determines presence or absence of an obstacle (including an obstacle on the ground, a hole, or a ditch) based on the generated point cloud data, detects an orientation of the obstacle, a distance to the obstacle, and a size of the obstacle in a case where there is an obstacle, and outputs a result of the detection. For example, in a case where the obstacle detection system is mounted on a transport machine such as an automobile, the information is used for warning the obstacle to an occupant, or is used for a purpose of automatically decelerating and stopping the transport machine.

[0009] However, in the conventional obstacle detection system, in a case where the dynamic range with respect to the brightness of the IR camera is insufficient with respect to the distribution of the brightness of the imaging subject, there is the following problem. If there is a region having a brightness of a maximum value / minimum value or close thereto in the IR image, the region is not sufficient data for indicating the characteristics of the imaging subject, so as a result, even if the left and right images are synthesized, an invalid region having no depth information occurs in the point cloud data.

[0010] For example, in a case where a manhole exists on a road, if the manhole is strongly reflected due to sunlight, as shown in FIG. 6A, the manhole is captured in a white circle in the central portion of the IR image, and the depth calculation of the manhole portion cannot be performed by matching the portion having characteristics in the left and right images. As a result, as shown in FIG. 6B, an invalid region in which the point cloud is missing occurs in a region corresponding to the manhole (in the drawing, a black portion). In addition, the point cloud data of the same figure shows a state in which the manhole is viewed from an oblique upper side. Figure 1 Figure 2

[0011] The invalid region of the point cloud has no distance information, so it cannot be determined whether the invalid region of the point cloud is an obstacle. However, in actual operation, it is determined that there is an obstacle in a case where the invalid region of the point cloud occurs for safety. Therefore, in a case where there is actually no obstacle in the invalid region of the point cloud, an obstacle is erroneously detected. For example, in a case where the obstacle detection system is mounted on a transport machine, an unnecessary warning is issued to a passenger, or an unnecessary deceleration or stop occurs.

[0012] The present application solves such a conventional problem, and aims to provide an obstacle detection device, an obstacle detection method, and an obstacle detection program having an erroneous detection prevention function.

[0013] Means for solving the problem

[0014] The obstacle detection device according to the present application includes a stereo camera, a generation means that generates point cloud data based on a stereo image captured by the stereo camera, an obstacle detection means that detects an obstacle based on the generated point cloud data, and a recognition means that recognizes a specific object by image recognition, and the obstacle detection means detects an obstacle using point cloud data after a region recognized as the specific object by the recognition means is removed.

[0015] ​​In one embodiment, the recognition means recognizes a specific object in a region corresponding to an invalid region of the point cloud included in the point cloud data. In one embodiment, the recognition means includes a recognition means that recognizes an invalid region of the point cloud included in the point cloud data. In one embodiment, the obstacle detection means does not determine the invalid region as an obstacle when the specific object is recognized by the recognition means. In one embodiment, the specific object is an object that does not become an obstacle, and the specific object is determined in advance. In one embodiment, the point cloud data is three-dimensional information including distance information obtained by synthesizing captured images of a stereo camera, and the invalid region of the point cloud is a region in which the distance information is missing. In one embodiment, the recognition means recognizes the specific object based on a captured image before the point cloud data is generated. In one embodiment, the recognition means recognizes the specific object based on an image captured by a camera different from the stereo camera and that is synchronized with the image of the stereo camera. In one embodiment, the stereo camera is an infrared camera. In one embodiment, the stereo camera captures the front of a mobile body.

[0016] Further, the obstacle detection device according to the present application includes a stereo camera, a generation means that generates point cloud data based on stereo images captured by the stereo camera, an obstacle detection means that detects an obstacle based on the generated point cloud data, and a recognition means that recognizes a specific object by image recognition, and the obstacle detection means determines whether to detect an obstacle using a recognition result of the recognition means based on brightness information of an image used by the recognition means. In one embodiment, the obstacle detection means detects an obstacle using only the point cloud data when the brightness information is below a threshold value, and detects an obstacle using the point cloud data and the recognition result of the recognition means when the brightness information exceeds the threshold value. In one embodiment, the obstacle detection means further determines whether to detect an obstacle using the recognition result of the recognition means based on a reliability of image recognition by the recognition means and the brightness information. In one embodiment, the obstacle detection means detects an obstacle using only the point cloud data when the brightness information is below a threshold value and the reliability is below a threshold value, and detects an obstacle using the point cloud data and the recognition result of the recognition means when at least one of the brightness information and the reliability exceeds a threshold value. In one embodiment, the threshold value of the brightness information is set for each specific object.

[0017] The mobile body according to the present application includes the obstacle detection device described above, and a control means that controls movement of the mobile body using a detection result of the obstacle detection device.

[0018] The obstacle detection method according to the present application uses a stereo image captured by a stereo camera to detect an obstacle, and includes a step of generating point cloud data including distance information of a captured object based on a stereo image captured by the stereo camera; a step of identifying the presence or absence of an invalid region of a point cloud included in the point cloud data; a step of identifying the presence or absence of a specific object in the invalid region of the point cloud; and a step of detecting an obstacle using the point cloud data after the invalid region of the point cloud is removed, in a case where the specific object is identified.

[0019] Further, the obstacle detection method according to the present application uses a stereo image captured by a stereo camera to detect an obstacle, and includes a step of generating point cloud data including distance information of a captured object based on a stereo image captured by the stereo camera; a step of identifying the presence or absence of an invalid region of a point cloud included in the point cloud data; a step of identifying the presence or absence of a specific object in the invalid region of the point cloud; and a step of determining whether to detect an obstacle using the identification result in the identifying step based on brightness information of an image used in the identifying step.

[0020] The obstacle detection program according to the present application is executed by an electronic device, and includes a step of generating point cloud data including distance information of a captured object based on a stereo image captured by the stereo camera; a step of identifying the presence or absence of an invalid region of a point cloud included in the point cloud data; a step of identifying the presence or absence of a specific object in the invalid region of the point cloud; and a step of detecting an obstacle using the point cloud data after the invalid region of the point cloud is removed, in a case where the specific object is identified.

[0021] Further, the obstacle detection program according to the present application is executed by an electronic device, and includes a step of generating point cloud data including distance information of a captured object based on a stereo image captured by the stereo camera; a step of identifying the presence or absence of an invalid region of a point cloud included in the point cloud data; a step of identifying the presence or absence of a specific object in the invalid region of the point cloud; and a step of determining whether to detect an obstacle using the identification result in the identifying step based on brightness information of an image used in the identifying step.

[0022] Effects of the Invention

[0023] According to the present application, the obstacle is detected using the point cloud data after the region in which the specific object is identified by the identifying means is removed, and thus it is possible to prevent false detection caused by the invalid region of the point cloud.

[0024] Further according to the present application, it is provided that whether to use the recognition result to detect the obstacle is determined based on the brightness information of the image used for recognizing the component, so as to prevent misrecognition of the image recognition and enable detection of the obstacle. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a diagram showing an example when a portion with high brightness is imaged in an IR image.

[0026] Figure 2 is a diagram showing a point cloud data obtained by synthesizing the IR stereo images shown in Figure 1 .

[0027] Figure 3 is a block diagram showing the structure of the obstacle detection device according to the embodiment of the present application.

[0028] Figure 4 is a diagram showing an example of installation of a stereo camera.

[0029] Figure 5 is a diagram showing an example of the method of identifying the invalid region performed by the invalid region identification unit.

[0030] Figure 6 is a flowchart showing the operation of the obstacle detection device according to the embodiment of the present application.

[0031] Figure 7 is a diagram showing the structure of the obstacle detection device according to the second embodiment of the present application.

[0032] Figure 8 is a diagram showing a platform (suspected pedestrian crossing) and point cloud data thereof.

[0033] Figure 9 is a bar chart showing the relationship between the reliability of image recognition and the normal detection rate (accuracy).

[0034] Figure 10 is a diagram showing the relationship between the sensitivity and the accuracy in image recognition when the threshold value is changed.

[0035] Figure 11 is a diagram showing the structure of the obstacle detection device according to the third embodiment of the present application.

[0036] Figure 12 is a flowchart showing the operation of the obstacle detection device according to the third embodiment of the present application.

[0037] Figure 13 is an example of an IR image of a pedestrian crossing.

[0038] Figure 14 is a diagram showing the point cloud data of Figure 13 .

[0039] Figure 15 is a flowchart showing the operation of the obstacle detection device according to the third embodiment of the present application.

[0040] Figure 16 is an example of an IR image of a well with orange lines.

[0041] Figure 17 is a graph showing point cloud data of Figure 16 .

[0042] Figure 18 is an example of an IR image of a well without orange lines.

[0043] Figure 19 is a graph showing point cloud data of Figure 17 .

[0044] Figure 20 is a block diagram showing the structure of a driving assistance system to which the obstacle detection device according to the embodiment of the present application is applied.

[0045] Explanation of Reference Signs

[0046] 100: obstacle detection device 110L, 110R: stereo camera

[0047] 120: point cloud data generation section 130: obstacle detection section

[0048] 140: invalid region identification section 150: object recognition section

[0049] 160: RGB camera DETAILED DESCRIPTION

[0050] The obstacle detection device according to the present application detects an obstacle using a stereo image captured by a stereo camera. In some embodiments, the obstacle detection device is mounted on a mobile body such as an automobile or a conveyance machine, and the detection result of the obstacle detection device is used for movement control of the mobile body or the like.

[0051] EMBODIMENT

[0052] Next, the embodiments of the present application will be described in detail with reference to the drawings. Figure 3is a block diagram showing the structure of an obstacle detection device according to an embodiment of the present application. The obstacle detection device 100 according to the present embodiment includes: IR cameras 110L, 110R (collectively referred to as a stereo camera 110) that image left and right IR images; a point cloud data generation section 120 that generates point cloud data after depth calculation by synthesizing the stereo images imaged by the stereo camera 110; an obstacle detection section 130 that detects an obstacle based on the point cloud data; an invalid region identification section 140 that identifies whether or not there is an invalid region of the point cloud in the point cloud data; and an object recognition section 150 that recognizes a specific object in the invalid region of the point cloud based on the IR images.

[0053] The structures of the point cloud data generation section 120, the invalid region identification section 140, the object recognition section 150, and the obstacle detection section 130 are implemented by hardware and / or software in a computer device or an electronic device. The hardware specifically includes a GPU (Graphics Processing Unit), a CPU, a storage medium such as a memory, and the software includes a program stored in a storage medium or the like or a program or an application downloaded from a server or the like via an application or a network.

[0054] The installation position of the stereo camera 110 is not particularly limited, but for example, as shown in Figure 4 , the stereo camera 110 is installed in the front of the vehicle M and images the front of the vehicle M. The IR cameras 110L, 110R use infrared rays, so imaging of an object is possible even at night. In addition, the IR cameras 110L, 110R sometimes irradiate a pattern or a design of infrared rays, but these infrared rays are light that is not visible to the human eye, so there is an advantage that discomfort is not felt. The IR cameras 110L, 110R generate a plurality of frames in one second, and the pixels constituting one frame express the imaged object in luminance, for example, in 256 gradations. The IR image imaged by the IR cameras is an image in which a region that reflects infrared rays is taken in white and a region that absorbs infrared rays is taken in black, and is a black-and-white gradation image. The number of pixels and the optical characteristics of the two IR cameras 110L, 110R are the same, and the coordinate positions of the IR cameras 110L, 110R are known. For example, the two IR cameras 110L, 110R are arranged at the same height (horizontally) at a certain distance apart.

[0055] The point cloud data generation section 120 acquires the IR images imaged by the IR cameras 110L, 110R. The point cloud data generation section 120 matches the left and right IR images, calculates the depth (distance) to the imaged object from the parallax, and finally generates point cloud data showing the 3-dimensional positions of the imaged object after synthesizing the left and right IR images, as shown in Figure 2 .

[0056] The invalid region identification unit 140 determines whether there are invalid regions in the point group data, that is, whether there are regions lacking distance information. If invalid regions exist, it identifies the location or size of the invalid regions. The invalid region identification unit 140 identifies invalid regions of point groups of a certain size or larger based on whether they are obstacles that would pose a driving hazard. For example, ... Figure 5 As shown, scan lines that do not contain a number of points exceeding a threshold within the intervals of scan lines (Y1, Y2, ..., Yn) in the Y direction (depth direction) are identified. In the example shown, scan lines Y3 to Y6, which scan the invalid region P of the point group, are identified. Next, within the region of the identified scan lines in the Y direction, scan lines that do not contain a number of points exceeding the threshold within the intervals of scan lines in the X direction (horizontal direction) are identified. In the example shown, scan lines X2 to X4 are identified. In this way, the region enclosed by scan lines Y3 to Y6 in the Y direction and scan lines X2 to X4 in the X direction is identified as an invalid region. The invalid region identification unit 140 provides the coordinate information of each corner of these four scan lines to the object identification unit 150. In addition, the method for identifying the invalid region P is not limited to the above, and other methods may be used.

[0057] If the object identification unit 150 acquires coordinate information of an invalid region representing a point group, it determines, based on the IR image captured by the IR camera 110L or 110R, whether the region and its surroundings constitute a specific object through image recognition, according to the features of the region corresponding to the invalid region of the point group and its surroundings. A specific object refers to an object that will not become an obstacle while the vehicle is in motion (e.g., wells, white lines, puddles, tracks, gratings). These specific objects may strongly reflect or absorb infrared radiation due to the intensity of sunlight, potentially exceeding the dynamic range of the brightness of the IR cameras 110L and 110R. The selection of which object to use as a specific object is predetermined.

[0058] The IR image used for image recognition is one aspect of the stereo image before it is synthesized from point group data. By using the stereo image for image recognition, the object recognition unit 150 can utilize the coordinate information of invalid regions representing point groups as is. The presence or absence of specific objects in and around these invalid regions is then identified.

[0059] In certain embodiments, the object recognition unit 150 determines the presence or absence of a specific object using AI (artificial intelligence). The specific object is, as described above, a well, a white line, a water puddle, a rail, a grating (cover), and the like, and features thereof are given to an image database as teacher data, and the image database is learned with respect to the features. The object recognition unit 150 performs recognition of the specific object in a region corresponding to the invalid region of the point cloud using the learned image database. In a case where the specific object is recognized, it is known that the invalid region of the point cloud is not a region that is dangerous to travel on (for example, a collapsed hole or a high difference that protrudes from the road, and the like). The recognition result of the object recognition unit 150 is provided to the obstacle detection unit 130.

[0060] The obstacle detection unit 130 determines the presence or absence of an obstacle on the basis of the point cloud data generated by the point cloud data generation unit 120, the recognition result of the invalid region recognition unit 140, and the recognition result recognized by the object recognition unit 150. In a case where there is an obstacle, the direction of the obstacle, the distance to the obstacle, and the size of the obstacle are detected.

[0061] In a case where there is an invalid region of a point cloud in the point cloud data and a specific object is recognized in the invalid region of the point cloud and in the vicinity thereof by the object recognition unit 150, the obstacle detection unit 130 determines that the invalid region of the point cloud is not an obstacle. In this case, the obstacle detection unit 130 determines the presence or absence of an obstacle with reference to the point cloud data after the invalid region is removed.

[0062] On the other hand, in a case where there is an invalid region of a point cloud in the point cloud data and a specific object is not recognized in the invalid region by the object recognition unit 150, the obstacle detection unit 130 determines that the invalid region of the point cloud is an obstacle. The detection result of the obstacle detection unit 130 is used for warning to an occupant of the vehicle M or driving assistance or automatic driving of the vehicle M.

[0063] Next, the operation of the obstacle detection device of the present embodiment is described with reference to the flowchart of Figure 6 First, the front of the vehicle is imaged by the stereo camera 110 (S100), and the imaged left and right IR images are provided to the point cloud data generation unit 120. The point cloud data generation unit 120 combines the stereo images and generates point cloud data that is 3-dimensional position information of the imaged object (S110). Next, the invalid region recognition unit 140 determines whether there is an invalid region of a point cloud in the generated point cloud data (S120). In a case where it is determined that there is an invalid region of a point cloud, coordinate information indicating the invalid region is provided to the object recognition unit 150, and the object recognition unit 150 performs recognition of a specific object in a region corresponding to the invalid region on the basis of the IR image of the IR camera 110L used in the generation of the point cloud data (S130).

[0064] In a case where the specific object is recognized by the object recognition section 150 (S140), the obstacle detection section 130 does not determine the invalid region of the point group as an obstacle (S150), and performs detection of an obstacle based on the point group data after the invalid region of the point group is removed (S160). On the other hand, in a case where the specific object is not recognized by the object recognition section 150, the obstacle detection section 130 determines the invalid region of the point group as an obstacle (S170).

[0065] Thus, according to the present embodiment, in a case where the invalid region of the point group exists, when the specific object is recognized in the region corresponding to the invalid region of the point group and the vicinity thereof based on the captured image before the point group data is synthesized, the invalid region of the point group is not determined as an obstacle, and thus, compared to a case where the invalid region of the point group is automatically determined as an obstacle, the false detection of an obstacle can be reduced. Further, according to the present embodiment, the performance of the obstacle detection device can be improved inexpensively without improving the performance (dynamic range or IR output) of the camera or adding other sensors.

[0066] In addition, in the present embodiment, an example in which the image recognition of the specific object is performed using the IR image before the point group data is synthesized is shown, but the image recognition using the IR image is merely for avoiding the false detection of an obstacle, and for complementing the function of the obstacle detection device. That is, the function of the obstacle detection device is difficult to be achieved only by the image recognition using the IR image. Because the image recognition can determine the presence or absence of an obstacle or the orientation of an obstacle with high accuracy, but it is difficult to determine the size of an obstacle or the distance to an obstacle with high accuracy.

[0067] Next, a second embodiment of the present application will be described. Figure 7 is a view showing the configuration of the obstacle detection device according to the second embodiment, and is the same as Figure 3 The same reference numerals are given to the same structures. In the present embodiment, an RGB camera 160 is newly included. The RGB camera 160 captures the front of the vehicle M, like the stereo camera 110, and provides the captured RGB color image to the object recognition section 150. In the previous embodiment, the object recognition section 150 performed the image recognition of the specific object using the IR image, but in the present embodiment, the object recognition section 150 performs the image recognition of the specific object using the color image captured by the RGB camera 160.

[0068] The RGB camera 160 is subjected to position adjustment or coordinate transformation of the captured RGB image so as to make the captured RGB image an image representing the same coordinate space as the IR image of the stereo camera 110. Further, the RGB image used by the object recognition unit 150 is synchronized with the IR image in which the invalid region of the point cloud is generated.

[0069] The IR image is a grayscale image representing the gradation of black and white, so the features of the object are limited compared to the RGB color image, and thus the accuracy of image recognition of the object is improved when using the RGB color image compared to the IR image. Further, since the RGB camera 160 is used differently from the stereo camera 110, the number of pixels or optical characteristics of the RGB camera 160 are not constrained by the stereo camera 110, and the RGB camera 160 is also able to capture a high-quality color image compared to the stereo camera 110.

[0070] Thus, according to the present embodiment, the image recognition of the specific object is performed using the RGB image in the case where the invalid region of the point cloud is recognized, and thus the accuracy of image recognition of the specific object is improved, and the false detection of the obstacle when the invalid region of the point cloud is generated can be reduced.

[0071] In addition, in the above-described embodiment, an example in which an IR camera is used as the stereo camera is shown, but the present application is not limited thereto, and a camera that captures a color image by visible light can also be used as the stereo camera. Further, in the above-described embodiment, the invalid region of the point cloud is assumed to be rectangular, but is not limited thereto, and the invalid region of the point cloud can also be recognized in a circular shape or a polygonal shape. Further, in terms of the positional relationship between the invalid region of the point cloud and the recognized specific object, not only the case where the two are completely overlapping, but also the case where the two are in a substantially overlapping relationship can also be considered as the specific object being recognized in the region in the vicinity of the invalid region.

[0072] Next, a third embodiment of the present application will be described. The first and second embodiments described above perform image recognition in order to compensate for the depth-lacking region (invalid region of the point cloud) generated when the dynamic range of the camera is insufficient with respect to the brightness of the captured object, but in AI-based image recognition, the recognition reliability calculated in the recognition process is used, and if the reliability is higher than a threshold value set in advance, the recognition result can be adopted as a correct recognition result. However, in such AI-based image recognition, if the appearance is similar, a non-obstacle can be mistakenly recognized as an obstacle or an obstacle can be mistakenly recognized as a non-obstacle.

[0073] The invalid region of the point cloud is likely to occur in an object that is consistently high in brightness such as a white line or a crosswalk, and for example, sometimes a non-obstacle is mistakenly recognized as an obstacle or an obstacle is mistakenly recognized as a non-obstacle depending on the appearance on the IR image. Figure 8The pallet (obstacle of a suspected crosswalk) that forms a height difference between the white lines is misrecognized as a crosswalk, and the groove of the pallet cannot be detected as an obstacle. In the case of the pallet, the white line portion is at approximately the same height as the road surface, and the groove portion between the white lines is a lower portion than the road surface. In the case of a high-brightness pallet, an invalid region occurs in the point cloud data due to the high brightness of the white line portion, but the groove portion is not high-brightness, so the point cloud is generated, that is, the height difference of the groove portion can be located.

[0074] It is considered that the above-described event can be solved by improving the accuracy of image recognition, but on the other hand, there is a concern that the sensitivity will decrease. For the improvement of accuracy, the following two methods are generally considered.

[0075] (1) In AI image recognition, data is increased and re-learned (for example, Figure 8 the pallet (suspected crosswalk) is not a crosswalk” such a label).

[0076] (2) The threshold value of the reliability of image recognition is increased to make the judgment criterion strict.

[0077] However, if the learning Figure 8 the suspected crosswalk, it is not known what features to focus on to judge it as a non-crosswalk, so it is difficult to predict the impact on the recognition results of other objects, and there is a concern that a part of the crosswalks will not be recognized.

[0078] Figure 9 is a bar graph showing the relationship between each reliability band and the recognition accuracy of the crosswalk, Figure 10 is a graph showing the relationship between the sensitivity and the accuracy when the reliability threshold value is changed. As Figure 9 indicated, even if the reliability of image recognition is low, the recognition accuracy of the crosswalk hardly changes, so in other words, even if the threshold value of the reliability is increased, the recognition accuracy of the crosswalk does not have a large impact. On the other hand, as Figure 10 indicated, if the threshold value of the reliability is increased, the accuracy is increased and moves to the right direction of the graph, and the sensitivity greatly decreases. In this case, the crosswalk that becomes dark due to the decrease in sensitivity is excluded from the recognition object, which becomes a problem.

[0079] Thus, there are many technical problems for the improvement of the accuracy of image recognition, so in the third embodiment, instead of improving the accuracy of image recognition as described in (1) or (2), it is made possible to accurately detect an obstacle such as Figure 8 the pallet (suspected crosswalk) that can be measured in distance but is easily misrecognized in image recognition.

[0080] Figure 11is a block diagram showing the structure of the obstacle detection device according to the third embodiment of the present application, in which the same reference numerals are given to the same structures as those of the first embodiment shown in FIG. 1. In the third embodiment, the object recognition section 150A includes an image recognition section 152 that performs image recognition of a specific object in the invalid region using the IR image, a reliability provision section 154 that provides reliability of the image recognition performed by the image recognition section 152, and a brightness provision section 156 that provides brightness of the IR image used for the image recognition. Figure 3

[0081] The image recognition section 152 performs recognition of whether the IR image captured by the IR camera 110L corresponds to a specific object using a database in which features of the specific object are learned. The specific object is, for example, a pedestrian crossing, a manhole, a white line, a rail, a grating (cover), or the like. The database stores learning data in which a teacher label indicating a relationship between a feature of image data and the image data being that of the specific object is learned, for each specific object. The database includes, for example, learning data related to a pedestrian crossing, learning data related to a manhole, learning data related to a white line, learning data related to a rail, and learning data related to a grating, respectively. The image recognition section 152 performs image recognition of the specific object using the IR image of the IR camera 110L and the database when the invalid region of the point group is recognized, as in the first embodiment.

[0082] The reliability provision section 154 provides reliability of the image recognition of the specific object to the obstacle detection section 130A when the image recognition of the specific object is performed by the image recognition section 152. For example, when the image recognition of a manhole is performed by the image recognition section 152, the reliability provision section 154 acquires reliability of the image recognition of the manhole. If the proportion of the features of the manhole obtained from the IR image coinciding with the features of the manhole on the database is high, the reliability is also made higher in proportion thereto.

[0083] The brightness provision section 156 calculates average brightness of the IR image used by the image recognition section 152, and provides the calculated average brightness to the obstacle detection section 130A. The average brightness of the IR image can be average brightness of the entire IR image, or average brightness of a partial region of the IR image used for the image recognition. In addition, the brightness provision section 156 can calculate average brightness of pixels at intervals, in addition to average brightness of all pixels of the IR image or the partial region, for example. Further, the IR image is a grayscale image indicating a gradation of black and white, and the brightness is a value of the grayscale. Furthermore, the brightness provision section 156 can be configured to calculate an average value of brightness distributed around the middle of the maximum brightness and the minimum brightness of the IR image.

[0084] ​The obstacle detection unit 130A determines whether or not to use the recognition result of the object recognition unit 150A for the judgment of the obstacle, based on the reliability from the reliability providing unit 154 and the brightness from the brightness providing unit 156.

[0085] Figure 12 is a flowchart illustrating the operation of the obstacle detection device of the present embodiment. The obstacle detection unit 130A determines whether or not the reliability provided from the reliability providing unit 154 is below a threshold value, if the recognition result from the object recognition unit 150A is acquired (S200). The threshold value is predetermined for each specific object, and is, for example, 0.6 (60%). Further, the obstacle detection unit 130A determines whether or not the recognition result is below the threshold value each time the image recognition result is acquired from the image recognition unit 152.

[0086] In the case where the reliability is below the threshold value, it is further determined whether or not the average brightness of the IR image provided from the brightness providing unit 156 is below a threshold value (S220). The threshold value is predetermined for each specific object, and is, for example, 60 when 1 pixel is 8-bit data (maximum brightness is 256). If the reliability and the average brightness are below the threshold values, respectively, the obstacle detection unit 130A does not use the recognition result of the object recognition unit 150A, and judges the obstacle only by the ranging result (point cloud data) (S230). That is, the reliability is low, and the result of the image recognition is more likely to be incorrect compared to the case where the reliability is high, and the average brightness is low, and there is a possibility that there is no point cloud invalid region, and if the recognition result of the image recognition unit 152 is used, there is a concern that the specific object is misrecognized.

[0087] On the other hand, in the case where the reliability is greater than the threshold value or the average brightness is greater than the threshold value, the obstacle detection unit 130A judges the obstacle using the recognition result of the object recognition unit 150A (S240). That is, the reliability is high, and the result of the image recognition is correct, and the average brightness is high, and there is a possibility that there is a point cloud invalid region, and in this case, the recognition result of the image recognition unit 152 is used in order to compensate for the invalid region of the point cloud data, and the detection accuracy of the obstacle is more likely to be improved.

[0088] Figure 13 is an example of an IR image of a pedestrian crossing, Figure 14 is an example of point cloud data generated using Figure 13 the IR image. Figure 13 The average brightness of the IR image of Figure 14As shown, a confusion of distance measurement is generated on a part thereof, and a part of the point group is missing. In such a case, even if the brightness is low, the reliability is high, and thus the judgment of the obstacle is made using the recognition result of the object recognition unit 150A.

[0089] Next, a flowchart of a modification of the third embodiment of the present application is shown in Figure 15 In this example, the obstacle detection unit 130A does not use the reliability, but only uses the average brightness of the IR image to determine whether or not to use the recognition result. Other than this, the flowchart is the same as that of Figure 12 .

[0090] Figure 16 is an example of an IR image of a well in which an orange circular line is formed around the well, and the average brightness of the region including the line is 128. The point group data is shown in Figure 17 As shown, a circular-shaped depth deficiency is generated on the position corresponding to the line. On the other hand, Figure 18 is an example of an IR image of a well in which there is no line like that shown in Figure 16 . The average brightness is 72, and the point group data is shown in Figure 19 As shown, no depth deficiency is generated.

[0091] If the threshold value of the average brightness is set to 60 as in the case of the crosswalk, a darker hole on the road will be misrecognized as Figure 18 a well like that. Therefore, in the case where the recognition result is a well, the threshold value of the average brightness is set to 80, and the recognition result when the average brightness is 80 or more is used to judge the obstacle. Thus, in the case where a well like that shown in Figure 16 is photographed, since the average brightness is higher than the threshold value, the obstacle is judged using the image recognition result of the well. On the other hand, in the case where a well like that shown in Figure 18 is photographed, in which the average brightness is lower than the threshold value, the obstacle is judged using Figure 19 the point group data shown in , and thus, the misrecognition of the well can be prevented.

[0092] According to the present embodiment, the recognition accuracy of the suspected crosswalk shown in Figure 8 has been improved by about 96% compared to the past. Further, in the usual actual driving in which the suspected crosswalk is not photographed, the change rate of the recognition sensitivity of the crosswalk also stays within 2%, and is not substantially changed. The suspected crosswalk can be detected as an obstacle without changing the sensitivity, and thus, like the past, the misdetection of the dark crosswalk photographed like that shown in Figure 13 can be prevented and the detection sensitivity of the obstacle can be improved.

[0093] The third embodiment described above can also be applied to the second embodiment. In this case, it is determined whether to use the recognition result of the image recognition unit 152 for obstacle detection based on the brightness information calculated from the RGB image captured by the RGB camera.

[0094] Next, a driving assistance system that utilizes the obstacle detection device of this embodiment, such as... Figure 20 As shown. The driving assistance system 200 includes the obstacle detection device 100 / 100A / 100B described in the previous first or third embodiments, a warning unit 210 that warns the occupants, and a driving assistance module 220 that assists in driving the vehicle M. The warning unit 210 includes, for example, a display unit or an audio output unit, and notifies the presence of an obstacle by means of image information or audio information when an obstacle is detected in front of the vehicle M. Furthermore, the driving assistance module 220 cooperates with the driving control of the vehicle M to slow down the vehicle M or change the direction of travel of the vehicle M to avoid the obstacle in order to prevent a collision with the obstacle.

[0095] Furthermore, the obstacle detection device of this embodiment can also be applied to conveying machines or mobile bodies other than vehicles.

[0096] The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to specific embodiments. Various modifications and alterations can be made within the scope of the spirit of the invention as set forth in the claims.

Claims

1. An obstacle detection device, comprising: 3D camera; The generation component generates point group data based on the stereo image captured by the stereo camera; The obstacle detection component detects obstacles based on the generated point cluster data; The invalid region identification component identifies whether there are invalid regions in the point group data. as well as The identification component identifies specific objects within invalid regions of the point group through image recognition. If the obstacle detection component identifies a specific object in the invalid area of ​​the point group by the identification component, it will not determine the invalid area of ​​the point group as an obstacle, but will use the point group data after removing the invalid area to detect the obstacle. If the identification component does not identify a specific object in the invalid area of ​​the point group, it will determine the invalid area of ​​the point group as an obstacle.

2. The obstacle detection device as described in claim 1, The specific object is one that will not become an obstacle, and the specific object is predetermined.

3. The obstacle detection device as described in claim 1, The point cluster data is 3D information containing distance information obtained by synthesizing images from a stereo camera. Invalid regions of the point cluster are regions lacking distance information.

4. The obstacle detection device as described in claim 1, The identification component identifies specific objects based on the camera image before generating the point group data.

5. The obstacle detection device as described in claim 1, The identification component identifies a specific object based on an image captured by a camera different from the stereo camera and synchronized with the image from the stereo camera.

6. The obstacle detection device as described in claim 1, The stereo camera is an infrared camera.

7. The obstacle detection device as described in claim 1, The stereo camera captures images of the front of the moving object.

8. The obstacle detection device as described in claim 1, The obstacle detection component determines whether to use the recognition result of the recognition component to detect obstacles based on the brightness information of the image used by the recognition component.

9. The obstacle detection device as described in claim 8, When the brightness information is below a threshold, the obstacle detection component uses only the point cluster data to detect obstacles; when the brightness information exceeds the threshold, it uses both the point cluster data and the recognition result of the recognition component to detect obstacles.

10. The obstacle detection device as described in claim 8, The obstacle detection component then determines whether to use the recognition result of the recognition component to detect obstacles based on the reliability of the image recognition performed by the recognition component and the brightness information.

11. The obstacle detection device as described in claim 10, When the brightness information and the confidence level are both below a threshold, the obstacle detection component uses only the point cluster data to detect obstacles. When at least one of the brightness information and the confidence level exceeds a threshold, the component uses the point cluster data and the recognition result of the recognition component to detect obstacles.

12. The obstacle detection device as described in claim 11, The threshold for the brightness information is set separately for each specific object.

13. An obstacle detection method, comprising detecting obstacles using stereo images captured by a stereo camera, including: The step of generating point group data containing distance information of the photographed object based on the stereo image captured by the stereo camera; The step of identifying the presence or absence of invalid regions in the point group data; The step of identifying the presence or absence of a specific object in the invalid region of a point group; as well as If a specific object is identified in the invalid region of the point group, the invalid region of the point group is not determined as an obstacle, and the point group data after removing the invalid region of the point group is used to detect obstacles. If the specific object is not identified in the invalid region of the point group, the invalid region of the point group is determined as an obstacle.

14. The obstacle detection method as described in claim 13, further comprising: The step of determining whether to use the identification result in the identification step to detect obstacles, based on the brightness information of the image used in the identification step.

15. A storage medium storing an obstacle detection program executed by an electronic device, the obstacle detection program comprising: The steps of generating point group data containing distance information of the photographed objects based on stereo images taken by a stereo camera; The step of identifying the presence or absence of invalid regions in the point group data; The step of identifying the presence or absence of a specific object in the invalid region of a point group; as well as If a specific object is identified in the invalid region of the point group, the invalid region of the point group is not determined as an obstacle, and the point group data after removing the invalid region of the point group is used to detect obstacles. If the specific object is not identified in the invalid region of the point group, the invalid region of the point group is determined as an obstacle.

16. The storage medium of claim 15, wherein the obstacle detection program further comprises: The step of determining whether to use the identification result in the identification step to detect obstacles, based on the brightness information of the image used in the identification step.

Images