IMAGE RECOGNITION DEVICE AND IMAGE RECOGNITION METHOD

Abstract

Targets that should be considered the same object in a three-dimensional space may be detected as a plurality of objects due to the influence of the external environment, characteristics of the targets, restrictions on the detection parameters, etc. Provided is an image recognition device comprising: a region detection unit that detects a plurality of detection regions from an image; a temporary connection processing unit that temporarily connects the plurality of detection regions; an object identity determination processing unit that determines whether the temporarily connected plurality of detection regions can be identified as a single object; and a detection region integration processing unit that, on the basis of the result of the determination by the object identity determination processing unit, determines a combination of detection regions among the plurality of detection regions that should be recognized as a single object, and reflects this determination in the object detection result.

Description

[Technical field]

[0001] The present invention relates to an image recognition device and an image recognition method for recognizing a three-dimensional object in an image captured by a camera. [Background technology]

[0002] In recent years, there has been an increasing demand for image recognition devices used in driving assistance systems, autonomous driving systems, etc. to improve their three-dimensional object detection performance. For example, in the case of pedestrian collision safety functions, nighttime pedestrian collision safety tests have been added to automobile assessments, and performance improvements are required. To achieve this, high recognition performance for three-dimensional objects is required.

[0003] Here, the abstract of Patent Document 1 states that the problem is to provide an object identification device that can identify the type of a target object even when the object is stationary, and can improve the identification performance of objects that look similar, and states that the solution is to include an object detection unit 1 that detects an object region where an object exists from an image acquired from a camera 9, an object distance information calculation unit 3 that calculates a distance variance indicating the distance distribution of the object region, and an object identification unit 4 that identifies the type of the object based on the distance variance indicating the distance distribution.

[0004] Furthermore, paragraph 0021 of the same document states, "The object detection unit 1 is a means for detecting an object region from an image acquired from the camera 9. This process detects a block of rigid body regions from an image, and various means can be used. For example, when a stereo camera is used to acquire an image, the distance on the image can be found from the parallax. The object region can be found by grouping adjacent regions that are close to each other on the image."

[0005] Thus, in Patent Document 1, in the case of a stereo camera that detects an object based on the parallax of images obtained from two cameras, the obtained parallax is grouped to detect an object in a three-dimensional space. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-81789 Summary of the Invention [Problem to be solved by the invention]

[0007] However, with conventional devices that detect objects from images, if the texture of an object cannot be obtained, there is a possibility that a single object may be detected as multiple objects, or multiple objects may be detected as a single object.

[0008] For example, in the case of detecting an object using disparity as in Patent Document 1, in areas of an image where the change in features such as edges and colors is extremely small, the difference in the change in features between the area where disparity is to be found and its surrounding areas becomes extremely small, which causes a problem that the correct disparity cannot be obtained.

[0009] In disparity grouping, the threshold for extracting invalid areas where disparity cannot be obtained is optimized according to the application used and the object to be detected, but the relationship between the size of an object in real space and the invalid area is not constant. For this reason, for example, if a grouping threshold for detecting a person is set, an object larger than a person, such as a car, may be divided into multiple objects. On the other hand, if a threshold is set according to a large object such as a car, a small object such as a person or a pole on the road may be merged with surrounding objects and acquired.

[0010] When such an event occurs in which a single object is detected as multiple objects, or multiple objects are detected as a single object, it becomes difficult to determine the type of object and to accurately determine its position, which hinders measurement of the object's movement, such as its speed.

[0011] Therefore, in order to properly realize driving assistance and autonomous driving, it is necessary to properly detect moving objects such as vehicles and pedestrians that are the subject of monitoring as independent single objects.

[0012] Therefore, an object of the present invention is to provide an image recognition device and an image recognition method that can appropriately detect three-dimensional objects such as vehicles and pedestrians that are subject to monitoring in driving assistance systems, autonomous driving systems, etc., as independent, single objects. [Means for solving the problem]

[0013] In order to solve the above problems, the image recognition device of the present invention is an image recognition device having an area detection unit that detects multiple detection areas from an image, a temporary combination processing unit that temporarily combines the multiple detection areas, a same object determination processing unit that determines whether the temporarily combined multiple detection areas can be identified as a single object, and a detection area integration processing unit that determines a combination of the multiple detection areas that should be recognized as a single object based on the determination result of the same object determination processing unit, and reflects this in the object detection result. Effect of the Invention

[0014] According to the image recognition device and image recognition method of the present invention, three-dimensional objects such as vehicles and pedestrians that are the subject of monitoring in driving assistance systems, autonomous driving systems, etc. can be appropriately detected as independent, single objects. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram showing an overall configuration of an image recognition apparatus according to a first embodiment. [Diagram 2] 4 is a flowchart showing the operation of the image recognition apparatus according to the first embodiment. [Diagram 3] 13 is an example of a detection area set on an image by detection processing. [Figure 4] 3 is a flowchart showing the processing operation of the detection process in FIG. 2; [Diagram 5] 4 is an example of a temporary joining process performed by the image recognition device of the first embodiment. [Figure 6] 4 is an example of a temporary joining process performed by the image recognition device of the first embodiment. [Figure 7] 4 is an example of a temporary joining process performed by the image recognition device of the first embodiment. [Figure 8] 4 is an example of a temporary joining process performed by the image recognition device of the first embodiment. [Figure 9] 4 is an example of a detection area set by the image recognition device of the first embodiment. [Figure 10] 4 is an example of a small area set by the image recognition device according to the first embodiment. [Figure 11] 4 is an example of a detection area set by the image recognition device of the first embodiment. [Figure 12] 4 is an example of a small area set by the image recognition device according to the first embodiment. [Figure 13] A graph plotting the lateral position and depth distance of each object in the image in Figure 11. [Figure 14] An example of pattern matching with a model pattern. [Figure 15] An example of how to extract suitable tentative bond regions. [Figure 16] 11 is a flowchart showing a processing operation of an image recognition apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, the image recognition device of the present invention will be described with reference to the drawings. EXAMPLES

[0017] First, an image recognition device 1 according to a first embodiment of the present invention will be described with reference to FIGS.

[0018] FIG. 1 is a block diagram showing the overall configuration of an image recognition device 1 of this embodiment. This image recognition device 1 is a stereo camera mounted on a vehicle (hereinafter referred to as "own vehicle V0"), and is responsible for the environment recognition function of a driving assistance system or an automatic driving system. The environment recognition function is a function for recognizing three-dimensional objects such as pedestrians, vehicles, traffic signals, signs, white lines, vehicle taillights, and headlights from a captured image in front of the vehicle. As a result, the driving assistance system or automatic driving system of the own vehicle V0 can control the brakes, steering, etc. based on the recognition result by the image recognition device 1, and perform driving assistance control or automatic driving control according to the environment outside the vehicle.

[0019] 1, the image recognition device 1 of this embodiment includes a left camera 11, a right camera 12, an image input interface 13, an image processing unit 14, a calculation processing unit 15, a storage unit 16, a CAN interface 17, and a control processing unit 18. The image recognition device 1 is also connected to a steering system, a drive system, a braking system, and the like of a host vehicle V0 (not shown) via a CAN (Controller Area Network).

[0020] In addition, the components of the image recognition device 1 other than the left camera 11 and the right camera 12 are a computer equipped with hardware such as a calculation device such as a CPU, a storage device such as a semiconductor memory, and a communication device, or a hard logic circuit. The calculation device executes a program deployed in the storage device or executes control incorporated in the hard logic circuit to realize each function of the image processing unit 14 and the calculation processing unit 15 described later, but below each part of the image recognition device 1 will be described in detail while omitting such well-known techniques as appropriate.

[0021] The left camera 11 is a camera that captures a left image P1 using an imaging element, and the right camera 12 is a camera that captures a right image P2 using an imaging element. The left camera 11 and the right camera 12 are installed on the upper part of the inner surface of the windshield or the like so as to capture an image of the area ahead of the vehicle V0, and the two cameras capture images simultaneously to form a stereo camera that captures a pair of synchronized images P (left image P1, right image P2).

[0022] The image input interface 13 takes in a left image P1 captured by the left camera 11 and a right image P2 captured by the right camera 12.

[0023] The internal bus 19 is a bus that relays communications between the image input interface 13, the image processing unit 14, the calculation processing unit 15, the memory unit 16, the CAN interface 17, and the control processing unit 18, and transmits the image P (P1, P2) captured by the image input interface 13 to the image processing unit 14 and the calculation processing unit 15, and transmits the outputs of both processing units to the memory unit 16.

[0024] The image processing unit 14 compares the left image P1 and the right image P2 captured by the image input interface 13, performs image correction such as correction of device-specific deviations caused by the imaging element and noise interpolation on each image P, and stores the results in the storage unit 16. The image processing unit 14 also calculates corresponding points between the left image P1 and the right image P2 to obtain parallax information, and stores this in the storage unit 16 as distance information corresponding to each pixel on the image. Note that, hereinafter, various processes are basically performed using the distance information, but various processes may be performed using the parallax information instead of the distance information.

[0025] The arithmetic processing unit 15 uses the image information and distance information (parallax information) stored in the memory unit 16 to recognize three-dimensional objects in order to grasp the environment around the vehicle, and stores the recognition results and part of the intermediate processing results in the memory unit 16. After recognizing three-dimensional objects in the image P, the arithmetic processing unit 15 performs calculations for vehicle control using the recognition results. The vehicle control policy obtained as a result of the vehicle control calculations and part of the recognition results are transmitted to the in-vehicle network CAN via the CAN interface 17, and the vehicle is controlled accordingly.

[0026] The control processing unit 18 monitors each processing unit to see if it is operating abnormally or if an error has occurred during data transfer, and prevents abnormal operation.

[0027] <Operation of image recognition device 1> Next, the operation of the image recognition device 1 having the above configuration will be outlined with reference to the flowchart of FIG.

[0028] [Step S1: Image processing] First, in step S1, the image processing unit 14 executes image processing. Specifically, the image processing unit 14 performs processing such as correction to absorb the inherent characteristics of the imaging element for each of the left image P1 and right image P2 captured by the image input interface 13, and outputs a left image P1' and a right image P2'. The corrected images P' (left image P1', right image P2') are stored in the image buffer B1 in the storage unit 16.

[0029] [Step S2: Parallax processing] Next, in step S2, the image processing unit 14 executes parallax processing. Specifically, the left and right corrected images P' (P1', P2') corrected in step S1 are used to compare the corrected images, thereby obtaining parallax information of the images obtained by the left camera 11 and the right camera 12. From the parallax between the left and right images, a point of interest on the image of a three-dimensional object is obtained as distance information to the three-dimensional object according to the principle of triangulation. This processing result (parallax information, distance information) is stored in the parallax buffer B2 in the storage unit 16.

[0030] [Step S3: Detection process] In step S3, the calculation processing unit 15 detects a three-dimensional object in three-dimensional space using the parallax information obtained in step S2. This detection result (detection area of ​​the three-dimensional object) is stored in the detection area buffer B3 in the storage unit 16. FIG. 3 is a diagram illustrating detection areas R1 to R4 of the three-dimensional object provisionally set on the image after the processing of step S2. In the following, the right direction of the image is the positive direction of the X axis, and the upward direction is the positive direction of the Y axis. In addition, the shape of the detection area on the image may be an indefinite shape, but in the following, the detection area is assumed to be rectangular in order to facilitate the calculation of the provisional combination process described later.

[0031] In Fig. 3, detection area R1 correctly detects a pedestrian as a single object, and detection area R2 correctly detects a preceding vehicle as a single object. On the other hand, detection area R3 erroneously detects the front part of vehicle V1, which is a single object, as an independent object, and detection area R4 erroneously detects the rear part of vehicle V1, which is a single object, as an independent object.

[0032] The following are some possible reasons why a single object, a vehicle V1, may be erroneously detected as being divided into detection areas R3 and R4. (1) If there is insufficient texture information on the side of the vehicle V1, disparity cannot be obtained in that area, and the side of the vehicle V1 cannot be detected as a whole, but may be detected as a divided part. (2) When the detection parameters for three-dimensional objects are inappropriately set, erroneous region segmentation may occur even if there is sufficient texture information on the side of the vehicle V1. For example, this may occur when the detection parameters are set to detect a part on the side of the vehicle V1 that has a depth distance as a separate three-dimensional object, or when the vehicle V1 is detected using detection parameters for detecting pedestrians, the front and rear of the vehicle V1 are erroneously detected as separate objects.

[0033] In such a case, in step S3, the separately detected detection areas R3 and R4 are combined into a single detection area of ​​a three-dimensional object, and the details of this combining process will be described later.

[0034] [Step S4: Recognition processing] In step S4, the arithmetic processing unit 15 performs a recognition process to identify the type of three-dimensional object for the detection area set on the image in step S3. As a result of this process, it is identified whether the three-dimensional object in each detection area corresponds to a pedestrian, a vehicle, a traffic light, a sign, a white line, a tail light or a head light of a car, etc. This recognition process is performed using the image information recorded in the image buffer B1 and the parallax information recorded in the parallax buffer B2. This is also the case when a radar such as a millimeter wave and an image sensor such as a camera are combined instead of the stereo camera.

[0035] [Step S5: Vehicle control processing] In step S5, the calculation processing unit 15 takes into consideration the result of the recognition of the three-dimensional object in step S4 and the state of the host vehicle V0 (speed, steering angle, etc.), and, for example, issues a warning to the occupants and performs control such as braking and steering angle adjustment of the host vehicle V0. Alternatively, it determines avoidance control for the recognized three-dimensional object, and outputs the result as automatic control information via the CAN interface 17. This makes it possible to realize a desired driving assistance system, an automatic driving system, etc.

[0036] <Details of step S3> Next, the details of step S3 executed by the arithmetic processing unit 15 will be described with reference to the flowchart of FIG.

[0037] [Step S3a: Area detection process] In step S3a, the calculation processing unit 15 detects the detection area of ​​the three-dimensional object using the parallax information obtained in step S2, and stores the detected detection area in the detection area buffer B3. As a result of this process, a detection area such as that shown in Fig. 3 is provisionally set on the image. As described above, at this point, a single object may be erroneously detected as multiple three-dimensional objects (see detection areas R3 and R4 in Fig. 3), so that the single object can be accurately detected as a single object by the subsequent process.

[0038] [Step S3b: Temporary joining process] In step S3b, the calculation processing unit 15 combines the multiple detection regions R stored in the detection region buffer B3 to create a rectangular temporary combined region C. In the following, it is assumed that a rectangle inscribed in the multiple rectangular detection regions is created as one temporary combined region C.

[0039] As a result of the processing in step S3a, a large number of detection areas may be set on the image. However, performing the temporary combination process for all combinations of detection areas is not realistic because the calculation cost becomes huge when combined with the subsequent processing. Therefore, in this step, in order to reduce the load of this combination process, the temporary combination process is performed in the following procedure.

[0040] First, if any two detection areas R recorded in the detection area buffer B3 are adjacent on the image, a tentative combined area C is created using those two adjacent detection areas R. Here, adjacent on the image refers to the relationship between the detection areas whose lateral positions (X positions) are closest to each other. Note that, although adjacent detection areas are identified below using the X position as the reference, adjacent detection areas may also be identified using the Y position as the reference. Also, both the X and Y positions may be used as the reference.

[0041] In the example of FIG. 5, there are detection areas R1 to R4, but the detection area R with the closest X position to detection area R1 is detection area R2, so these two detection areas R1 and R2 are set as targets for temporary joining, and a temporary joining area C1 shown by a dashed line is created. Furthermore, since detection area R2 is adjacent to detection area R3, a temporary joining area C2 shown by a dashed line is also created. In this manner, a temporary joining area C is created between adjacent detection areas. In other words, a temporary joining area C is not created between non-adjacent detection areas (for example, detection areas R1 and R4), and therefore the calculation load of the temporary joining process can be reduced accordingly.

[0042] In addition, in this step, not only the detection regions R are tentatively joined, but also the detection region R and the tentative joined region C may be tentatively joined. For example, in the tentative joined region C1 in Fig. 6, excluding the detection regions R1 and R2 used in the element, the detection region R3 is the closest detection region in the horizontal position (X position). Therefore, the tentative joined region C1 and the adjacent detection region R3 may be tentatively joined to create the tentative joined region C3.

[0043] The tentative joining area C3 in Fig. 6 includes the left vehicle and the front of the right vehicle, so it may seem like an undesirable joining area for recognizing a single three-dimensional object, but there are cases where such a tentative joining is necessary. An example where such a tentative joining is necessary is shown in Fig. 7.

[0044] In step S3a, the detection area is set so that a standard-sized object can be detected, but an object larger than a standard-sized vehicle (specifically, a standard-sized car), such as a large truck, may be divided into multiple detection areas and detected. The detection areas R1, R2, and R3 in FIG. 7 are areas in which a part of a large truck is erroneously detected as one detection area. In such a case, the tentative combined area in which the detection areas R1 and R2 are combined, or the tentative combined area in which R2 and R3 are combined, does not become a frame in which the large truck is correctly detected. Therefore, in such a case, the tentative combined area C1, which is a single detection area for the large truck, can be obtained by combining the remaining detection areas with the tentative combined area of ​​the detection areas R1 and R2, or the tentative combined area of ​​the detection areas R2 and R3.

[0045] Furthermore, the combination of temporary combinations in the case where the detection areas included in the detection area buffer B3 also have information in three-dimensional space such as parallax information and distance information will be described. FIG. 8 is an example of detection on a public road. The detection areas R1 to R5 each have a depth distance as shown in the figure. At this time, the objects for which the temporary combination area C is created are limited to only those whose distance from the detection area is within 10%, for example. By defining such conditions, objects with significantly different distances such as the detection areas R1, R2, and R3 are not created as temporary combination areas, and the calculation cost of the temporary combination process can be reduced by combining the detection areas R4 and R5, which are close in distance, to create only the temporary combination area C1. In addition, when the detection areas have information on the horizontal distance or Euclidean distance in the real space, it is possible to prevent the temporary combination of detection areas with significantly different horizontal distances or Euclidean distances based on the information. This makes it possible to prevent the creation of a temporary combination area that includes both the detection area R2 and the detection area R3 in FIG. 5, which have significantly different horizontal distances. The threshold for creating the provisional region is not limited to the 10% mentioned above, but may be a fixed value or may be dynamically determined in consideration of the accuracy of the sensor in distance. In addition, the combination based on the distance may be combined with the condition of adjacent areas on the image. This makes it possible to reduce calculation costs.

[0046] Furthermore, in step S3a, when detecting an object in a three-dimensional space as a cylinder or cube and limiting the area in the depth direction, an example of the detection area being divided and a temporary combining method as a countermeasure will be described. As shown in Fig. 9, a large vehicle (such as a bus or truck) at the edge of the field of view is imaged only on one side, but since it is long in the depth direction, it may be divided into multiple detection areas R1 and R2 and detected in step S3a. The division in the depth direction is a necessary concept for dividing and detecting multiple objects adjacent to each other in an image, such as when vehicles are parked next to each other in the oncoming lane, but in the environment of Fig. 9, it is necessary to combine the detection areas R1 and R2.

[0047] However, when using the method described in Figure 8 of combining detection areas with a distance difference of within 10%, the detection area R1 (depth distance 8 m) at the rear of the bus and the detection area R2 (depth distance 10 m) at the front have a difference of more than 10% in the depth threshold value, so the method in Figure 8 cannot create a tentative combined area C by combining detection area R1 and detection area R2.

[0048] Therefore, as shown in Figure 10, the judgment is made by focusing on the distance of pixels or small regions within the detection region. Here, small regions with the same distance information in the vertical direction are used. Small regions R1a to R1d are small regions obtained by dividing detection region R1 into four, and each small region has a depth distance, with the average of these depth distances being the depth distance of detection region R1, which is 8m. Similarly, small regions R2a to R2c are small regions obtained by dividing detection region R2 into thirds, and each small region has a depth distance, with the average of these depth distances being the depth distance of detection region R2, which is 10m.

[0049] In this case, the detection area R1 is 8 m away from the detection area R2, but the small area R1d and the small area R2a are adjacent to each other on the image and have a close depth distance according to the criteria in Fig. 8. When parts of the detection areas are close to each other like this, it is possible to create a temporary combined area C1 using the detection areas R1 and R2, and to recognize this temporary combined area C1 as a single object. Note that, although small areas were created in the vertical direction in the above description, this is based on the results of an example in which the depth distance is stably calculated by averaging the distance of each pixel in the vertical direction, and the method of defining the small areas is not limited to this.

[0050] Furthermore, a case where unnecessary merging occurs and a solution for the method of viewing the distance of small areas in the detection area will be described. FIG. 11 illustrates an example of the camera angle of view of the vehicle V0 and the corresponding image P. In the camera angle of view of the vehicle V0, the vehicles V1 to V4 in the oncoming lane are shown as a line of cars in the image P. When the objects (vehicles V1 to V4) are photographed overlapping with each other in this way, in step S3a, only the visible parts are acquired as the detection area, such as the detection areas R1 and R2. The detection areas R1 and R2 detected at this time will necessarily include the side areas of the vehicles V2 and V3. In this case, when the distance between the objects is short, the small area R1a of the detection area R1 of the front vehicle V2 and the small area R2a of the detection area R2 of the rear vehicle V3 will be close in distance, and there is a possibility that a tentative merged area will be erroneously created from the detection area R1 corresponding to the vehicle V2 and the detection area R2 corresponding to the vehicle V3. A method for preventing such erroneous temporary joining processing will be described with reference to FIGS.

[0051] Fig. 12 is a conceptual diagram illustrating a small area set inside the detection area R1 and a small area set inside the detection area R2 in Fig. 11. Note that, although Fig. 12 shows only the small area R1a and the small area R2a, multiple small areas are set inside each detection area.

[0052] FIG. 13 is a plot of the horizontal position and depth distance on the image of each small area in FIG. 12. When a vehicle group (vehicles V1 to V4) is detected, the relationship between the horizontal position and depth distance on the image P is expressed as shown in the graph at the bottom of FIG. 13. At this time, for an object such as a vehicle that can be approximated as a cube, the small area group at the front part and the small area group at the side part captured by the camera have a difference in inclination compared to the surrounding small areas. Therefore, for example, the inclination is calculated using a certain small area and its adjacent left and right small areas. If the inclination is the same for the small areas used in the temporary combination process of the detection areas in step S3b, it is assumed that they are capturing a piece of the same object, and temporary combination is performed. In addition, since the position in the depth direction and the inclination obtained therefrom vary depending on the sensor accuracy, it may be determined that they are the same object if the difference in inclination is below a threshold. In the above description, three points are obtained as small areas, but this is also determined depending on the sensor accuracy and is not limited to the above.

[0053] [Step S3c: Identical object determination process] In step S3c, the calculation processing unit 15 judges whether the tentatively connected region C created in step S3b is the same object. The judgment of the same object is determined according to the target to be judged and the desired accuracy. For example, when information that is not calculated during object detection, such as a series of horizontal or diagonal edges, is connected, it is judged to be the same object. Also, by comparing the colors of small regions, if the difference is below a certain value, it is judged to be the same object. Furthermore, when the object to be detected is limited to a certain value such as a vehicle, a method with even higher accuracy can be used.

[0054] When the object to be determined as the same object is, for example, a vehicle, the object to be combined with the temporary combined area is determined by pattern matching with a model pattern or by using a classifier created by machine learning. FIG. 14 shows an example of pattern matching with a model pattern. The temporary combined area C1 given by the temporary combining process is acquired in a form that includes the vehicle on the image. The image area corresponding to this temporary combined area C1 is compared with the similarity of a predetermined model pattern M, and if the similarity is equal to or higher than a certain level, it is determined to be the same object. As a method of calculating the similarity, an edge is extracted within the temporary combined area C1, and the similarity is calculated by performing normalized correlation. The method of determining the same object is not limited to the above, and may be arbitrarily determined for an object in which an event occurs that would cause the object to be divided and detected by the area detection process in step S3a.

[0055] [Step S3d: Detection area integration processing] In step S3d, the calculation processing unit 15 updates the detection area buffer by combining the detection area buffer B3 with the tentatively combined area that has passed the tentative combining process in step S3b and the identity determination process in step S3c. The area to be recorded is selected based on the calculation cost required for the subsequent process and the accuracy of each process. For example, if the accuracy of the identity determination process is high, it is selected not to save the detection area that was the basis of the tentatively combined area. In addition, there may be cases where overlapping detection areas are obtained on the image by such a combining process. An example is shown in FIG. 15. The tentatively combined areas C1 and C2 are obtained for two vehicles facing sideways, and the tentatively combined area C3 is also obtained so as to straddle the two vehicles. In such a case, the tentatively combined area C3 can be rejected by using the similarity obtained in the identity determination process in step S3c. For example, the similarity between the tentative bonded region C1 and the model pattern M is compared with that between the tentative bonded region C3 and the model pattern M, the tentative bonded region C3 having a relatively low similarity to the model pattern M is rejected, and the tentative bonded region C1 having a relatively high similarity is updated and saved in the detection region buffer B3. Similarly, the similarity between the tentative bonded region C2 and the model pattern M is compared with that between the tentative bonded region C3 and the model pattern M, the tentative bonded region C3 having a relatively low similarity to the model pattern M is rejected, and the tentative bonded region C2 having a relatively high similarity is updated and saved in the detection region buffer B3.

[0056] As described above, in the past, it was necessary to execute the detection process multiple times with different object detection parameters so that a single three-dimensional object would not be divided into multiple detection areas. However, according to the image recognition device of Example 1, even if a single three-dimensional object in an image is divided into multiple detection areas and erroneously detected, it is possible to accurately create a detection area equivalent to the single three-dimensional object by combining the multiple detection areas, thereby improving the accuracy of the calculation process of the type, position, and speed of the target.

[0057] In other words, according to the image recognition device and image recognition method of the present invention, three-dimensional objects such as vehicles and pedestrians that are the subject of monitoring in driving assistance systems, autonomous driving systems, etc. can be properly detected as independent, single objects. EXAMPLES

[0058] Next, an image recognition device 1' according to a second embodiment of the present invention will be described with reference to Fig. 16. Note that a duplicated description of points in common with the first embodiment will be omitted. In the image recognition device 1 of the first embodiment, a stereo camera consisting of a left camera 11 and a right camera 12 is used to detect a three-dimensional object, but in the image recognition device 1' of this embodiment, an optical camera 11' and a radar sensor 12' are used to detect a three-dimensional object instead of the stereo camera. Hereinafter, the processing operation of this embodiment will be described with reference to Fig. 16.

[0059] First, in step S1, image processing such as correction to absorb the inherent characteristics of the image sensor is performed on the image P captured by the optical camera 11'. The results of the image processing are stored in the image buffer B1. In addition, the distance to the three-dimensional object is obtained by the radar sensor 12'.

[0060] In step S3, a three-dimensional object in three-dimensional space is detected based on the distance to the object. The distance information used for detection is stored in the distance buffer B4. In addition, the detection process in step S3 associates the image with the distance as required for subsequent processing.

[0061] In step S4, a recognition process is performed to identify the type of three-dimensional object in the detection area set by the detection process in step S3.

[0062] In step S3 of this embodiment, the distance to the three-dimensional object output from the radar sensor 12' is used as an input, so that the detection process needs to take into account the sensor characteristics of the radar sensor 12' used for distance measurement; however, the process after the detection area is determined can be performed in the same manner as in the configuration using the stereo camera described in the image recognition device 1.

[0063] The present invention is not limited to the above-mentioned examples, and other embodiments that are within the scope of the technical concept of the present invention are also included in the scope of the present invention as long as they do not impair the characteristics of the present invention. Also, the above-mentioned embodiments may be combined with the modified examples. [Explanation of symbols]

[0064] 1, 1'...image recognition device, 11, 12...camera, 13...image input interface, 14...image processing unit, 15...arithmetic processing unit, 16...memory unit, 17...CAN interface, 18...control processing unit, 19...internal bus, CAN...vehicle network, B1...image buffer, B2...parallax buffer, B3...detection area buffer, B4...distance buffer, V...vehicle, R...detection area, C...temporary connection area

Claims

1. An area detection unit that detects a plurality of detection areas from an image; a temporary joining processing unit that temporarily joins the plurality of detection regions; an identical object determination processing unit that determines whether the temporarily combined multiple detection areas can be identified as a single object; a detection area integration processing unit that determines a combination of the plurality of detection areas that should be recognized as one object based on a determination result of the same object determination processing unit, and reflects the determined combination in an object detection result; having The temporary joining processing unit joins adjacent detection areas among the plurality of detection areas, The image recognition device is characterized in that the temporary merging processing unit merges the regions of object candidates among the multiple detection regions, even if the distance between the object candidates is greater than or equal to a threshold, if the object candidates are adjacent object candidates and the distance between the small regions adjacent to the object candidates is less than or equal to a threshold.

2. 2. The image recognition device of claim 1, An image recognition device characterized in that the distance of the small region is a distance corresponding to pixels in a region of an object candidate, or a distance corresponding to a region made up of a plurality of pixels.

3. 3. The image recognition device according to claim 2, An image recognition device characterized in that the gradient of the distance of each small region is calculated for each small region, and if the distance of the small region is below a threshold value but the amount of change in the gradient is above a threshold value, the device does not combine the regions of the object candidate.

4. An area detection unit that detects a plurality of detection areas from an image; a temporary joining processing unit that temporarily joins the plurality of detection regions; an identical object determination processing unit that determines whether the temporarily combined multiple detection areas can be identified as a single object; a detection area integration processing unit that determines a combination of the plurality of detection areas that should be recognized as one object based on a determination result of the same object determination processing unit, and reflects the determined combination in an object detection result; having the tentative joining processing unit joins, among the plurality of detection areas, detection areas whose distance is equal to or less than a threshold, The image recognition device is characterized in that the temporary merging processing unit merges the regions of object candidates among the multiple detection regions, even if the distance between the object candidates is greater than or equal to a threshold, if the object candidates are adjacent object candidates and the distance between the small regions adjacent to the object candidates is less than or equal to a threshold.

5. In the image recognition device according to claim 4, An image recognition device characterized in that the distance of the small region is a distance corresponding to pixels in a region of an object candidate, or a distance corresponding to a region made up of a plurality of pixels.

6. In the image recognition device according to claim 5, An image recognition device characterized in that the gradient of the distance of each small region is calculated for each small region, and if the distance of the small region is below a threshold value but the amount of change in the gradient is above a threshold value, the device does not combine the regions of the object candidate.

7. A region detection step of detecting a plurality of detection regions from an image; a temporary combining step of temporarily combining the plurality of detection regions; a same object determination step of determining whether the temporarily combined multiple detection regions can be identified as a single object; a detection area integration step of determining a combination of the plurality of detection areas to be recognized as one object based on a result of the same object determination step, and reflecting the determined combination in an object detection result; having The temporary combining step combines adjacent detection regions among the plurality of detection regions, The image recognition method is characterized in that the tentative combination processing step combines the regions of object candidates among the multiple detection regions, even if the distance between the object candidates is equal to or greater than a threshold, if the object candidates are adjacent object candidates and the distance between the small regions adjacent to the object candidates is equal to or less than a threshold.

8. An area detection step of detecting a plurality of detection areas from an image; a temporary combining step of temporarily combining the plurality of detection regions; a same object determination step of determining whether the temporarily combined multiple detection regions can be identified as a single object; a detection area integration step of determining a combination of the plurality of detection areas to be recognized as one object based on a result of the same object determination step, and reflecting the determined combination in an object detection result; having The tentative combining process combines, among the plurality of detection regions, detection regions whose distance is equal to or less than a threshold value, The image recognition method is characterized in that the tentative combination processing step combines the regions of object candidates among the multiple detection regions, even if the distance between the object candidates is equal to or greater than a threshold, if the object candidates are adjacent object candidates and the distance between the small regions adjacent to the object candidates is equal to or less than a threshold.

Images