IMAGE PROCESSING DEVICE

Abstract

Image processing device (204) comprising the following: an object detection unit (241) that detects an object from an image; a region separation unit (242) that separates an image area in which the object was detected into several sub-areas; a range selection unit (243) that selects a sub-range to be used to calculate a magnification ratio from the several separate sub-ranges; and a TTC calculation unit (244) that calculates the time until collision with the target object from the magnification ratio calculated using the selected sub-area, wherein the image processing device is characterized in that The area selection unit (243) selects an area in which the time to collision is matched and stores the extracted area as a small deformation area if multiple means can calculate the time to collision simultaneously, and selects the stored small deformation area as a sub-area to be used for magnification ratio calculation if only the calculation of the time to collision based on the magnification ratio is possible.

Description

Technical field

[0001] The present invention relates to an image processing device. State of the art

[0002] The prior art in this technical field is represented by Japanese patent no. JP 5 687 702 B2 (PTL 1). The publication describes as the problem "the creation of a vehicle environment monitoring device in which a reduction in the accuracy of the calculation of a distance between an object and a vehicle based on an image captured by a single camera is suppressed," and describes as the solution "a vehicle environment monitoring device comprising: a distance calculation unit that calculates a distance between an object and a vehicle in real space corresponding to an image section extracted from an image captured by a single camera mounted on the vehicle; and an object type determination unit that determines the type of object in real space corresponding to the image section."The distance calculation unit determines whether the change in shape of the image segment or the shape of the object in real space corresponding to the image segment occurs within a predetermined time period. If the change in shape exceeds a predetermined level, it performs a first distance calculation to calculate the distance between the object and the vehicle. This is done by applying the size of the object segment extracted from the captured image to a correlation between a distance from the vehicle in real space (assumed by the object type) and the size of the image segment in the captured image. If the change in shape is equal to or less than the predetermined level, it performs a second distance calculation to calculate the distance between the object and the vehicle based on a change in the size of an image segment of the object.which is extracted from a time-series image captured by the camera.” Furthermore, PTL 2 describes a device for vehicle environment monitoring that detects objects by calculating the area magnification rate of their images between two time-delayed camera images. If this magnification rate exceeds a predefined threshold, the object is detected as a three-dimensional object. This method thus enables the differentiation between nearby objects and a more distant background.because nearby objects exhibit a higher area magnification rate. PTL 3 describes a driver assistance device for estimating the future collision time (TTC) using a monocular camera. For this purpose, images of an obstacle are captured at several points in time, and its image size is determined. A characteristic curve for the development of the collision probability is approximated from the temporal change in this image size. Based on this characteristic curve, the future collision time, in particular the time at the current point in time, is then estimated. PTL 4 describes a device for calculating the collision time (TTC). It uses time series data of the size of a detected object. The device determines whether the object was detected using a whole-body or a partial-body feature catalog. Based on this determination, a frequency range of the time series data is attenuated.before the magnification rate and final collision time are calculated from the filtered data. PTL 5 further describes a pedestrian detection device that first identifies a region with uniform movement as a moving object in image sequences. Below this region, a lower area is defined and analyzed for non-uniform movements of feature points characteristic of walking legs. Alternatively, this area is compared with stored leg patterns. If a match is found, the combined region is defined as the entire body of the pedestrian. List of prior art patent literature PTL 1: JP 5 687 702 B2 PTL 2: JP 2016 - 009 331 A PTL 3: JP 2012 - 098 776 A PTL 4: WO 2016 / 152 807 A1 PTL 5: JP 2009 - 157 581 A Summary of the invention: Technical problem

[0003] According to the vehicle environment monitoring device described in PTL 1, it is possible to create a vehicle environment monitoring device that suppresses a reduction in the accuracy of the calculation of a distance between an object and a vehicle based on an image captured by a single camera.PTL 1 discloses a method for switching according to a condition between the first distance calculation processing for calculating a distance between an object and a vehicle by applying the size of an image section of the object, which is extracted from a captured image, to a correlation between a distance from the vehicle in a real space, which is determined under the assumption of the type of object, and the size of an image section in the captured image, and the second distance calculation processing for calculating a distance between the object and the vehicle based on a change in the size of an image section of the object, which is extracted from a captured time series image.

[0004] In this case, the method specified by the second distance calculation processing uses a change in the appearance of a target object in an image, that is, a change in a relative positional relationship with a sensor, and consequently is known to be excellent at estimating the time to collision (TTC). Since accurate TTC estimation is essential for vehicle control, it is desirable to use this method in as many scenarios as possible. However, according to PTL 1, the method is switched under the condition that "a change in the shape of an image section or a change in the shape of an object in a real space corresponding to the image section is determined within a predetermined time period and the change in shape corresponds to a predetermined level," and the accuracy of the TTC decreases when the first distance calculation processing is performed.

[0005] One object of the present invention is to provide an image processing device that is capable of estimating the TTC between a target object and a vehicle with high accuracy. Solution to the problem

[0006] To solve the problem, an image processing device with the features of claim 1 is provided. Advantageous further developments are described in the dependent claims. Advantageous effects of the invention

[0007] According to the present invention, it is possible to create an image processing device that is capable of estimating the TTC between a target object and a vehicle with high accuracy.

[0008] Other problems, configurations and effects than those described above will be clarified from the following description of embodiments. Brief description of the drawings [ Fig. 1] Fig. Figure 1 is an explanatory diagram showing a schematic configuration of an in-vehicle camera system according to Example 1. [ Fig. 2] Fig. Figure 2 is an explanatory diagram that illustrates a configuration of an object detection device according to Example 1. [ Fig. 3] Fig. Figure 3 is an explanatory diagram of object detection using a bird's-eye view image. [ Fig. 4] Fig. Figure 4 is an explanatory diagram that shows an example of output from an input image to a range selection unit. [ Fig. 5] Fig. Figure 5 is an explanatory diagram that shows an example of relative positions with reference to an object and captured images at two points in time. [ Fig. 6] Fig. Figure 6 is an explanatory diagram that illustrates a configuration of an object detection device according to Example 2. [ Fig. 7] Fig. Figure 7 is a flowchart that represents an example of a process of an image processing unit according to Example 2. [ Fig. 8] Fig. Figure 8 is an explanatory diagram that illustrates a configuration of an object detection device according to Example 3. [ Fig. 9] Fig. Figure 9 is a flowchart that represents an example of a process of an image processing unit according to Example 3. [ Fig. 10] Fig. Figure 10 is an explanatory diagram that presents an example (example of determination using a short-time optical flow) of range selection using an optical flow. [ Fig. 11] Fig. Figure 11 is an explanatory diagram that presents an example (example of determination using a long-term optical flow) of area selection using an optical flow. [ Fig. 12] Fig. Figure 12 is an explanatory diagram that illustrates a configuration of an object detection device according to Example 4. [ Fig. 13] Fig. Figure 13 is a flowchart that represents an example of a process of an image processing unit according to Example 4. [ Fig. 14] Fig. Figure 14 is an explanatory diagram that shows an example of a stereo area and monocular areas in a stereo camera. [ Fig. 15] Fig. Figure 15 is an explanatory diagram that illustrates a configuration of an object detection device according to Example 5. [ Fig. 16] Fig. Figure 16 is a flowchart that represents an example of a process of an image processing unit according to Example 5. Description of embodiments

[0009] Examples of the present invention are described below with reference to the drawings. [Example 1]

[0010] An outline of an in-vehicle camera system equipped with an object detection device according to Example 1 is shown with reference to Fig. As described in Figure 1, a camera 101 is mounted on a vehicle 100 in the vehicle's internal camera system. The object detection device 102 is mounted on the camera 101 and measures, for example, the distance and relative speed to an object in front and transmits the distance and relative speed to a vehicle control unit 103. The vehicle control unit 103 controls a brake and an accelerator pedal 105 and a steering system 104 based on the distance and relative speed received from the object detection device 102.

[0011] The camera 101 includes the object detection device 102, which is located in Fig. Figure 2 shows the object detection device 102. It comprises an imaging element 201, a memory 202, a CPU 203, an image processing unit (image processing device) 204, an external output unit 205, and the like. The components forming the object detection device 102 are connected via a communication line 206. The image processing unit (image processing device) 204 comprises an object detection unit 241, an area separation unit 242, an area selection unit 243, and a TTC calculation unit 244. The CPU 203 performs arithmetic processing, described below, according to an instruction from a program stored in memory 202.

[0012] An image (obtained by capturing the surroundings of the vehicle 100) captured by the imaging element 201 is transferred to the image processing unit 204. The object detection unit 241 detects an object, compares object detection results obtained for each calculation cycle, and tracks the same object in relation to each other. The area of ​​the target object (detected object) in the captured image is subdivided into several areas by the area separation unit 242. An area that meets a predetermined condition among the several subdivided areas is selected by the area selection unit 243. The selected area is transferred to the TTC calculation unit 244, and the TTC is calculated based on a relative change (i.e., a magnification ratio) in the size of the area. The calculated TTC is transferred by the external output unit 205 to the outside of the object detection device 102.In the vehicle-internal camera system described above, the TTC is used by the vehicle control unit 103 to determine the vehicle control of the accelerator pedal and brake 105, the steering 104 and the like.

[0013] The components of the image processing unit 204 are described below. (Object detection unit 241)

[0014] The object detection unit 241 detects and tracks an object using an image captured by the imaging element 201. For example, a method for detecting an object using an image is known to employ a difference between bird's-eye view images. In this method, as described in Fig. As shown in Figure 3, an object is detected using two images, 301 and 302, captured over a time series. Of these two images, 301 and 302, the previously captured image 301 is transformed into a bird's-eye view. Simultaneously, a change in appearance due to the vehicle's movement is calculated from information such as vehicle speed, and an image 304, predicted to be captured in the current frame, is generated. A difference image 306 is generated by comparing the predicted image 304 with an image 305, which is a bird's-eye view into which the actually captured image 302 has been transformed. The difference image 306 has a difference value in each pixel; an area without a difference is displayed in black, and an area with a difference is displayed in white.If there is no difference in the prediction, the same image is obtained, and no difference occurs for a road surface. However, a difference occurs in an area where an obstacle (three-dimensional object or the like) is present. An object can be detected by detecting this difference. An object detected in the current frame is compared to an object detected in a previous frame based on correlations between the positions and textures in images. If the objects are determined to be the same and matched, the same object can be tracked in the image. Of course, means for detecting and tracking an object from an image are not limited to this. (Area separation unit 242)

[0015] The area separation unit 242 divides the imaged area (image area) of the detected object (object tracked in the image) into several areas (sub-areas). Various methods can be considered for the separation agent, but in a case where the target object is, for example, a pedestrian, the detected area can simply be divided into upper and lower (two) areas. (Area selection unit 243)

[0016] The area selection unit 243 selects an area (that is, it is used to calculate a magnification ratio) to be transferred to the TTC calculation unit 244 from among several separate areas (sub-areas). In this case, the TTC calculation unit 244 calculates the TTC from the magnification ratio of the target object (described later). To calculate the magnification ratio accurately, it is desirable that there be no other change in the image's appearance besides the magnification and reduction caused by a change in distance. If the target object is a pedestrian, an area containing the movement of a foot during walking, and the like, are unsuitable. Therefore, in this example, the upper area, corresponding to the torso, is selected from the two subdivided areas, upper and lower.

[0017] An example of the output so far is in Fig. Figure 4 illustrates this. An image 401, captured by the imaging element 201, is input, and the object detection unit 241 detects an object 402 from the image 401. The area separation unit 242 separates the area (image area) of the object 402 into an upper body area 403 and a lower body area 404, and the area selection unit 243 selects the upper body area 403 and outputs the upper body area 403 to the TTC calculation unit 244. (TTC calculation unit 244)

[0018] The TTC calculation unit 244 calculates the TTC, which is the time until a collision with the target object, from the magnification ratio of the image area (sub-area) received by the area selection unit 243. An object detected at time t and time t-1 is considered. Fig. Figure 5 presents bird's-eye views 505 and 506, images 501 and 502, which are captured at two times, and depict the positional relationship between a vehicle 504 and a target object 503 at those two times. At time t, the distance to object 503 is smaller than at time t-1, and it is observed that the size of object 503 in the image increases accordingly. At this time, if the actual height of object 503 is H [mm], the height of object 503 on a screen is h [px], the distance between the vehicle 504 and object 503 is Z [mm], the relative speed between the vehicle 504 and object 503 is rv [mm / s], the focal length of the camera is f [mm], and there is one processing cycle S [s], then the following equations (1) to (4) are satisfied. [Equation 1] Zt−1=Hfht−1 [Equation 2] Zt=Hfht [Equation 3] rv=Zt−1−zts [Equation 4] TTC=ztrv=sztzt−1−zt=sht−1ht−ht−1

[0019] In this case, if the magnification ratio in the image α = h t / h t-1 is obtained the following equation (5). [Equation 5] TTC−sα−1

[0020] Since the value of the processing cycle S is precisely obtained, the accuracy of the TTC calculation depends on the magnification ratio α. Although attention is paid to the object's height in this example, since the same equations apply to the length of any section, it is also obvious that these same equations apply not only in the case of height but also in the case where the magnification ratio of the entire image area is α. Furthermore, the magnification ratio α, calculated using the selected area, represents the rate of change of the area's size and can be 100% or more or less than 100% (in which case it can be called the reduction factor).

[0021] To calculate the exact magnification ratio α, it is necessary to ensure a range that exhibits, as far as possible, no change in appearance other than the magnification and reduction due to a change in distance. In this example, however, the range is obtained by the range separation unit 242 and the range selection unit 243, which are described above.

[0022] The TTC calculated by the TTC calculation unit 244, as described above, is transferred from the external output unit 205 to the vehicle control unit 103 outside the object detection device 102 and is used for determining the vehicle control in the vehicle control unit 103.

[0023] As described above, the image processing unit (image processing device) 204 according to this example comprises: the object detection unit 241, which detects an object from an image; the area separation unit 242, which separates an image area in which the object has been detected into several sub-areas; the image selection unit 243, which selects a sub-area to be used to calculate a magnification ratio from the several separated sub-areas; and the TTC calculation unit 244, which calculates the time until a collision with the object as the target from the magnification ratio calculated using the selected sub-area.

[0024] According to the image processing unit (image processing device) 204 described above, it is possible to create the image processing unit (image processing device) 204, which is capable of estimating the TTC between the target object and the vehicle with high accuracy, in the vehicle's in-vehicle camera system according to Example 1.

[0025] [Example 2 (Example of range selection for a target object of a special type)] Example 2 is a modification of Example 1 and shows an example in which the range selection unit 243 changes a selection condition according to the type of an object when the type of the object detected by the object detection unit 241 is identified.

[0026] It is possible to select a suitable area according to the result of the identification by providing, as prior knowledge, an area that is predicted to be less deformed for each type of object.

[0027] Fig. Figure 6 is a diagram illustrating a configuration of an object detection device 102 according to Example 2. In addition to the configuration of Example 1, an image processing unit 204 of the object detection device 102 includes an object identification unit 245, which identifies the type of an object detected by the object detection unit 241, and a small deformation area database 246, which stores and saves information about an area predicted to have small deformation for reference by the area separation unit 242 and the area selection unit 243.

[0028] Fig. Figure 7 presents an example of a flow chart for the image processing unit 204 according to this example, in particular the area separation unit 242 and the area selection unit 243.

[0029] The object detection unit 241 detects an object from an image (S701), and the object identification unit 245 identifies the type (pedestrian, bicycle, motor vehicle, or the like) of the object based on the object detection result (S702). The area separation unit 242 and the area selection unit 243 refer to the database 246 for the area with small deformation of an area predicted to have small deformation, according to the identification result of S702, and determine an area separation condition (S703) and an area selection condition (S705) so that an area predicted to have less deformation can be separated and selected from the image area.For example, if it is identified as a pedestrian, a section of its head or body is separated and selected; if it is identified as a bicycle, a section of its upper body, including a saddle or tire, is separated and selected. Alternatively, an object that is less likely to be deformed and very likely to be near the target object under a specific condition can be registered in database 246 for the area of ​​small deformation. A mapped area of ​​the object can then be searched for, and if a registered object (an object that is less likely to be deformed) is found, processing to separate and select the area can be performed.In a case where, for example, he is identified as a pedestrian and the width is equal to or greater than a threshold, a stroller or suitcase is searched for, and if the stroller or suitcase is nearby, the stroller or suitcase is separated and selected.

[0030] A procedure can be considered in which, when a child is identified, a schoolbag is sought, and when the schoolbag is nearby, it is segmented and selected. The area segmentation unit 242 segments the area according to the area segmentation condition specified in S703 (S704), and the area selection unit 243 selects the area according to the area selection condition specified in S705 (S706). The TTC calculation unit 244 calculates the TTC from the magnification ratio calculated using the image area received from the area selection unit 243 (S707).

[0031] As described above, in the image processing unit (image processing device) 204 according to Example 2, the area separation unit 242 and the area selection unit 243 separate a special area (for example, an area that is predicted to be less deformed for each type of detected object) in the image according to the type of detected object.

[0032] According to Example 2, it is possible to create the image processing unit (image processing device) 204 which implements a more stable TTC calculation when the type of a target object is identified. [Example 3 (Example of range selection for any target object)]

[0033] Example 3 is a modification of Example 1 and shows an example in which the area separation unit 242 separates a mapped area of ​​an object according to the amount of deformation, and the area selection unit 243 selects an area that is determined to have a small deformation. Since the amount of deformation is determined for the entire area in which the object is detected, it is possible to select a suitable area for any target object without requiring any processing to identify the object or any prior knowledge.

[0034] Fig. Figure 8 is a diagram illustrating a configuration of an object detection device 102 according to Example 3. In addition to the configuration of Example 1, an image processing unit 204 of the object detection device 102 has an optical flow computation unit 247, which calculates an optical flow for an entire area (image area) in which an object is detected by the object detection unit 241.

[0035] Fig. Figure 9 presents an example of a flow chart for the image processing unit 204 according to this example, in particular the area separation unit 242 and the area selection unit 243.

[0036] The object detection unit 241 detects an object from an image (S901). The object detection result is obtained for each processing cycle. The optical flow computation unit 247 calculates an optical flow in an object detection area for each processing cycle (S902). The optical flow is a result of tracking a sufficiently small local area and can represent a time-series motion for each local area within the entire area in which the object was detected. The area separation unit 242 separates the entire area in which the object was detected based on the computation result (result of tracking the local area) from S902 (S903). The area selection unit 243 selects the area (in other words, the optical flow calculated in S902) to be separated in S903 from the viewpoint of a short-term optical flow or a long-term optical flow (S904, S905).

[0037] First, a determination example is given using a short-time optical flow with reference to Fig. As described in section 10, if an optical flow is calculated for each of a mapped region 801 at time t-1 and a mapped region 802 at time t, and a straight line is drawn for each associated local region at the two times, straight lines in a region that can only be expressed by a change in magnification / reduction converge to a certain vanishing point 803. At that time, a straight line in a region 804 where a deformation other than magnification or reduction is dominant, such as the tip of a pedestrian's foot, does not pass through the vanishing point 803. The region selection unit 243 can use this property to select a region (subregion) with small deformation by selecting the local regions that belong to the straight lines that converge to the vanishing point 803.

[0038] Next, an example of determination using a long-term optical flow will be presented with reference to Fig. As described in section 11, when an optical flow is used from a mapped area 805 at any time tk to the mapped area 802 at time t, an optical flow 807 in an area that can only be expressed by a change in magnification / reduction becomes a straight line, and the degree of motion is also stabilized for each processing cycle. At the same time, an optical flow 808 in an area with a deformation other than magnification and reduction, such as the tip of a pedestrian's hand, takes a trajectory other than a straight line, and the direction and degree of motion change considerably for each processing cycle. The area selection unit 243 can use this property to select an area (sub-area) with small deformation by selecting an area where the optical flow stably traces a straight trajectory.

[0039] Then the TTC calculation unit 244 calculates the TTC from the magnification ratio calculated using the image area received from the area selection unit 243 (S906).

[0040] It should be noted that in this case the area separation unit 242 and the calculation unit 247 for the optical flux are separate blocks, but of course the calculation unit 247 for the optical flux can be provided in the area separation unit 242 and the calculation of the optical flux described above can be carried out in the area separation unit 242.

[0041] As described above, in the image processing unit (image processing device) 204 according to Example 3, the area selection unit 243 selects a sub-area to be used for magnification ratio calculation using an optical flow in a captured image area of ​​an object being tracked in an image.

[0042] According to Example 3, it is possible to create the image processing unit (image processing device) 204 which is able to select an area with a small time series tracking for any object and calculate the TTC. [Example 4 (Example of area selection for an object that performs a repeated movement)]

[0043] Example 4 is a modification of Example 1 and shows an example that is effective when a target object, such as a pedestrian, performs a repetitive movement. Since, for example, a walking motion is a repetition of a certain pattern, the time required to assume the same posture occurs at each specific time interval. In this example, the area to be selected by the area selection unit 243 can be maximized by extracting two time points when the same posture is performed from an object detection area with a past object detection result, instead of object detection results at fixed two time points, and selecting the area.

[0044] Fig. Figure 12 is a diagram illustrating a configuration of an object detection device 102 according to Example 4. In addition to the configuration of Example 1, an image processing unit 204 of the object detection device 102 includes a similarity calculation unit 248, which calculates a similarity between an object (object tracked in an image) detected by the object detection unit 241 at one of two time points and an object detected by the object detection unit 241 at the other time point.

[0045] Fig. Figure 13 shows a flow chart of the image processing unit 204 according to this example, in particular the area separation unit 242 and the area selection unit 243.

[0046] The object detection unit 241 detects an object from an image (S1301). The object detection result (image area) is stored in memory 202 or the like. In this case, the current time is t and the elapsed time is k. The similarity calculation unit 248 first sets the time k to t-1 (S1302) and calculates the similarity between the object at time t and an object at time k with reference to the stored object detection result (S1303). Various known methods can be considered for the similarity calculation. It is determined whether the similarity calculated in S1303 is equal to or greater than a predefined threshold (S1304), and if the similarity is less than the threshold, the time k is set to k-1 as a case where attitudes are different (S1305), and the similarity is recalculated (S1303).Since the purpose in this case is to determine two points in time when the same posture is performed, for example, past object detection results can be stored as long as the memory allows, the similarity to all object detection results can be calculated, and the time when the similarity is highest can be set as time k. If the similarity is equal to or greater than the threshold, that is, if the time k at which the same posture can be determined can be found, the area separation unit 242 and the area selection unit 243 separate and select the area based on the object detection results at time t and time k (that is, from the image areas acquired in the past) (S1306).

[0047] Then the TTC calculation unit 244 calculates the TTC from the magnification ratio calculated using the image area received from the area selection unit 243 (S1307).

[0048] As described above, in the image processing unit (image processing device) 204 according to Example 4, the area selection unit 243 selects a sub-area to be used for the magnification ratio calculation from image areas that were previously captured for an object tracked in an image.

[0049] According to Example 4, even in a case where a deformed area of ​​a target object mapped in the last two frames is dominant, it is possible to select a wide area without deformation using a past object detection result (image area of ​​the detected and tracked object that was mapped in the past) and improve the accuracy of the TTC. [Example 5 (Example of range selection using sensor union)]

[0050] Example 5 is a modification of Example 1 and shows an example that is effective when other distance-measuring sensors besides the camera (monocular camera) 101 are available at the same time. Fig. Figure 14, for example, represents a surveillance area for a stereo camera and a monocular camera. The distance to a target object is measured based on the principle of triangulation in an area (also called the stereo area) 1003, which can be monitored by two cameras 1001 and 1002. However, triangulation cannot be used in areas (also called monocular areas) 1004 and 1005, each of which can only be monitored by one camera (in this case, camera 1001 or camera 1002). In these areas, the TTC must be calculated using the method described in this example. Similarly, when a distance-measuring sensor, such as a millimeter-wave radar or LiDAR, and a monocular camera are used simultaneously, there is an area that can be observed by multiple sensors at the same time and an area that can only be observed by the monocular camera.As described above, if a distance-measuring sensor other than the monocular camera (stereo camera, millimeter wave radar, LiDAR or the like) is available at the same time, the TTC can be calculated simultaneously by multiple means, including the monocular camera and other distance-measuring sensors.

[0051] Fig. Figure 15 represents a configuration of an object detection device 102 according to Example 5. In addition to the configuration of Example 1, an image processing unit 204 of the object detection device 102 includes a determination unit 249 for the possibility of TTC calculation by another sensor, which determines whether the TTC can be calculated by a sensor capable of measuring distance (hereinafter referred to as another sensor) other than the monocular camera from an image area in which an object was detected by the object detection unit 241; an extraction unit 250 for a small deformation area, which extracts a less deformed area suitable for magnification ratio calculation from the image area in which the object was detected by the object detection unit 241; and a determination result database 251 for the small deformation area, which stores the area.which was extracted by the extraction unit 250 for the area with small deformation, is stored and saved as an area with small deformation.

[0052] Fig. Figure 16 shows a flow chart of the image processing unit 204 according to this example, in particular the area separation unit 242 and the area selection unit 243.

[0053] The object detection unit 241 detects an object from an image (S1601). The determination unit 249 for the possibility of TTC calculation by another sensor determines whether an area in which the object was detected by the object detection unit 241 is an area that can be observed simultaneously by the monocular camera and the other sensor, for example, the stereo camera, thus determining whether the TTC can be calculated by the other sensor (S1602). If the determination unit 249 for the possibility of TTC calculation by another sensor determines that the area in which the object was detected by the object detection unit 241 is an area that can be observed simultaneously by the monocular camera and the other sensor, for example, the stereo camera (Yes in S1602), the other sensor calculates the TTC (S1603). Since, for example, the stereo camera can directly measure the distance,The TTC can be calculated using the distance. Next, the area separation unit 242 divides the object area (image area) captured by the monocular camera into several areas (S1604). The area selection unit 243 selects an area to be used for the magnification ratio calculation from the several separated areas (S1605), and the TTC calculation unit 244 performs the TTC calculation using the magnification ratio of the selected area (S1606). The extraction unit 250 for the small deformation area compares the TTC (S1606) calculated by the monocular camera (using the magnification ratio) with the TTC (S1603) calculated by the other sensor, for example, the stereo camera, and extracts an area (that is, an area in which the TTC can be matched) for which a value (value close to a predefined threshold) close to the TTC is found.the area calculated by the other sensor, for example the stereo camera, can be calculated as an area (area of ​​small deformation) that is less deformed and suitable for calculating the magnification ratio (S1607), and stores and saves the area as an area of ​​small deformation in the determination result database 251 for the area of ​​small deformation, so that the area can be selected for the TTC calculation using the subsequent magnification ratio (S1608).

[0054] If the determination unit 249 determines, for the possibility of TTC calculation by another sensor, that the area in which the object was detected by the object detection unit 241 is not an area that can be observed simultaneously by the monocular camera and the other sensor, for example the stereo camera (No in S1602), it is not possible to calculate the TTC by the other sensor; only the calculation of the TTC based on the magnification ratio is possible, and consequently the TTC calculation by the other sensor is not carried out, and the above-described S1604, S1605, and S1606 are carried out.At this time (that is, when only the calculation of the TTC based on the magnification ratio becomes possible), the area separation unit 242 and the area selection unit 243 separate and select the area using the area with small deformation that is stored and stored in the determination result database 251 for the area with small deformation (S1604, S1605).

[0055] As described above, in the image processing unit (image processing device) 204 according to Example 5, the area selection unit 243 extracts an area in which the TTC is matched, stores the extracted area as a small deformation area if multiple means can calculate the TTC simultaneously, and selects and stores the small deformation area as the sub-area to be used for the magnification ratio calculation if only the calculation of the TTC based on the magnification ratio is possible.

[0056] According to Example 5, if data from multiple sensors are available, an area suitable for TTC calculation using the magnification ratio can be selected using a common monitoring area, and even if a target object moves to an area monitored only by a monocular camera, an area with no deformation can be accurately selected using a past area selection result, and the accuracy of the TTC can be improved.

[0057] The examples of the present invention have been described above. In the examples above, the accuracy of the TTC can be improved compared to the prior art. Therefore, the present invention is particularly suitable for application to a collision damage reduction brake in a situation where very accurate TTC is required, for example, at an intersection. It should be noted that the present invention is not limited to the examples described above and includes various modifications. The examples described above have been provided for the convenience of understanding the present invention in detail and are not necessarily limited to those with all the described configurations.Furthermore, part of the configuration of a particular example can be replaced by the configuration of another example, and the configuration of one example can be added to the configuration of another example. For any given part of the configuration of each example, it is also possible to add, remove, or replace other configurations.

[0058] Furthermore, some or all of the above configurations can be configured by hardware or implemented by a processor executing a program. Additionally, the control and information lines represent what is considered necessary for the description, and not all control and information lines on the product are specified. In practice, it can be assumed that almost all configurations are interconnected. Reference symbol list 100 vehicles 101 Camera 102 Object detection device 103 Vehicle control unit 201 Imaging element 202 storage 203 CPU 204 Image processing unit (image processing device) 205 external output unit 206 Communications Management 241 Object detection unit 242 Area separation unit 243 Area Selection Unit 244 TTC calculation unit 245 Object identification unit (Example 2) 246 Database for the area with small deformation (Example 2) 247 Calculation unit for optical flow (Example 3) 248 Similarity calculation unit (Example 4) 249 Unit of determination for the possibility of TTC calculation by another sensor (Example 5) 250 extraction units for the area with small deformation (Example 5) 251 Determination result database for the area with small deformation (example)

Claims

[1] Image processing device (204) comprising the following: an object detection unit (241) that detects an object from an image; a region separation unit (242) that separates an image area in which the object was detected into several sub-areas; a range selection unit (243) that selects a sub-range to be used to calculate a magnification ratio from the several separate sub-ranges; and a TTC calculation unit (244) that calculates the time until collision with the target object from the magnification ratio calculated using the selected sub-area, wherein the image processing device characterized by is that The area selection unit (243) selects an area in which the time to collision is matched and stores the extracted area as a small deformation area if multiple means can calculate the time to collision simultaneously, and selects the stored small deformation area as a sub-area to be used for magnification ratio calculation if only the calculation of the time to collision based on the magnification ratio is possible. [2] Image processing device (204) according to claim 1, wherein the area separation unit (242) and the area selection unit (243) separate and select a specific area in the image according to the type of detected object. [3] Image processing device (204) according to claim 1, wherein the area selection unit (243) selects a partial area to be used for magnification ratio calculation using an optical flow in a detected image area of ​​the object being tracked in the image. [4] Image processing device (204) according to claim 1, wherein the area selection unit (243) selects a partial area to be used for the magnification ratio calculation from image areas that were previously captured for the object being tracked in the image.

Images