Image processing apparatus and image processing method

Abstract

An image processing device is provided with: a distance calculation unit that calculates a distance to a detection target searched for from an acquired image; a velocity calculation unit that calculates the velocity of the detection object; and a filter processing unit that performs filter processing on the calculated distance and speed. Here, the filter processing unit performs a plurality of filter processes for a predetermined period of time, and then selects one filter process from among the plurality of filter processes.

Description

Technical Field

[0001] The present invention relates to an image processing device and an image processing method. Background Art

[0002] In recent years, advanced driver assistance systems and autonomous driving systems for vehicles (mobile objects) that use external recognition sensors such as on-board cameras and radars have attracted much attention. For example, as driving assistance functions that improve the convenience and comfort of the vehicle, there are ACC (Adaptive Cruise Control) and AEB (Autonomous Emergency Braking). ACC measures the distance and relative speed of the vehicle ahead and automatically performs tracking driving control. AEB automatically applies the brakes to avoid or mitigate the collision when there is a risk of collision with the vehicle ahead, pedestrians, bicycles, etc. In order to improve the performance of ACC and AEB, the demand for object detection technology is increasing.

[0003] Among the object detection technologies for vehicle-mounted cameras, there is a technology that combines a monocular camera and a radar to detect objects such as vehicles from images captured by the monocular camera, and uses the radar to measure the distance and relative speed to the object. In addition, there is also a technology that uses the projection difference (parallax) of multiple images captured by a stereo camera to measure the distance to the object and calculate the relative speed based on the distance change between frames. When detecting an object using these on-vehicle cameras, since the observation error associated with the distance and speed calculated in units of frames of captured images is large, the vehicle control is smoothed by filtering to make it easier to handle.

[0004] Patent document 1 describes an object detection device, which includes: a first distance measuring unit that performs distance measurement by performing stereo matching between a pair of left and right images taken by a stereo camera and detecting parallax for the same object; and a second distance measuring unit that detects parallax by performing stereo matching based on brightness difference when the sensitivity balance between the pair of left and right images of the stereo camera is disturbed due to backlighting, etc. and the reliability of the distance image generated by the first distance measuring unit is reduced. Furthermore, in the object detection device described in Patent Document 1, the distance measurement by the first distance measuring unit and the distance measurement by the second distance measuring unit are selectively switched and input to the recognition processing unit, so that continuous distance measurement can be performed even in deteriorated environmental conditions such as backlighting. Prior art literature Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2008-151659 Summary of the invention Problem that the invention aims to solve

[0006] However, when performing filtering processing during distance measurement, it is necessary to assign an initial value to the distance measurement value for calculation. As an easy method of setting the initial value, it is conceivable to assign an arbitrary fixed value or an observed value immediately after the start of detection as the initial value.

[0007] However, since the state quantity is unknown immediately after the object is detected, if the initial value deviates from the true value, the error becomes larger even after the filtering process. In particular, the error tends to become larger immediately after the detection of a distant vehicle with low observation accuracy or an oncoming vehicle with a high relative speed, and it takes time for the filter error to converge. This error immediately after the detection becomes a problem in controlling the situation of rapidly approaching an oncoming vehicle with a higher relative speed or a distant vehicle.

[0008] For example, the object detection device described in Patent Document 1 changes the processing content to cope with deteriorating environmental conditions such as backlighting, but it is difficult to reduce errors immediately after the start of object detection.

[0009] An object of the present invention is to provide an image processing device and an image processing method which can quickly converge a filter error when detecting an object through image processing. Technical means of solving problems

[0010] In order to solve the above-mentioned problem, for example, the structure described in the technical solution is adopted. The present invention includes multiple means for solving the above-mentioned problems. To give one example, an image processing device is provided, which includes: a distance calculation unit, which calculates the distance to a detection object searched from an acquired image; a speed calculation unit, which calculates the speed of the detection object; and a filter processing unit, which performs filter processing on the distance calculated by the distance calculation unit and the speed calculated by the speed calculation unit, and the filter processing unit selects a filter processing from multiple filter processing after performing multiple filter processing for a specified period of time. Effects of the Invention

[0011] According to the present invention, it is possible to reduce the speed error immediately after the start of object detection. The other objects, structures, and effects than those described above will become clear from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing a configuration example of an image processing device according to an embodiment of the present invention. Figure 2 This is a block diagram showing an example of the hardware configuration of an image processing device according to an embodiment of the present invention. Figure 3 This is a flowchart showing an example of image processing operation according to one embodiment of the present invention. Figure 4 This is a diagram schematically showing the relationship between the relative speed of the operation and the elapsed time during the parallel filtering process according to an embodiment of the present invention. Figure 5 This is a diagram schematically showing the relationship between the distance and the elapsed time of the operation during the parallel filtering process according to one embodiment of the present invention. DETAILED DESCRIPTION

[0013] Hereinafter, an image processing device and an image processing method according to an embodiment of the present invention will be described with reference to the drawings.

[0014] [Configuration of Image Processing Device] Figure 1 1 is a diagram showing the configuration of an image processing device 100 according to the present embodiment. The image processing device 100 of the present embodiment detects and recognizes objects such as vehicles and persons traveling in front of the vehicle using images captured by a camera mounted on a vehicle as a moving object.

[0015] The image processing device 100 includes two cameras 101 and 102 as imaging devices, an image signal processing unit 110 , a recognition processing unit 120 , a storage area 140 , and a control signal output unit 150 . The two cameras 101 and 102 are stereo cameras, camera 101 is a left camera, and camera 102 is a right camera. The left camera 101 and the right camera 102 are arranged horizontally in such a manner as to obtain a field of view in front of the vehicle.

[0016] The image signals captured by the left camera 101 and the right camera 102 are provided to the image capturing unit 111 of the image signal processing unit 110 for processing and become image signals with a fixed frame period. In addition, the images captured by the left camera 101 and the right camera 102 may be any images as long as they are color images, grayscale images, infrared images, etc., which are suitable for object detection processing. The left image captured by the left camera 101 and the right image captured by the right camera 102 are recorded in the image information storage section 141 of the storage area 140 .

[0017] The left image captured by the left camera 101 and the right image captured by the right camera 102 are supplied to the parallax calculation section 112 . The parallax calculation unit 112 takes the left image and the right image as input information, searches for an area in the left image that is similar to a specific area in the right image, and calculates the parallax between the right image and the left image. The parallax calculation unit 112 performs the parallax calculation on the entire area of ​​the right image, and outputs a parallax image with the left image. The parallax image obtained in this way is recorded in the image information storage unit 141 of the storage area 140.

[0018] The image information generated by the image signal processing unit 110 is provided to the recognition processing unit 120 via the image information storage unit 141 in the storage area 140 . The recognition processing unit 120 includes a detection target search processing unit 121 , an object tracking processing unit 122 , a distance calculation unit 123 , a speed calculation unit 124 , a filter processing unit 125 , and a parallel filter processing unit 130 .

[0019] The parallel filter processing unit 130 includes an initial value setting unit 131 , a plurality of filter processing units 132 , an observation residual accumulation calculation unit 133 , an initial value selection unit 134 , and a tracking determination unit 135 .

[0020] The detection object search processing unit 121 in the recognition processing unit 120 receives the left image, the right image, and the parallax image as input, searches for an object such as a vehicle that is a detection object from the image, and outputs the position and size information of the detection object on the image as a processing result. The processing result output by the detection object search processing unit 121 is recorded in the detection object information storage unit 142 of the storage area 140.

[0021] The object tracking processing unit 122 performs object tracking processing using the left image, the right image, the parallax image, and the processing result of the detection object search processing unit 121 as input. That is, the object tracking processing unit 122 predicts the position of the current time based on the position information of the object information at the previous time, and performs object search as it is highly likely that the same object exists around the predicted position. In this way, the object tracking processing unit 122 generates time-series continuous tracking information of the same object. The generated time-series continuous tracking information of the same object is recorded in the detection object information storage unit 142 of the storage area 140.

[0022] The distance calculation unit 123 receives the parallax image and the tracking information as input, performs distance calculation processing to calculate the distance from the cameras 101 and 102 to the object being tracked, and records the calculated distance information in the detection target information storage unit 142 of the storage area 140 . The speed calculation unit 124 uses the left image, the right image, the parallax image, and the tracking information to perform speed calculation processing to calculate the relative speed between the vehicle and the object being tracked based on the change in distance information between the current time and the past time, etc. Then, the relative speed information calculated by the speed calculation unit 124 is recorded in the detection object information storage unit 142 of the storage area 140.

[0023] The filter processing unit 125 performs filter processing on the distance and relative speed of the object being tracked recorded in the detection target information storage unit 142 of the storage area 140 as input, and estimates the state quantities of the distance and relative speed of the object being tracked.

[0024] As described below, the parallel filter processing unit 130 performs a plurality of different filter processes on object information immediately after tracking is started. The initial value setting unit 131 sets different speed initial values ​​for the plurality of filters, and records the set speed initial values ​​in the speed initial value storage unit 143 of the storage area 140. As the speed initial value set here, a fixed value arbitrarily determined in advance may be assigned. Alternatively, a speed initial value determined according to the vehicle speed may be assigned. In addition, different speed initial values ​​may be assigned according to the recognition result of the detection object, such as whether the detection object is a vehicle or a pedestrian.

[0025] The plurality of filter processing units 132 maintain independent state quantities for the plurality of filters, respectively, and use the distance and relative speed of the object being tracked as input, perform filter processing according to the respective assigned initial speed values, and estimate the state quantity of the object being tracked. Then, each estimated filter result is recorded in the parallel filter state quantity storage unit 144 of the storage area 140.

[0026] The observation residual cumulative calculation unit 133 calculates the residuals between the estimated distance values ​​predicted by the plurality of filters and the observed distance value at this time obtained by the distance calculation unit 123 , and cumulatively adds the results. The cumulative addition values ​​of the residuals are recorded in the observation residual cumulative value storage unit 145 of the storage area 140 .

[0027] The initial value selection unit 134 selects a filter with the smallest residual from the observation residual accumulation values ​​of the respective filtering results as the best filtering result. The tracking finalization determination unit 135 determines whether tracking is finalized based on the number of tracking continuation times of the object being tracked and the initial value selection result. The object detection result obtained by the recognition processing unit 120 is sent to the control signal output unit 150 and then used as input information of the vehicle control system. For example, the object detection result obtained by the recognition processing unit 120 is used for the control of ACC and AEB in the vehicle control system.

[0028] [Hardware Configuration of Image Processing Device] Figure 2 An example of the hardware configuration when the image processing apparatus 100 is configured by a computer is shown.

[0029] The image processing apparatus 100 includes a CPU (Central Processing Unit) 100a, a ROM (Read Only Memory) 100b, a RAM (Random Access Memory) 100c, a nonvolatile memory 100d, a network interface 100e, an input unit 100f, and an output unit 102e, which are connected to a bus respectively.

[0030] The CPU 100a is a calculation processing unit that reads out a program code of software that realizes the functions performed by the image processing apparatus 100 from the ROM 100b or the nonvolatile memory 100d and executes the program code. The ROM 100b stores the program code. The RAM 100c is used as a work area for executing calculation processing.

[0031] Under the control of the CPU 100a, the operation processing according to the program code is executed in the RAM 100c, so that various processing function units are formed in the RAM 100c. Figure 1 The image signal processing unit 110 and the recognition processing unit 120 are shown.

[0032] As the nonvolatile memory 100d, for example, a large-capacity information storage medium such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), or a memory card is used. The nonvolatile memory 100d stores software for realizing the functions of the image processing apparatus 100, data obtained by executing the program, and setting data such as initial values.

[0033] As the network interface 100e, for example, a NIC (Network Interface Card) is used to transmit and receive data with other devices. For example, the network interface 100e communicates with an external device and transmits image data. The input unit 100 f performs input processing on image data captured by the cameras 101 and 102 . The output unit 100 g performs output processing as a control signal output unit 150 .

[0034] In addition, Figure 2, the hardware configuration of the image processing device 100 when a computer is used is shown, but part or all of the functions performed by the image processing device 100 may be implemented by hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0035] [Object detection processing] Figure 3 Detailed Description of the Invention This is a flowchart showing the object detection process performed by the image processing device 100 of the present embodiment.

[0036] The image processing in the image processing device 100 is performed on each image captured by the cameras 101 and 102 at a predetermined cycle. First, the image capturing unit 111 acquires a stereoscopic image captured by the cameras 101 and 102 as the capturing devices (step S11). When the stereoscopic image is acquired, the parallax calculation unit 112 calculates the parallax in each pixel based on the projection difference of each image, and generates a parallax image (step S12). The stereoscopic image and the parallax image are recorded in the image information storage unit 141 of the storage area 140.

[0037] Next, the detection target search processing unit 121 uses the stereo image and the parallax image from the image information storage unit 141 as input information to perform an object search (step S13 ). Here, a specific example of object search performed by the detection object search processing unit 121 is described. For example, the detection object search processing unit 121 groups adjacent disparities with close values ​​from the disparity image and generates a rectangle surrounding an area formed by the grouped disparities.

[0038] When the vertical width and horizontal width of the generated rectangle are greater than a certain value, the detection object search processing unit 121 generates the position information (coordinates on the parallax image) and size information (horizontal width, vertical width) of the rectangle as a detection object candidate because there is a possibility that an object to be recognized exists. In addition, multiple detection object candidates can also be obtained. At this moment, the detection object search processing unit 121 can narrow down the detection object candidates based on, for example, the position of the rectangle on the parallax image, the aspect ratio of the rectangle, etc., or can output all the obtained recognition candidates.

[0039] Furthermore, the detection object search processing unit 121 may use features such as shape from the image to identify the detection object candidate, such as whether the detection object is a vehicle or a pedestrian, in order to narrow down the object information that is more helpful for control. For example, an identifier obtained by machine learning by pre-collecting a large number of images of vehicles and other objects may be used to perform object recognition. The result obtained by the detection target search processing unit 121 in this way is recorded in the detection target information storage unit 142 in the storage area 140 .

[0040] return Figure 3 The flowchart of is continued. Next, the object tracking processing unit 122 performs object tracking processing on the object searched by the detection object search processing unit 121 (step S14). Here, the object tracking processing unit 122 predicts the position at this moment based on the position information of the object information at the previous moment. And, the object tracking processing unit 122 performs object search as it is likely that the same object exists around the predicted position. Thus, the object tracking processing unit 122 generates time-series continuous tracking information of the same object.

[0041] As the tracking processing, for example, a method of using the image of the object at the last moment as a template to search for the object at this moment by template matching, a method of estimating the movement amount of each pixel in the detection frame of the object being tracked by optical flow, etc. can be considered. The object tracking processing unit 122 predicts the motion of the object at this moment based on the position at the last moment and the past movement amount.

[0042] In addition, right after the detection begins, the filtering results of distance and speed may become unstable, or even if a single image is used for detection and recognition, it may be detected or recognized incorrectly. Therefore, the detection object is treated as undetermined for a certain period of time from the start of detection, and the reliability of detection and recognition can be improved by using information from multiple frames for judgment. Therefore, the object tracking processing unit 122 sets the object to an undetermined tracking state if a certain period has not passed since the start of detection, and sets the object to a determined tracking state if tracking has continued for more than a certain period, and determines whether the object is an object to be detected. In this way, the object tracking processing unit 122 updates the object information at this time and records it in the detection object information storage unit 142 of the storage area 140.

[0043] Next, the distance calculation unit 123 uses the parallax information to perform distance calculation processing to calculate the horizontal distance from the cameras 101 and 102 to the object (step S15). In a stereo camera, the distance can be calculated based on the principle of triangulation using the parameters of the baseline length, focal length, pixel size and parallax between the cameras 101 and 102. The distance information calculated by the distance calculation unit 123 is recorded in the detection object information storage unit 142 of the storage area 140.

[0044] Next, the recognition processing unit 120 determines whether the detected object has started tracking or whether this is the first detection (step S16). Then, in the case of the first detection ("Yes" in step S16), in the parallel filtering processing unit 130, the repeated processing of step S17 is performed according to the number of filters executed in parallel.

[0045] The filtering process here includes, for example, a process using a Kalman filter, etc. The Kalman filter is considered to be suitable for a system that estimates the current state based on observations including errors, such as measuring the position and velocity of an object through image processing. In addition, the parallel filter processing unit 130 dynamically finds the optimal parameters based on the error of the observation value obtained by observation at each moment and the estimated error of the filter, and estimates the state quantity at the current moment with high accuracy. However, since the Kalman filter improves accuracy by repeatedly observing and predicting and updating the state quantity, the initial state after the filtering process starts is sometimes different from the optimal parameters. In particular, when the observed object is the relative speed of a vehicle, the true value differs greatly depending on whether the vehicle is traveling at the same speed as the vehicle itself, stopped, or approaching as an oncoming vehicle. Therefore, it is considered that the error is large right after the detection starts.

[0046] In the present embodiment, in step S17 , the parallel filter processing unit 130 executes a plurality of Kalman filters to which different parameters are assigned in parallel, and estimates the state of the distance and relative speed of the object respectively. At this time, the parallel filtering processing unit 130 can always perform parallel filtering processing unconditionally. For example, when there are multiple detection objects in order to reduce the processing load, it can also give priority to objects close to the vehicle to determine whether parallel filtering processing needs to be performed.

[0047] If the parallel filtering process in step S17 is described, first, the initial value setting unit 131 sets different initial speed values ​​for each filter executed in parallel (step S18). The different initial speed values ​​set here can be obtained from the initial speed value storage unit 143 of the storage area 140, for example. In addition, as the speed initial value set here, a fixed value determined arbitrarily in advance can be given, for example, an initial value determined according to the vehicle speed can also be given. In addition, the speed initial value can also be changed according to the recognition result of the detection object such as whether the detection object is a vehicle or a pedestrian.

[0048] Then, the plurality of filter processing units 132 perform the processing of each filter, that is, the updating processing of the speed filter (step S19). In the filter processing of this embodiment, the distance information and speed information obtained by observation are used as input, but since there is no past information at the first execution, the speed calculation is not performed. Instead, the speed initial value set in step S18 is input.

[0049] Next, the plurality of filter processing units 132 predict the distance of the object at the next time point based on the distance and the speed in the state quantity estimated by the filter processing in step S19 (step S20 ). The plurality of filter processing units 132 record the processing results of steps S18 to S20 in the parallel filter state quantity storage unit 144 of the storage area 140 , and terminate the processing at this time.

[0050] When the object detected in step S16 has started tracking ("No" in step S16), the speed calculation unit 124 calculates the relative speed of the object based on the information of the past time and the current time (step S21). Regarding the method of calculating the speed, the relative speed can be calculated from the time change of the distance according to the distance, or the speed can be calculated using the optical flow of the object projected on the image. The speed information calculated by the speed calculation unit 124 is recorded in the detection object information storage unit 142 of the storage area 140.

[0051] Next, the recognition processing unit 120 determines whether the object whose speed is calculated in step S21 has been tracked (step S22). In step S22, if the tracking is not confirmed ("Yes" in step S22), the recognition processing unit 120 counts the elapsed time of the object being continuously tracked since the detection and tracking start, and determines whether the number of continuous tracking times is greater than a certain number (step S23).

[0052] When the number of continuous tracking times is less than the predetermined number in step S23 (No in step S23 ), the parallel filter processing unit 130 repeatedly performs the filtering process of steps S25 to S27 described below by the number of filters executed in parallel (step S24 ).

[0053] The filter processing of steps S25 to S27 will be described. First, similarly to step S19 , the plurality of filter processing units 132 execute the processing of each filter, that is, the updating processing of the execution speed filter (step S25 ). First, in the plurality of filter processing units 132 , if the object is being tracked, past information exists and the speed is calculated in step S21 , so the distance information and speed information obtained by observation are input to the filter.

[0054] Next, the observation residual cumulative calculation unit 133 calculates the difference between the observed value of the distance at this time and the estimated value of the distance estimated by the filter, accumulates the residual, and records it in the observation residual cumulative value storage unit 145 of the storage area 140 (step S26). Furthermore, similar to the process of step S20 , the parallel filter processing unit 130 predicts the distance of the object at the next time based on the distance and speed estimated by the filter process (step S27 ).

[0055] The parallel filter processing unit 130 records the results obtained by executing these steps S25 to S27 in the parallel filter state quantity storage unit 144 of the storage area 140 and ends the processing at this time.

[0056] In addition, when the number of continuous tracking times in step S23 is greater than a certain number ("Yes" in step S23), the initial value selection unit 134 performs an initial value selection process (step S28). In the initial value selection process in step S28, the result of the best filter process is selected from the multiple filter processes executed. Then, only the selected filter process is executed after the next time. At this time, the initial value selection unit 134 selects the value with the smallest distance observation residual cumulative value as the best filtering result.

[0057] If different initial values ​​are assigned to the filter, the estimated result will change even if the same distance or speed is used as input information. If the filter is assigned appropriate initial values, it is considered that the estimated value and the observed value have the same transition, and the accumulated value of the observation residual becomes smaller. On the other hand, it is considered that a filter with inappropriate initial values ​​will produce a deviation between the estimated value and the observed value, and the cumulative value of the observation residual will increase. Therefore, after a certain period of filtering processing has passed, the optimal filtering result can be selected by comparing the cumulative value of the observation residual.

[0058] After the initial value selection process is performed, the tracking determination unit 135 determines whether tracking is determined (step S29). If the tracking continues to be successful for a certain period of time and the selection of the filter result can be executed, the tracking determination unit 135 determines that tracking is determined. Then, similar to the process of step S19, the plurality of filter processing units 132 perform a process of updating the selected filter (step S30). Furthermore, the plurality of filter processing units 132 predict the distance of the object at the next moment based on the distance and speed estimated by the filter process (step S31). The parallel filter processing unit 130 records the result obtained by executing the process of steps S28 to S31 in the parallel filter state quantity storage unit 144 of the storage area 140 , and ends the process at this time.

[0059] In addition, when it is determined in step S22 that tracking is confirmed ("No" in step S22), the multiple filter processing units 132 perform update processing only on the selected filter (step S32). Then, the multiple filter processing units 132 predict the distance of the object at the next moment based on the distance and speed estimated by the filter processing in step S32 (step S33). The results of the execution in steps S32 to S33 are recorded in the parallel filter state quantity storage unit 144 of the storage area 140 , and the processing at this point in time is terminated.

[0060] [Operation example of parallel filtering processing] Figure 4 and Figure 5 It is a schematic diagram showing an operation example of the parallel filtering process according to the present embodiment. Figure 4 : is a graph showing the true value, the velocity observation value, and the velocity output by the parallel filtering process, wherein the horizontal axis represents the elapsed time and the vertical axis represents the relative velocity v of the object. Figure 5 d is a graph showing the true value, the distance observation value, and the distance output by the parallel filtering process, wherein the horizontal axis represents the elapsed time and the vertical axis represents the distance d.

[0061] exist Figure 4 In FIG. 3 , the true value 311 of the relative velocity of the object to be detected is indicated by a dotted line. Figure 4 In the example, the object is detected at time t0, and the observed value 312 of the relative speed obtained at each time after that is indicated by a dot-dash line.

[0062] When object detection starts at time t0, the initial values ​​of the five filtering processes provided by the initial value setting unit 131 are set to v A 、v B 、v C 、v D 、v E ,exist Figure 4 The initial velocity value v is shown in A 、v B 、v C 、vD 、v E The filtering processing results of the speed are 301, 302, 303, 304, and 305. exist Figure 5 In the figure, the observed value 321 of the relative speed obtained at each moment is also represented by a dotted line, showing the initial value v of the speed A 、v B 、v C 、v D 、v E The filtering processing results of the distances below are 311, 312, 313, 314, and 315. Here, as an example, the initial value is determined to be 5 patterns, but it may be more or less than this.

[0063] When the tracking at the next moment is not determined and the number of tracking times is less than a certain number, each filtering process is performed by multiple filtering processing units 132. Although the speed observation value 312 obtained by the speed calculation unit 124 has deviations, it is distributed near the true value 311, so each filtering result gradually approaches the true value over time.

[0064] On the other hand, the distance observation value 321 obtained by the distance calculation unit 123 is as follows: Figure 5 The plurality of filter processing units 132 perform respective filter processing to estimate the state quantity of the distance, but at this time, in addition to the distance observation value, the state quantity of the speed is also reflected, so if there is a speed error, there will also be an error in the distance. That is, if the speed estimation result is greater than the true speed value, the distance estimation result is greater than the actual value and moves toward the distance away, and if the speed estimation result is less than the true speed value, the distance estimation result is less than the actual value and moves toward the distance away.

[0065] like Figure 5 As shown, the larger the speed error is, the larger the difference between the distance observation value and the filter estimated distance is. By accumulating the observation residual at each moment, the deviation from the true value can be quantified.

[0066] Here, it is assumed that the number of tracking times reaches a certain number or more at time t1, and the initial value selection process is performed by the initial value selection unit 134. At this time, the filter with the smallest observation residual cumulative value of the distance calculated in each filtering process is selected as the optimal filter, but Figure 4 and Figure 5 In the example shown, the initial value v D The filtering results 304 and 314 at the time t1 are selected as the best results. The results after the time t1 are effective as information used for control, and only the results with the smallest error are used for control. That is, after time t1, only the initial value v DThe filtering processing results 304 and 314 are further filtered by the filtering processing unit 125.

[0067] [Effects of this embodiment] According to the image processing device of the present embodiment and the image processing method executed by the image processing device described above, parallel filtering is performed, and at the effective time ( Figure 4 , Figure 5 After t1, the filter result with small error is selected. Therefore, no matter which object is the relative speed, the time required for the filter to converge can be shortened. Therefore, even for a distant vehicle with low observation accuracy or an oncoming vehicle with a high relative speed, an appropriate filter result can be selected, and the speed error after the object detection is just started can be reduced even in various situations. Therefore, when ACC or AEB is performed based on the output of the image processing device of this embodiment, the performance of ACC or AEB can be improved.

[0068] Here, in the image processing device of the present embodiment, the distance is calculated by the distance calculation unit 123 based on the parallax of multiple images obtained from two cameras 101 and 102 as stereo cameras. This makes it possible to simply calculate the distance based on the comparison of the images, and to appropriately perform the distance calculation based on the selection of the filtering result with the small error.

[0069] In addition, in the image processing device of this embodiment, Figure 4 and Figure 5 As described above, by assigning different initial velocities to each of the plurality of filtering processes, the possibility of obtaining an appropriate filtering result can be increased, and the effect of reducing the velocity error immediately after the start of object detection can be enhanced.

[0070] Furthermore, in the image processing device of the present embodiment, the predicted distance of the initial speed after a specified period is compared with the distance observed based on the parallax of multiple images, and the filtering process with the smallest distance difference is selected as the best filtering process, thereby being able to select an appropriate filtering process through a simple operation based on the calculation of the distance difference.

[0071] Furthermore, in the image processing device of the present embodiment, by having an initial speed setting unit for setting multiple initial speeds, multiple filter processing units, an observation residual accumulation calculation unit, and an initial value selection unit for selecting the filter processing with the smallest observation residual accumulation value, it is possible to simply and quickly select the optimal filter processing based on the observation residual accumulation value.

[0072] [Modifications] In addition, the embodiment examples described so far have been described in detail in order to explain the present invention in an easily understandable manner, and are not necessarily limited to those having all the described configurations. For example, in the above-mentioned embodiment, the distance calculation unit 123 of the recognition processing unit 120 calculates the distance based on two images of the stereo camera, but if possible, the distance can also be calculated based on the image of one camera, or the distance of the object can be calculated by referring to the information of other distance measuring units.

[0073] In addition, as a parallel filtering process, Figure 4 and Figure 5 The five initial speeds shown are just an example, and other numbers of parallel filtering processes may be performed. The multiple initial speeds may be switched to appropriate speeds according to the vehicle's conditions (driving speed, type of road on which the vehicle is traveling, etc.).

[0074] Furthermore, the fact that the speed and distance obtained by the image processing device are used in ACC and AEB is also an example, and the speed and distance obtained by the image processing device in this embodiment can also be used in other controls.

[0075] In addition, Figure 1 and Figure 2 In the configuration diagram shown, control lines and information lines are only those lines that are considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. In fact, it can be considered that almost all components are connected to each other. In addition, when the image processing device of this example is constructed using an information processing device such as a computer, the program for implementing the image processing device can be prepared in a non-volatile memory or memory within the computer, and can also be placed in an external memory, IC card, SD card, optical disk or other recording medium, and can also be transmitted. Explanation of symbols

[0076] 100…image processing device, 100a…CPU, 100b…ROM, 100c…RAM, 100d…non-volatile memory, 100e…network interface, 100f…input unit, 100g…output unit, 101…camera (left camera), 102…camera (right camera), 102e…output unit, 110…image signal processing unit, 111…image capturing unit, 112…parallax calculation unit, 120…recognition processing unit, 121…detection object search processing unit, 122…object tracking processing unit, 123… Distance calculation unit, 124…speed calculation unit, 125…filter processing unit, 130…parallel filter processing unit, 131…initial value setting unit, 132…multiple filter processing units, 133…observation residual accumulation calculation unit, 134…initial value selection unit, 135…tracking determination judgment unit, 140…storage area, 141…image information storage unit, 142…detection object information storage unit, 143…speed initial value storage unit, 144…parallel filter state quantity storage unit, 145…observation residual cumulative value storage unit, 150…control signal output unit.

Claims

1. An image processing device comprising: a distance calculation unit, which calculates the distance to the detection object searched from the acquired image; a speed calculation unit that calculates the speed of the detection object; and A filter processing unit that performs filter processing on the distance calculated by the distance calculation unit and the speed calculated by the speed calculation unit, wherein the image processing device is characterized in that: The filter processing unit selects one filter process from among the plurality of filter processes after performing the plurality of filter processes for a predetermined period of time.

2. The image processing device according to claim 1, characterized in that The distance calculation unit calculates the distance based on parallax of a plurality of images obtained from a stereo camera.

3. The image processing device according to claim 2, characterized in that: The plurality of filtering processes are filtering processes that are executed while assigning different initial speeds to each of them.

4. The image processing device according to claim 3, characterized in that: The selected filtering process is a filtering process that compares the predicted distance of the initial speed after the predetermined period with the distance observed based on the parallax of a plurality of images and minimizes the difference in distance.

5. The image processing device according to claim 4, characterized in that: The filtering processing unit comprises: an initial speed setting unit for setting different initial speeds in a plurality of filtering processes; a plurality of filter processing units for performing filter processing at the plurality of initial speeds set by the initial speed setting unit to estimate the distance of each detection object being tracked; an observation residual accumulation calculation unit, which calculates the residuals between the distances processed by the plurality of filter processing units and the calculated value of the distance obtained by the distance calculation unit, and accumulates and adds the residuals to obtain an observation residual accumulation value; as well as An initial value selection unit selects a filtering process that minimizes the observation residual accumulation value obtained by the observation residual accumulation calculation unit.

6. An image processing method, characterized in that: include: Distance calculation processing, calculating the distance to the detection object searched from the acquired image; Speed ​​calculation processing, calculating the speed of the detected object; as well as The parallel filtering process performs filtering on the distance calculated by the distance calculation process and the speed calculated by the speed calculation process, and when performing filtering, after performing multiple filtering processes for a predetermined period of time, one filtering process is selected from the multiple filtering processes.

Images