Image processing device, image processing method and program

Abstract

To accurately calculate a distance to an object in an image.SOLUTION: An image processing device according to an embodiment is mounted on a vehicle, and comprises a control unit. The control unit calculates a positional variation of an object projected on the basis of image recognition onto virtual planes with the same height between a present frame and a previous frame. The control unit also calculates a distance change tolerance by converting a vehicle speed calculated from the positional variation to a distance. In addition, the control unit adjusts the distance to the object at this time by adjusting a distance variation which is a difference between a distance to the object at the previous time which has been already adjusted using the tolerance and the distance to the object at the present time using the tolerance.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The disclosed embodiments relate to an image processing device, an image processing method, and a program. [Background technology]

[0002] Conventionally, a technology has been proposed for determining whether a vehicle has run a red light, based on the light status (green, yellow, red, etc.) of a traffic light detected from information received by an optical beacon or an image captured by a camera, and the vehicle's operating status (e.g., braking) (see Patent Document 1, for example).

[0003] In this technology, when determining the illumination state of a traffic light using an image captured by a camera, it is necessary to detect the traffic light itself in the image. In such cases, a known method is to use a machine learning algorithm such as deep learning to extract the traffic light to be detected as a rectangular area called a bounding box.

[0004] Furthermore, when determining the illumination state of a traffic light, it is necessary to accurately calculate the distance from the vehicle to the traffic light based on the camera coordinates of the rectangular area. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-069051 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the conventional technology has room for further improvement in terms of accurately calculating the distance to a traffic light.

[0007] When calculating the distance to a traffic light based on the camera coordinates of a rectangular area, for example, if there is variation in the position of the extracted rectangular area, the distance calculated based on that position will also be unstable.

[0008] On the other hand, even if the position of the rectangular area itself is correct, if the vehicle body shakes, particularly in the vertical direction, the calculated distance will still be unstable.

[0009] This problem is not limited to traffic lights, but is a common problem when calculating the distance to any object in an image.

[0010] One aspect of the embodiment has been made in consideration of the above, and aims to provide an image processing device, an image processing method, and a program that can accurately calculate the distance to an object in an image. [Means for solving the problem]

[0011] According to one aspect of the embodiment, an image processing device is mounted on a vehicle and includes a control unit. The control unit calculates a position change amount of an object projected based on image recognition onto a virtual plane at the same height between a current frame and a previous frame. The control unit also calculates a distance change tolerance by converting the vehicle speed estimated from the position change amount into a distance. The control unit also adjusts the current distance to the object by adjusting the distance change amount, which is the difference between the previous distance to the object, which has been adjusted by the tolerance amount, and the current distance to the object, by using the tolerance amount. [Effects of the Invention]

[0012] According to one aspect of the embodiment, the distance to an object in an image can be calculated with high accuracy. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram (part 1) outlining an image processing method according to an embodiment. [Figure 2] FIG. 2 is a diagram (part 2) outlining the image processing method according to the embodiment. [Figure 3] FIG. 3 is a diagram (part 3) outlining the image processing method according to the embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of the configuration of the image processing device according to the embodiment. [Figure 5] FIG. 5 is an explanatory diagram (part 1) of the distance calculation process. [Figure 6] FIG. 6 is an explanatory diagram (part 2) of the distance calculation process. [Figure 7] FIG. 7 is an explanatory diagram (part 3) of the distance calculation process. [Figure 8] FIG. 8 is an explanatory diagram (part 4) of the distance calculation process. [Figure 9] FIG. 9 is a flowchart showing a processing procedure executed by the image processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of an image processing device, an image processing method, and a program disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

[0015] In the following description, the image processing device 10 according to the embodiment is assumed to be an in-vehicle device mounted on a vehicle V. The image processing device 10 is a device capable of detecting an object in an image based on an image captured by a camera 3 mounted on the vehicle V, such as a red light violation determination device or a drive recorder. In the present embodiment, the object is assumed to be a traffic light 300.

[0016] First, an overview of an image processing method according to an embodiment will be described with reference to Figs. 1 to 3. Fig. 1 is a diagram (part 1) outlining the image processing method according to an embodiment. Fig. 2 is a diagram (part 2) outlining the image processing method according to an embodiment. Fig. 3 is a diagram (part 3) outlining the image processing method according to an embodiment.

[0017] One method for calculating the distance to the traffic light 300 in the image captured by the camera 3 is to use the parallax of a stereo camera. However, this method requires multiple cameras, which poses a problem of increasing the system size.

[0018] There is also a method that utilizes the parallax in moving stereo with a monocular camera, i.e., the parallax between frames, but this method has the problem that the processing costs are high when accurately matching the target object, the traffic light 300, between different frames.

[0019] Therefore, in this embodiment, as a method for inexpensively calculating the distance to the traffic light 300 using a monocular camera, a method is adopted in which, assuming that the height of the traffic light 300 is relatively constant, the camera coordinates of a rectangular area are converted into world coordinates using camera parameters, and the distance on a virtual plane assuming the height of the traffic light 300 is calculated. This calculation method will be described using FIG. 1.

[0020] As shown in Figure 1, the image processing device 10 calculates a virtual vehicle position indicating the position of the vehicle V projected onto a virtual plane (hereinafter referred to as the "virtual plane") at the same height in time-series space, based on a rectangular area BB indicating the position of the traffic light 300 in an image 201 taken from the vehicle V.

[0021] First, in a coordinate system represented by three mutually perpendicular axes including a vertical axis, the image processing device 10 calculates the coordinates of the traffic light 300 in a virtual plane obtained by fixing the coordinate value of the vertical axis as world coordinates.

[0022] As shown in Figure 1, a position in space is represented by coordinate values ​​on the X, Y, and Z axes. The X, Y, and Z axes are perpendicular to each other. The Z axis is a vertical axis. The Y axis is parallel to the direction of travel of the vehicle V in Figure 1. The X axis is perpendicular to the Z and Y axes.

[0023] Here, the center of the rectangular area BB is set to be a distance calculation point 221. The image processing device 10 converts the two-dimensional coordinates (camera coordinates) of the distance calculation point 221 in the image 201 into three-dimensional world coordinates using the camera parameters. Then, the image processing device 10 calculates the distance to the distance calculation point 221 in the world coordinates as the distance (dist) from the vehicle V to the traffic light 300.

[0024] The camera parameters include internal and external parameters of the camera 3. The internal parameters include dot pitch, focal length, distortion (elevation image height TBL), etc. The external parameters include the coordinate values ​​of the X-axis, Y-axis, and Z-axis of the shooting position of the camera 3, and values ​​(PAN, ROLL, TILT) that represent the attitude of the camera 3.

[0025] Furthermore, as a parameter separate from the camera parameters, the image processing device 10 refers to a pre-specified height from the road surface of the traffic light 300. The height of the traffic light 300 is specified within the range of 5.0 m to 5.5 m, for example.

[0026] The camera parameters and the height of the traffic light 300 may be stored as parameter information in the storage unit 11, which will be described later.

[0027] Here, it is assumed that the pre-specified height of traffic light 300 is 5 m. In this case, the virtual plane is an XY plane when the coordinate value of the Z axis is fixed at 5 m. As a result, the virtual plane can be said to be a virtual plane obtained by fixing the coordinate value of the vertical axis in a coordinate system represented by three mutually orthogonal axes, including the vertical axis.

[0028] However, when calculating the distance to traffic light 300 based on the camera coordinates of rectangular area BB in this manner, if there is variation in the position of the extracted rectangular area BB, i.e., distance calculation point 221, the distance calculated based on such distance calculation point 221 will also become unstable.

[0029] This phenomenon is conceptually shown in Figure 2. That is, if the same traffic light 300 is extracted as rectangular areas BB1, BB2, and BB3 with different positions and sizes, as shown in Figure 2, there will be variation in the distances between them.

[0030] On the other hand, even if the position and size of the rectangular area BB are correct, if the body of the vehicle V sways, particularly in the vertical direction, the calculated distance will still be unstable.

[0031] Therefore, in the image processing method according to the embodiment, a position change amount of an object projected based on image recognition onto a virtual plane at the same height between the current and previous frames is calculated. Furthermore, in the image processing method according to the embodiment, a distance change tolerance is calculated by converting the vehicle speed estimated from the position change amount into a distance. Furthermore, in the image processing method according to the embodiment, a distance change amount, which is the difference between the previous distance to the object, which has already been adjusted by the tolerance amount, and the current distance to the object, is adjusted by the tolerance amount.

[0032] In other words, in the image processing method according to the embodiment, the control unit 12 (see FIG. 4) of the image processing device 10 performs a time-direction filtering process such as averaging on the distance (XY coordinates on the virtual plane) calculated using the above-described calculation method, taking into consideration variations in the position of the rectangular area BB. Then, the control unit 12 handles the time change (vector data) of the XY coordinates after the filtering process as trajectory information of the traffic light 300 moving relatively as seen from the vehicle V.

[0033] 3, the control unit 12 receives X and Y coordinates on a virtual plane as input, and calculates a target movement amount (step S1), estimates the vehicle speed (step S2), smooths the speed (step S3), and calculates a distance change allowance (step S4). The control unit 12 also calculates the movement amount from the previous time (step S5). Then, the control unit 12 adjusts the movement amount (step S6) based on the execution results of steps S4 and S5, and outputs the adjusted X and Y coordinates.

[0034] In step S1, the control unit 12 calculates the object movement amount, which is the movement amount [m] of the object, i.e., traffic light 300, by subtracting the distance [m] of the current frame from the distance [m] of the previous frame. The object movement amount can also be referred to as the "position change amount." In addition, in step S2, the control unit 12 estimates the host vehicle speed [km / h] by converting the object movement amount [m] calculated in step S1 into the host vehicle speed.

[0035] In step S3, the control unit 12 performs filtering to calculate an approximate speed of the vehicle estimated in step S2. For the filtering, an infinite impulse response (IIR) filter, which has a large effect of smoothing with a small-scale circuit, is used. This filtering reduces the influence of variations in the rectangular area BB and the shaking of the vehicle body.

[0036] In addition, in step S4, the control unit 12 converts the vehicle speed [km / h] after the IIR filter processing in step S3 into distance [m], thereby calculating the allowable distance change amount (hereinafter referred to as the ``distance change allowable amount'') [m].

[0037] In step S5, the control unit 12 calculates the amount of movement from the previous frame based on the difference between the current distance and the previous (adjusted) distance. The amount of movement from the previous frame can be rephrased as "amount of change in distance." In step S6, the control unit 12 appropriately adjusts the amount of movement from the previous frame calculated in step S5 using the allowable distance change calculated in step S4. Then, the control unit 12 calculates the distance to the traffic light 300 in the current frame based on the adjusted X and Y coordinates.

[0038] As described above, the image processing method according to the embodiment is an image processing method executed by the image processing device 10 mounted on the vehicle V, and includes calculating the amount of position change of an object projected based on image recognition onto a virtual plane at the same height between the current and previous frames. The image processing method according to the embodiment also includes calculating an allowable amount of distance change by converting the vehicle speed estimated from the amount of position change into a distance. The image processing method according to the embodiment also includes adjusting the current distance to the object by adjusting, by the allowable amount, the amount of distance change, which is the difference between the previous distance to the object, which has been adjusted by the allowable amount, and the current distance to the object.

[0039] Therefore, according to the image processing method of the embodiment, the distance to an object in an image can be calculated with high accuracy.

[0040] An example of the configuration of the image processing device 10 to which the image processing method according to the above-described embodiment is applied will now be described in more detail.

[0041] Fig. 4 is a block diagram showing an example of the configuration of an image processing device 10 according to an embodiment. Note that Fig. 4 shows only components necessary for explaining the features of this embodiment, and omits descriptions of general components.

[0042] In other words, the components shown in Figure 4 are conceptual functional components and do not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each block is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0043] In addition, in the description using FIG. 4, the description of components that have already been described may be simplified or omitted.

[0044] 4, the image processing device 10 includes a storage unit 11 and a control unit 12. The image processing device 10 is also connected to a camera 3 and an output unit 5.

[0045] The camera 3 is mounted on the vehicle V and is provided so as to be able to capture an image of the area in front of the vehicle V. Note that the camera 3 may be, for example, a 360-degree camera that is able to capture an image of the entire periphery of the vehicle V.

[0046] The output unit 5 is an output device for outputting data. The output unit 5 is realized by, for example, a display, a speaker, etc. The output unit 5 may also be an external device that executes information processing based on the data output by the image processing device 10.

[0047] The camera 3 and the output unit 5 are connected to the image processing device 10 via an in-vehicle network such as a Controller Area Network (CAN). The camera 3 and the output unit 5 may also be connected to the image processing device 10 wirelessly via Bluetooth (registered trademark), Wi-Fi (registered trademark), UWB (Ultra Wide Band), or the like.

[0048] The image processing device 10 is a computer mounted on the vehicle V, and as described above, is, for example, a red light ignition determination device. The image processing device 10 executes at least steps S1 to S6 described with reference to FIG.

[0049] The storage unit 11 is realized by a storage device such as a RAM (Random Access Memory) or a flash memory, and in the example of FIG. 4, stores an image recognition AI 11a and processing parameter information 11b.

[0050] The image recognition AI 11a is an AI (Artificial Intelligence) model for image recognition. Specifically, the image recognition AI 11a is a DNN (Deep Neural Network) model or the like that is trained using a machine learning algorithm. The image recognition AI 11a is loaded as a DNN model into an image recognition unit 12a (described later), and is then configured to be able to distinguish various objects contained in an image captured by a camera 3 when the image is input to the image recognition unit 12a. Furthermore, the image recognition AI 11a is further configured to be able to distinguish the light emission state of a traffic light 300 when a traffic light 300 is identified as an object.

[0051] The processing parameter information 11b is information about various parameters used in the distance calculation process executed by the distance calculation unit 12b, which will be described later. The processing parameter information 11b includes the camera parameters described above, the height of the traffic light 300, and the like.

[0052] The control unit 12 is a controller, and is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, executing a program according to the embodiment stored in the storage unit 11 using RAM as a work area. The control unit 12 can also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0053] The control unit 12 has an image recognition unit 12a, a distance calculation unit 12b, and a determination unit 12c, and realizes or executes the functions and actions of information processing described below.

[0054] The image recognition unit 12a acquires the captured image captured by the camera 3 and inputs the captured image to the image recognition AI 11a that has read the image. Furthermore, the image recognition unit 12a acquires the image recognition result output from the image recognition AI 11a as a result of inputting the captured image, and outputs the acquired image recognition result to the distance calculation unit 12b and the determination unit 12c. The image recognition result includes the traffic light 300 extracted as a rectangular area BB.

[0055] The camera 3 captures moving images, and the image recognition unit 12a extracts a rectangular area BB from each frame of the moving images.

[0056] Based on the processing parameter information 11b, the distance calculation unit 12b converts the coordinate values ​​in the camera coordinate system of the distance calculation point 221, which is the center of the rectangular area BB, into coordinate values ​​in the world coordinate system on the virtual plane.

[0057] Furthermore, the distance calculation unit 12b receives the X and Y coordinates from the converted coordinate values ​​as input and executes the above-mentioned steps S1 to S6. This will be described in detail with reference to Figs. 5 to 8. Fig. 5 is an explanatory diagram (part 1) of the distance calculation process. Fig. 6 is an explanatory diagram (part 2) of the distance calculation process. Fig. 7 is an explanatory diagram (part 3) of the distance calculation process. Fig. 8 is an explanatory diagram (part 4) of the distance calculation process.

[0058] 5, distance calculation unit 12b calculates the movement amount (target movement amount) of traffic light 300 by subtracting the distance in the current frame (current distance) from the distance in the previous frame (previous distance) (step S1 described above). The previous distance corresponds to the position of traffic light 300 on the virtual plane in the previous frame. The current distance corresponds to the position of traffic light 300 on the virtual plane in the current frame.

[0059] At this time, since the object (traffic light 300) is stationary, the object movement amount appears relatively as the host vehicle movement amount in the opposite direction.

[0060] Then, the distance calculation unit 12b estimates the vehicle speed by converting the calculated target movement amount into the vehicle speed (step S2 described above). Specifically, the distance calculation unit 12b estimates the vehicle speed using the formula "vehicle speed [km / h] = target movement amount [m] × (60 × 60 / operation cycle [ms])". The operation cycle is, for example, 100 [ms].

[0061] Then, the distance calculation unit 12b performs a filter process on the estimated vehicle speed to calculate an approximate speed. In this filter process, the distance calculation unit 12b uses an IIR filter configured as shown in Fig. 6. The coef (coefficient), i.e., the adjustment coefficient, is "5", for example.

[0062] By performing such filtering, the distance calculation unit 12b can estimate the vehicle speed smoothed in the time direction, as shown in Fig. 7. This makes it possible to suppress variations in the position and size of the rectangular area BB itself, or variations in the vehicle speed caused by changes in the posture of the vehicle body (i.e., camera 3) between frames. In particular, changes in TILT (vertical angle) have a significant effect on the accuracy of depth distance, so this is effective in maintaining such accuracy.

[0063] Then, the distance calculation unit 12b calculates the allowable distance change amount by converting the host vehicle speed after the IIR filter processing into a distance (step S4 described above). Specifically, the distance calculation unit 12b calculates the allowable distance change amount using the formula "allowable distance change amount [m] = host vehicle speed after the IIR filter processing [km / h] × (operation cycle [ms] / (60 × 60)) × adjustment coefficient."

[0064] The operating cycle is, for example, 100 ms. The adjustment coefficient is, for example, 1.5. Note that the host vehicle speed is the moving distance per unit time (host vehicle movement amount), and as shown in Figure 5, "host vehicle movement amount = target movement amount", so if the host vehicle speed can be estimated, the distance the target can move (allowable distance change amount) can also be determined from that.

[0065] 8, the distance calculation unit 12b calculates the amount of movement from the previous time (before adjustment) from the difference between the current distance before adjustment and the previous distance after adjustment (step S5 described above).Then, the distance calculation unit 12b adjusts the calculated amount of movement from the previous time [A] using the allowable distance change amount [B] (step S6 described above).

[0066] At this time, distance calculation unit 12b calculates an adjustment ratio [B / A] if the amount of movement [A] since the previous time is greater than the distance change allowance [B], as shown in the center diagram of Fig. 8. Then, distance calculation unit 12b adjusts the current distance by adjusting the amount of movement [A] since the previous time by the adjustment ratio [B / A], as shown in the right diagram of Fig. 8.

[0067] Returning to the explanation of Fig. 4, the distance calculation unit 12b then outputs the calculated adjusted current distance to the determination unit 12c. The determination unit 12c determines whether or not the vehicle V has run a red light based on the image recognition result of the image recognition unit 12a and the calculation result of the distance calculation unit 12b.

[0068] The determination unit 12c estimates the behavior of the vehicle V based on, for example, the trajectory of each object recognized by image recognition. The determination unit 12c also estimates the trajectory of the traffic light 300 relative to the vehicle V based on the traffic light 300 recognized by image recognition and the distance calculated by the distance calculation unit 12b.

[0069] The determination unit 12c also acquires the light emission state of the traffic light 300 determined by the image recognition unit 12a. The determination unit 12c also determines whether the vehicle V has run a red light, based on the estimated behavior of the vehicle V, the trajectory of the traffic light 300, and the light emission state of the traffic light 300.

[0070] For example, if the light of the traffic light 300 indicating whether or not there is a right of way in the traveling direction of the vehicle V is red, and the vehicle V continues to pass for a predetermined time or longer at a predetermined speed or faster, the determination unit 12c determines that the vehicle V has ignored the traffic light. The determination unit 12c outputs the determination result to the output unit 5.

[0071] In addition, when the image processing device 10 constitutes part of the function of a drive recorder, the determination unit 12c may notify the determination result that the vehicle V has ignored a traffic light as an event and cause the drive recorder to record the event.

[0072] Next, a processing procedure executed by the image processing device 10 according to the embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the processing procedure executed by the image processing device 10 according to the embodiment.

[0073] 9, the control unit 12 of the image processing device 10 acquires a current frame image from the camera 3 (step S101). Then, the control unit 12 performs image recognition on the acquired frame image (step S102).

[0074] Then, the control unit 12 calculates the amount of movement of the object recognized by the image (step S103), and estimates the vehicle speed and smoothes it using an IIR filter (step S104).

[0075] Then, the control unit 12 converts the estimated vehicle speed into a distance and calculates it as the allowable distance change amount (step S105).

[0076] Furthermore, the control unit 12 calculates the amount of movement from the previous time from the difference between the current distance and the adjusted previous distance on the virtual plane (step S106).

[0077] Then, the control unit 12 adjusts the amount of movement from the previous time calculated in step S106 using the distance change allowance calculated in step S105, and calculates the adjusted current distance (step S107).Then, the control unit 12 repeats the process from step S101.

[0078] As described above, the image processing device 10 according to the embodiment is an image processing device mounted on a vehicle V and includes a control unit 12. The control unit 12 calculates the amount of position change of an object projected based on image recognition onto a virtual plane at the same height between the current and previous frames. The control unit 12 also calculates the allowable amount of distance change by converting the vehicle speed estimated from the amount of position change into a distance. The control unit 12 also adjusts the current distance to the object by adjusting the amount of distance change, which is the difference between the previous distance to the object, which has been adjusted by the allowable amount, and the current distance to the object, by using the allowable amount.

[0079] Therefore, the image processing device 10 according to the embodiment can accurately calculate the distance to an object in an image.

[0080] Furthermore, the control unit 12 smoothes the vehicle speed in the time direction and converts the smoothed vehicle speed into a distance to calculate the allowable amount.

[0081] Therefore, the image processing device 10 according to the embodiment can suppress variations in the position and size of the rectangular area BB itself, or variations in the vehicle speed caused by changes in the posture of the vehicle body (i.e., the camera 3) between frames. Furthermore, the allowable amount can be calculated based on the vehicle speed with the variations suppressed.

[0082] Furthermore, the control unit 12 smoothes the vehicle speed using an IIR filter.

[0083] Therefore, the image processing device 10 according to the embodiment can effectively smooth the vehicle speed at low cost using a small-scale circuit. Furthermore, this filtering process can reduce the influence of variations in the rectangular area BB and the shaking of the vehicle body.

[0084] Furthermore, the control unit 12 estimates the vehicle speed by converting the amount of change in position into the vehicle speed.

[0085] Therefore, the image processing device 10 according to the embodiment can estimate the vehicle speed by image recognition.

[0086] Furthermore, the control unit 12 performs image recognition of at least the traffic light 300 as the object, and performs image recognition of the light emission state of the traffic light 300.

[0087] Therefore, the image processing device 10 according to the embodiment can recognize the image of the traffic light 300 and its light emission state.

[0088] The control unit 12 also estimates the trajectory of the traffic light 300 relative to the vehicle V based on the image-recognized traffic light 300 and the calculated distance to the traffic light 300.

[0089] Therefore, according to the image processing device 10 according to the embodiment, the trajectory of the traffic light 300 can be estimated with high accuracy by image recognition.

[0090] Furthermore, the control unit 12 estimates the behavior of the vehicle V based on the trajectory of the object.

[0091] Therefore, according to the image processing device 10 according to the embodiment, the behavior of the vehicle V can be estimated based on the movement of each object in the image including the traffic light 300.

[0092] Furthermore, the control unit 12 determines whether or not the vehicle V has run a red light based on the behavior of the vehicle V, the path of the traffic light 300, and the light emission state.

[0093] Therefore, the image processing device 10 according to the embodiment can determine whether or not the vehicle V has run a red light through image recognition.

[0094] Furthermore, an image processing method according to the embodiment is an image processing method executed by an image processing device 10 mounted on a vehicle V, and includes calculating a position change amount of an object projected based on image recognition onto a virtual plane at the same height between a current frame and a previous frame. The image processing method according to the embodiment also includes calculating an allowable distance change amount by converting the vehicle speed estimated from the position change amount into a distance. The image processing method according to the embodiment also includes adjusting the current distance to the object by adjusting, by the allowable amount, a distance change amount that is the difference between the previous distance to the object, which has been adjusted by the allowable amount, and the current distance to the object.

[0095] Therefore, according to the image processing method of the embodiment, the distance to an object in an image can be calculated with high accuracy.

[0096] The program according to the embodiment also causes the image processing device 10 (corresponding to an example of a "computer") to calculate the amount of change in position of an object projected based on image recognition onto a virtual plane at the same height between the current and previous frames. The program according to the embodiment also causes the image processing device 10 to calculate an allowable amount of distance change by converting the vehicle speed estimated from the amount of change in position into a distance. The program according to the embodiment also causes the image processing device 10 to adjust the current distance to the object by adjusting, by the allowable amount, the amount of distance change, which is the difference between the previous distance to the object, which has been adjusted by the allowable amount, and the current distance to the object.

[0097] Therefore, according to the program of the embodiment, the distance to the object in the image can be calculated with high accuracy.

[0098] In the above-described embodiment, the distance to the object, i.e., traffic light 300, is calculated and the disregard for the traffic light is determined by image recognition, but it is of course possible to appropriately combine sensor data from various sensors mounted on the vehicle V. For example, the behavior of the vehicle V may be estimated using the sensor values ​​of a steering sensor or an acceleration sensor, and the vehicle speed may be obtained using the sensor value of a speed sensor.

[0099] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0100] 3 Camera 5 Output section 10 Image processing device 11 Storage section 11a Image recognition AI 11b Processing parameter information 12 Control Unit 12a Image recognition unit 12b Distance calculation section 12c Judgment part V vehicle

Claims

1. An image processing device mounted on a vehicle, the image processing device having a control unit, The control unit Calculating the amount of change in position of the object projected based on image recognition onto a virtual plane at the same height between the current and previous frames; calculating an allowable amount of distance change by converting the vehicle speed estimated from the amount of change in position into a distance; a distance change amount, which is a difference between the previous distance to the object, which has been adjusted using the tolerance, and the current distance to the object, is adjusted using the tolerance, thereby adjusting the current distance to the object; Image processing device.

2. The control unit smoothing the host vehicle speed in the time direction and converting the smoothed host vehicle speed into a distance to calculate the allowable amount; The image processing device according to claim 1 .

3. The control unit smoothing the host vehicle speed using an IIR filter; The image processing device according to claim 2 .

4. The control unit The vehicle speed is estimated by converting the position change amount into the vehicle speed.

4. The image processing device according to claim 1, 2 or 3.

5. The control unit image-recognizing at least a traffic light as the object and image-recognizing the light emission state of the traffic light; 5. The image processing device according to claim 1.

6. The control unit estimating a trajectory of the traffic light relative to the vehicle based on the image-recognized traffic light and the calculated distance to the traffic light; The image processing device according to claim 5 .

7. The control unit Estimating the behavior of the vehicle based on the trajectory of the object; 7. The image processing device according to claim 5 or 6.

8. The control unit determining whether the vehicle has run a red light based on the behavior of the vehicle, the trajectory of the traffic light, and the light emission state; The image processing device according to claim 7 .

9. An image processing method executed by an image processing device mounted on a vehicle, comprising: Calculating a position change amount of an object projected based on image recognition onto a virtual plane at the same height between the current frame and the previous frame; calculating an allowable amount of distance change by converting the vehicle speed estimated from the amount of change in position into a distance; adjusting the current distance to the object by adjusting, by the allowable amount, a distance change amount that is a difference between the previous distance to the object that has been adjusted by the allowable amount and the current distance to the object; An image processing method comprising:

10. Calculating the amount of change in position of the object projected based on image recognition onto a virtual plane at the same height between the current frame and the previous frame; calculating an allowable amount of distance change by converting the vehicle speed estimated from the amount of change in position into a distance; adjusting the current distance to the object by adjusting, using the tolerance, a distance change amount that is a difference between the previous distance to the object that has been adjusted using the tolerance and the current distance to the object; A program that causes a computer to execute the following.

Images