Object detection device

Abstract

To provide an object detection device that can estimate a state around a vehicle with high accuracy by using a state space model.SOLUTION: An object detection device includes: an image acquisition unit that acquires an image around a vehicle photographed by an imaging device; a representative point estimation unit that estimates a representative point that can be regarded as a ground position of a specific object existing around the vehicle in the photographed image by image analysis processing of the photographed image; a coordinate calculation unit that calculates, on the basis of the estimation result by the representative point estimation unit and an internal parameter of the imaging device, relative coordinates, which are coordinates of the representative point in a three-dimensional relative coordinate system corresponding to a space based on a specific position of the vehicle; a coordinate conversion unit that converts, on the basis of movement information indicating a movement trajectory of the vehicle, relative coordinates into absolute coordinates, which are coordinates in the three-dimensional absolute coordinate system that corresponds to a space based on a specific position on the earth; and a state estimation unit that estimates a state around the vehicle by inputting the absolute coordinates into a state space model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an object detection device. [Background technology]

[0002] Conventionally, as a technology for detecting objects around a vehicle, an object detection device has been proposed that performs object detection (for example, detection of obstacles such as people and other vehicles) on images captured by an on-board camera. The object detection device can detect the presence or absence of an object, detect the distance to the object, calculate the three-dimensional coordinates of the object, and, if the object is moving, identify the moving speed and moving direction of the object, and these detection results can be used for vehicle control. As such an object detection device, a technology has been disclosed in which the position of a person (pedestrian) included in an image captured by an on-board camera is estimated as coordinates in a coordinate system (relative coordinate system) that is set with the position of the on-board camera as the origin and displaces together with the vehicle, and values ​​based on the coordinates are input to a state estimator (state space model) such as a Kalman filter, thereby tracking the person. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4751429 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, if the position of an object such as a person is estimated as coordinates in a relative coordinate system that moves together with the vehicle, and the coordinates in the relative coordinate system are input into a state space model to estimate the state around the vehicle, it becomes difficult to estimate the state with high accuracy when the vehicle or object is moving at high speed, for example.

[0005] The present invention has been made in view of the above, and provides an object detection device capable of estimating the state around a vehicle with high accuracy by using a state space model. [Means for solving the problem]

[0006] An object detection device as one aspect of the present invention includes an image acquisition unit that acquires an image of the situation around the vehicle captured by an imaging device, a representative point estimation unit that estimates a representative point that can be regarded as the ground contact position of a specific object present around the vehicle in the captured image by image analysis processing of the captured image, a coordinate calculation unit that calculates relative coordinates that are the coordinates of the representative point in a three-dimensional relative coordinate system corresponding to a space based on a specific position of the vehicle based on the estimation result by the representative point estimation unit and internal parameters of the imaging device, a coordinate conversion unit that converts the relative coordinates into absolute coordinates that are coordinates in a three-dimensional absolute coordinate system corresponding to a space based on a specific position on the earth based on movement information indicating the vehicle's movement trajectory, a state estimation unit that estimates the state around the vehicle by inputting the absolute coordinates into a state space model, and an output unit that outputs the estimation result by the state estimation unit. Effect of the Invention

[0007] According to the present invention, it is possible to provide an object detection device that can estimate the state around a vehicle with high accuracy by using a state space model. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a vehicle on which an object detection device according to an embodiment is mounted. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a control system including an object detection device mounted on a vehicle according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional configuration of the object detection device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a captured image and a region of interest according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a representative point estimated from a region of interest according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a three-dimensional relative coordinate system and a three-dimensional absolute coordinate system according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a situation in which the state estimation unit of the embodiment continues estimating a state during a period in which a representative point has not been acquired. [Figure 8] FIG. 8 is a diagram illustrating an example of a state estimation method in the state estimation unit according to the embodiment. [Figure 9] FIG. 9 is a flowchart illustrating an example of a process in the object detection device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The configuration of the embodiment described below and the actions and effects brought about by the configuration are merely examples, and the present invention is not limited to the following description.

[0010] The object detection device of this embodiment is a device that estimates the state of the surroundings of a vehicle based on an image captured by an imaging device mounted on the vehicle.

[0011] FIG. 1 is a diagram showing an example of a vehicle 10 equipped with an object detection device according to an embodiment. The vehicle 10 may be, for example, an automobile using an internal combustion engine as a drive source (internal combustion engine automobile), an automobile using an electric motor as a drive source (electric automobile, fuel cell automobile, etc.), or an automobile using both of them as drive sources (hybrid automobile). The vehicle 10 may be equipped with various transmissions, and may be equipped with various devices (systems, parts, etc.) required to drive the internal combustion engine or the electric motor. The type, number, layout, etc. of devices related to the drive of the wheels 12 (front wheels 12F, rear wheels 12R) of the vehicle 10 may be set in various ways.

[0012] As illustrated in FIG. 1, the vehicle 10 is provided with a plurality of imaging devices 14, for example, four imaging devices 14a to 14d. The imaging devices 14 are digital cameras incorporating imaging elements such as a charge coupled device (CCD) or a CMOS image sensor (CIS). The imaging devices 14 can output image data (video data) at a predetermined frame rate. Each of the imaging devices 14 has a wide-angle lens (for example, a fisheye lens) and can capture an image in a range of, for example, 140° to 220° in the horizontal direction. In addition, for example, the optical axis of the imaging devices 14 (14a to 14d) arranged on the outer periphery of the vehicle 10 may be set to face diagonally downward. The imaging devices 14 (14a to 14d) sequentially capture images of the situation around the vehicle 10 and output image data of images captured of the road surface and objects on the road surface. The objects on the road surface may include, for example, people (pedestrians), other vehicles, buildings, road structures, road markings, etc.

[0013] The imaging device 14a is provided, for example, on the front side of the vehicle 10, i.e., on the front side in the vehicle longitudinal direction and at an end portion substantially in the center in the vehicle width direction, such as the front bumper 10a or the front grille, and can capture a forward image including the front end portion of the vehicle 10 (for example, the front bumper 10a). The imaging device 14b is provided, for example, on the rear side of the vehicle 10, i.e., on the rear side in the vehicle longitudinal direction and at an end portion substantially in the center in the vehicle width direction, such as an upper position of the rear bumper 10b, and can capture an image of the rear area including the rear end portion of the vehicle 10 (for example, the rear bumper 10b). The imaging device 14c is provided, for example, on the right end portion of the vehicle 10, such as the right door mirror 10c, and can capture a right side image including an area centered on the right side of the vehicle 10 (for example, an area from the right front to the right rear). The imaging device 14d is provided, for example, on the left end of the vehicle 10, such as the left door mirror 10d, and is capable of capturing a left side image including an area centered on the left side of the vehicle 10 (for example, the area from the left front to the left rear).

[0014] For example, by performing arithmetic processing or image processing on each image data obtained by the imaging devices 14a to 14d, it is possible to display images in each direction around the vehicle 10 and perform periphery monitoring. Furthermore, by performing arithmetic processing or image processing based on each image data, it is possible to generate images with a wider viewing angle, generate and display virtual images (bird's-eye view images (planar images), side-view images, front-view images, etc.) of the vehicle 10 viewed from above, in front, to the side, etc., and perform periphery monitoring.

[0015] As described above, image data captured by each imaging device 14 is displayed on a display device in the vehicle cabin to provide a user, such as a driver, with information about the situation around the vehicle 10. The image data is also used to estimate the state around the vehicle 10, and the estimation result indicating the estimated state is used to control the vehicle 10. The estimation result may include, for example, the presence or absence of an object (e.g., a person or another vehicle present around the vehicle 10), the position of the object, the distance from the vehicle 10 to the object, the moving direction and moving speed of the object, etc.

[0016] FIG. 2 is a diagram showing an example of the configuration of a control system 100 including an object detection device mounted on a vehicle 10 of the first embodiment. A display device 16, an audio output device 18, and the like are provided in the cabin of the vehicle 10. The display device 16 is, for example, a liquid crystal display (LCD) or an organic electroluminescent display (OLED). The audio output device 18 is, for example, a speaker. The display device 16 is covered with a transparent operation input unit 20 such as a touch panel. A user (for example, a driver) can visually recognize an image displayed on a display screen of the display device 16 through the operation input unit 20. The user can also perform an operation input by touching, pressing, or moving the operation input unit 20 with a finger or the like at a position corresponding to an image displayed on the display screen of the display device 16. The display device 16, the audio output device 18, the operation input unit 20, and the like are provided in a monitor device 22 located, for example, in the center of the dashboard of the vehicle 10 in the vehicle width direction, i.e., the left-right direction. The monitor device 22 may have an appropriate operation input unit such as a switch, a dial, a joystick, a push button, etc. The monitor device 22 may also function as, for example, a navigation system or an audio system.

[0017] As illustrated in FIG. 2, the control system 100 (including the object detection device) includes an ECU 24 (Electronic Control Unit), a wheel speed sensor 26, a steering angle sensor 28, a shift sensor 30, a driving support unit 32, and the like, in addition to the imaging device 14 (14a to 14d) and the monitor device 22. In the control system 100, the ECU 24, the monitor device 22, the wheel speed sensor 26, the steering angle sensor 28, the shift sensor 30, the driving support unit 32, and the like are electrically connected via an in-vehicle network 34 as an electric communication line. The in-vehicle network 34 is configured as, for example, a CAN (Controller Area Network). The ECU 24 can control various systems by sending control signals through the in-vehicle network 34. The ECU 24 can also receive operation signals from the operation input unit 20 and various switches, and detection signals from various sensors such as the wheel speed sensor 26, the steering angle sensor 28, and the shift sensor 30, and the like, via the in-vehicle network 34. It should be noted that various systems (such as a steering system, a braking system, a drive system, etc.) and various sensors for driving the vehicle 10 are connected to the in-vehicle network 34, but in FIG. 2, configurations that are less relevant to the object detection device of this embodiment are not shown in the figure and will not be described.

[0018] The ECU 24 is configured with a computer or the like, and the hardware and software work together to perform at least a part of the control of the vehicle 10. Specifically, the ECU 24 includes a CPU (Central Processing Unit) 24a, a ROM (Read Only Memory) 24b, a RAM (Random Access Memory) 24c, a display control unit 24d, an audio control unit 24e, and an SSD (Solid State Drive) 24f.

[0019] The CPU 24a reads out a program stored (installed) in a non-volatile storage device such as the ROM 24b, and executes arithmetic processing according to the program. The CPU 24a executes, for example, image analysis processing on an image captured by the imaging device 14, and generates an estimation result indicating the state of the surroundings of the vehicle 10. The estimation result is used, for example, to control a steering system, a braking system, a drive system, and the like.

[0020] The ROM 24b stores each program and parameters required for executing the program. The RAM 24c is used as a work area when the CPU 24a executes various arithmetic processes, and is also used as a temporary storage area for various data (such as image data captured sequentially (in time series) by the imaging device 14) used in the arithmetic processes of the CPU 24a. The display control unit 24d mainly executes processing of image data acquired from the imaging device 14 and output to the CPU 24a, and converts image data acquired from the CPU 24a into image data for display on the display device 16, among the arithmetic processes of the ECU 24. The audio control unit 24e mainly executes processing of audio acquired from the CPU 24a and output to the audio output device 18, among the arithmetic processes of the ECU 24. The SSD 24f is a rewritable non-volatile storage unit, and continues to store data acquired from the CPU 24a even when the power supply of the ECU 24 is turned off. The CPU 24a, the ROM 24b, the RAM 24c, and the like can be integrated in the same package. The ECU 24 may be configured to use other processors or logic circuits such as a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 24a. A HDD (Hard Disk Drive) may be provided instead of or in addition to the SSD 24f, or the SSD 24f and the HDD may be provided separately from the ECU 24.

[0021] The wheel speed sensor 26 is a sensor that detects the amount of rotation of the wheel 12 and the number of rotations per unit time. The wheel speed sensor 26 is arranged on each wheel 12, and outputs the number of wheel speed pulses indicating the number of rotations detected at each wheel 12 as a sensor value. The wheel speed sensor 26 may be configured using, for example, a Hall element. The CPU 24a calculates the vehicle speed, acceleration, etc. of the vehicle 10 based on the detection value acquired from the wheel speed sensor 26, and executes various controls.

[0022] The steering angle sensor 28 is a sensor that detects the amount of steering of a steering part such as a steering wheel. The steering angle sensor 28 can be configured using, for example, a Hall element, etc. The CPU 24a acquires the amount of steering of the steering part by the driver, the amount of steering of the front wheels 12F during automatic steering when performing parking assistance, etc. from the steering angle sensor 28 and performs various controls.

[0023] The shift sensor 30 is a sensor that detects the position of the movable parts (bar, arm, button, etc.) of the gear shift operating unit, and detects information indicating the operating state of the transmission, the state of the gear stage, and the direction in which the vehicle 10 can travel (D range: forward direction, R range: reverse direction), etc.

[0024] The driving support unit 32 provides control information to the steering system, the brake system, the drive system, etc., in order to realize driving support for moving the vehicle 10 based on the moving route calculated by the control system 100 or the moving route provided from the outside. For example, the driving support unit 32 executes fully automatic control for automatically controlling all of the steering system, the brake system, the drive system, etc., or executes semi-automatic control for automatically controlling part of the steering system, the brake system, the drive system, etc. In addition, the driving support unit 32 may execute manual control for causing the driver to perform driving operations by providing operation guides for the steering system, the brake system, the drive system, etc. to the driver so that the vehicle 10 can move along the moving route. In this case, the driving support unit 32 may provide operation information to the display device 16 and the audio output device 18. In addition, even when executing semi-automatic control, the driving support unit 32 can provide information on the driver to operate, for example, accelerator operation, etc., via the display device 16 and the audio output device 18.

[0025] FIG. 3 is a diagram showing an example of the functional configuration of the object detection device of the embodiment. Here, a case where the functional configuration of the object detection device is realized by the CPU 24a is illustrated. The CPU 24a executes a program read from the ROM 24b to realize modules such as an image acquisition unit 101, a representative point estimation unit 102, a coordinate calculation unit 103, a coordinate conversion unit 104, a state estimation unit 105, and an output unit 106, as shown in FIG. 3. Note that the manner for realizing such functional configuration is not limited to this. For example, at least a part of the functional units 101 to 106 may be configured by dedicated hardware (circuits, etc.), or may be configured using a processor other than the CPU 24a. Furthermore, the CPU 24a may realize various modules necessary for controlling the vehicle 10 other than the functional units 101 to 106.

[0026] The image acquisition unit 101 acquires captured images showing the surroundings of the vehicle 10 including the road surface on which the vehicle 10 is located, captured by the imaging devices 14. The image acquisition unit 101 may sequentially acquire captured images captured by each imaging device 14 (14a to 14d) and provide them to the representative point estimation unit 102. The image acquisition unit 101 may selectively acquire captured images of possible travel directions so that object detection in the possible travel directions can be performed based on information on the possible travel directions (forward direction or backward direction) of the vehicle 10 that can be acquired from the shift sensor 30 and information on the turning direction of the vehicle 10 that can be acquired from the steering angle sensor 28.

[0027] The representative point estimation unit 102 estimates a representative point that can be regarded as a ground contact position of a specific object existing around the vehicle 10 in the captured image by performing image analysis processing on the captured image acquired by the image acquisition unit 101. The specific object can be, for example, a person (pedestrian), another vehicle, a road structure, etc. A specific method for estimating a representative point from a captured image may be appropriately determined according to the specifications of the imaging device 14 and the vehicle 10, and for example, the representative point can be estimated using a trained model generated by machine learning (deep learning) performed in advance. More specifically, for example, a region of interest surrounding an area in the captured image where a person (an example of a specific object) can be considered to exist is set, and image data and feature amounts of the region of interest are input to the trained model to estimate a representative point that can be regarded as a ground contact position (a contact point between a person and a road surface) of the person. The region of interest may be set by appropriately using a publicly known or new technology, and may be set based on an estimation result by an appropriate trained model, or may be set by an algorithm that does not use a trained model, such as rule-based processing.

[0028] Fig. 4 is a diagram showing an example of a captured image 50 and a region of interest 52 according to the embodiment. Fig. 4 shows an example of a captured image 50 that is a wide-angle image captured by an imaging device 14d installed on the left side of a vehicle 10. The captured image 50 includes an image of the side of the vehicle 10, an image of a plurality of people 54 present on the road, and a plurality of regions of interest 52 that are set to surround each of the people 54.

[0029] The attention area 52 exemplified here is a rectangular area (bounding box) set so as to surround a person, and the size (width×height) of each attention area 52 is set according to the size of an object regarded as a person 54 in the captured image 50 by an appropriate algorithm. The attention area 52 can be defined, for example, by specifying the center coordinates, width and height of the attention area 52 on the captured image 50. Note that, although the attention area 52 set so as to surround a person is mentioned here, the attention area may be set so as to surround a predetermined object other than a person (for example, another vehicle, a road structure, etc.).

[0030] Fig. 5 is a diagram showing an example of a representative point P estimated from a region of interest 52 according to the embodiment. In Fig. 5, a portion corresponding to one region of interest 52 is excerpted from a captured image 50 and illustrated. The region of interest 52 illustrated here is illustrated tilted so that an image of a person 54 therein stands substantially upright.

[0031] The representative point P exemplified here is the intersection point between a normal line 56 extending from a reference point 55 defined within the attention area 52 to the road surface and the frame line of the attention area 52. The reference point 55 is, for example, a point corresponding to the abdomen of the person 54, and may be the center point or center of gravity of the attention area 52, or a point in the vicinity thereof. Note that the above-mentioned method of estimating the representative point P is merely an example, and the method of estimating the representative point P is not limited to the above. For example, the representative point P may be estimated using a trained model capable of directly estimating the representative point P from the captured image 50 (image data, feature amounts, etc. of the captured image 50) without setting the attention area 52 as described above.

[0032] Returning to FIG. 3, the coordinate calculation unit 103 calculates relative coordinates, which are the coordinates of the representative point P in the three-dimensional relative coordinate system, based on the estimation result by the representative point estimation unit 102 and the internal parameters of the imaging device 14. In other words, the coordinate calculation unit 103 converts the coordinates of the representative point P in the captured image 50 into relative coordinates in the three-dimensional relative coordinate system based on the internal parameters of the imaging device 14 and the like. The three-dimensional relative coordinate system is a three-dimensional coordinate system corresponding to a space based on a specific position of the vehicle 10, and displaces with the movement of the vehicle 10. The specific position of the vehicle 10 can be set arbitrarily, and may be, for example, the midpoint of both rear wheels 12L and 12R, the installation position of the imaging device 14, or the like.

[0033] The coordinate conversion unit 104 converts the relative coordinates calculated by the coordinate calculation unit 103 into absolute coordinates, which are the coordinates of the representative point P in a three-dimensional absolute coordinate system, based on the movement information indicating the movement trajectory of the vehicle 10. The three-dimensional absolute coordinate system is a three-dimensional coordinate system corresponding to a space based on a specific position on the earth, and is fixed to a certain position regardless of the movement of the vehicle 10. The specific position on the earth can be set arbitrarily, but may be, for example, the position of the vehicle 10 at the start of operation of the object detection device, or the position of the vehicle 10 at a predetermined time before the time when the absolute coordinates were calculated.

[0034] 6 is a diagram showing an example of a three-dimensional relative coordinate system and a three-dimensional absolute coordinate system according to the embodiment. The three-dimensional relative coordinate system is a three-dimensional coordinate system having an origin O' at a specific position of the vehicle 10 (in this example, the midpoint of both rear wheels of the vehicle 10) and consisting of an X' axis, a Y' axis, and a Z' axis which are perpendicular to each other. The origin O' can be set arbitrarily, and may be, for example, the installation position of the imaging device 14. The X' axis corresponds to the width direction of the vehicle 10, the Y' axis corresponds to the height direction of the vehicle 10, and the Z' axis corresponds to the front-rear direction of the vehicle 10.

[0035] The three-dimensional absolute coordinate system is a three-dimensional coordinate system with a specific position on the earth as the origin O and consisting of an X-axis, a Y-axis, and a Z-axis that are mutually perpendicular. The origin O can be set arbitrarily, and may be, for example, a specific position (such as the midpoint of both rear wheels) of the vehicle 10 at the start of operation of the object detection device (software for operating it), or a specific position of the vehicle 10 at a time point that is a predetermined time before the time point (period) at which the absolute coordinates are calculated. The X-axis corresponds to the width direction of the vehicle 10 (dashed line) when the vehicle 10 is located at the origin 0, the Y-axis corresponds to the height direction of the vehicle 10 when the vehicle 10 is located at the origin 0, and the Z-axis corresponds to the front-rear direction of the vehicle 10 when the vehicle 10 is located at the origin 0.

[0036] 6 illustrates an example in which the vehicle 10 moves from position A to position B when the vehicle 10 is located at the origin 0. The movement path from position A to position B can be identified by a method such as odometry based on movement information including, for example, the wheel speed (movement distance) of the vehicle 10, the steering angle (steering history), the shift position (shift operation history), and the like.

[0037] The coordinate calculation unit 103 of this embodiment converts the coordinates of the representative point P in the captured image 50 estimated by the representative point estimation unit 102 into a three-dimensional relative coordinate system based on the internal parameters of the imaging device 14, etc.<X’,Y’,Z’> Then, the coordinate conversion unit 104 of this embodiment converts the relative coordinates (x', y', z') into a three-dimensional absolute coordinate system based on movement information indicating the movement trajectory of the vehicle 10.<X,Y,Z> Transforms absolute coordinates (x,y,z) in

[0038] 3, the state estimation unit 105 estimates the state around the vehicle 10 by inputting the absolute coordinates (x, y, z) corresponding to the representative point P that can be regarded as the grounded position of the person 54, which have been converted (acquired) by the coordinate conversion unit 104, into the state space model. Note that the absolute coordinates may be directly input to the state space model, or a feature amount generated based on the absolute coordinates may be input.

[0039] The output unit 106 outputs the estimation result by the state estimation unit 105 to a predetermined system (for example, the driving support unit 32 or a display in the vehicle cabin).

[0040] As described above, according to the state estimation unit 105 of the present embodiment, a fixed three-dimensional absolute coordinate system is used regardless of the movement of the vehicle 10 or the person 54 (objects to be tracked).<X,Y,Z> The state of the surroundings of the vehicle 10 is estimated based on the absolute coordinates (x, y, z) at the point A. This makes it possible to estimate the state of the surroundings of the vehicle 10 with high accuracy even when the vehicle 10 or the person 54 is moving at high speed.

[0041] The state space model used for estimating the state includes an observation equation and a state equation, and may be configured using, for example, a Kalman filter. Parameters used in the Kalman filter (for example, noise variance of the observation equation and the state equation) are set in advance, for example, by a maximum likelihood method. When absolute coordinates are input as observation values, the state space model of this embodiment outputs a state (state estimate) indicating absolute coordinates and speed (relative speed between the vehicle 10 and the person 54, and absolute speeds of the vehicle 10 and the person 54) as an estimation result. That is, according to the state space model of this embodiment, the state of the current period is estimated from state estimate values ​​(for example, absolute coordinates and speed) indicating the state of one period before, and the estimated state of the current period is updated (corrected) based on absolute coordinates as the observation values ​​of the current period. At this time, since the observation values ​​used for updating the state are absolute coordinates, it is possible to estimate the state with high accuracy regardless of the moving speed of the vehicle 10 and the person 54.

[0042] The processes performed by the state estimation unit 105 in this embodiment include a tracking target selection process, an object position estimation process, and a delay correction process.

[0043] The tracking target selection process is a process for selecting a representative point P to be tracked from among the representative points P estimated by the representative point estimation unit 102. For example, if a representative point P detected in the captured image 50 continues to remain undetected for a predetermined period, the representative point P may be deleted from the tracking targets. In this case, the representative point P that has been detected for multiple consecutive periods becomes the detection target.

[0044] The object position estimation process is a process for estimating the position of the tracked representative point P. This process includes, for example, a statistical process, a post-estimation process, a setting process, and a pre-estimation process.

[0045] The statistical processing is a process of calculating a statistical value (e.g., the median of multiple absolute coordinates) based on multiple values ​​(e.g., absolute coordinates as observed values) acquired in time series for the representative point P that has newly become the tracking target.

[0046] The posterior estimation process is a process for calculating a posterior estimate and a posterior estimate covariance from a prior estimate, a prior estimate covariance, and an observed value. k , the prior estimated covariance is P' k , the observations are y k、 Posterior estimate x k , the posterior estimated covariance is P k Then, the posterior estimate x k and the posterior estimated covariance P k can be calculated, for example, according to the following formulas (1) to (7).

[0047] TIFF2025051210000002.tif97170

[0048] where c and σ v 2 are constant parameters (units are [-] and [mm 2 ]).

[0049] The setting process is to set the new observation y k If the posterior estimation process can be performed, the posterior estimate x k Let be the absolute coordinate and absolute velocity, and the new observed value yk If the posterior estimation process cannot be performed because the posterior estimation process cannot be performed, the prior estimation value x' k is the absolute coordinate and absolute velocity.

[0050] The prior estimation process is a process of calculating the prior estimate value and the prior estimate covariance of the next period from the posterior estimate value and the posterior estimate covariance. k-1 , the posterior estimated covariance is P k-1 , the prior estimate is x' k , the prior estimated covariance is P' k Then, the prior estimate x' k and the prior estimated covariance P' k can be calculated, for example, using the following formulas (8) and (9).

[0051] TIFF2025051210000003.tif55170

[0052] Here, σ w 2 is a constant parameter [mm 2 s -4 ], dt is the software execution time × (current frame - the frame where the last posterior estimation was performed or the stock of observations reached its upper limit + 1) [s].

[0053] The delay correction process is a process for correcting the estimation result by the state estimation unit 105 so as to reduce the effect of the time (delay) required for processing from acquisition of the captured image 50 to completion of state estimation.

[0054] Furthermore, in the case where the representative point P is not acquired in a certain period, the state estimation unit 105 of this embodiment continues to perform state estimation based on the representative point P acquired prior to that period for a predetermined period following the period in which the representative point P was not acquired.

[0055] FIG. 7 is a diagram showing an example of a situation in which the state estimation unit 105 of the embodiment continues estimating the state during a period in which the representative point P has not been acquired. FIG. 7 shows a case in which the representative point P is acquired in periods t-2, t-1, t+2, and t+3, and the representative point P is not acquired in periods t and t+1. In such a case, the state estimation unit 105 of the embodiment continues estimating the state for a predetermined period Δt after the period t based on the representative point P acquired before the period t in which the representative point P was not acquired (for example, the representative point P in the period t-1). Note that the period Δt illustrated here is a period for two periods, but the period Δt should be appropriately set according to the specifications of the vehicle 10, etc. As a result, even in a situation in which the representative point P cannot be temporarily acquired (for example, when the person 54 is temporarily invisible due to an obstacle, etc.), the person 54 can be tracked.

[0056] Moreover, the state estimation unit 105 of this embodiment executes state estimation using a statistical value calculated based on at least three absolute coordinates as an initial value.

[0057] FIG. 8 is a diagram showing an example of a state estimation method in the state estimation unit 105 of the embodiment. In FIG. 8, the observation value 61 indicates the observation value of the absolute coordinate of the representative point P acquired in each of the cycles t-2 to t+2. The state estimation value 71 indicates the state estimation value as an estimation result in each of the cycles t to t+2. As shown by the dashed arrow in FIG. 8, the state estimation unit 105 of the embodiment estimates the state estimation value 71 in a certain cycle t using as an initial value a statistical value (e.g., a median value, etc.) of the absolute coordinate as three observation values ​​61 acquired before the cycle t. Thereafter, each state estimation value 71 is updated (corrected) by a Kalman filter or the like using the observation values ​​61 in each of the cycles t to t+2. That is, for cycles t+1, t+2, ... after cycle t, the state estimation value 71 is updated (corrected) based on the observation values ​​61 acquired in each of the cycles t+1, t+2, ... in the same manner as a normal Kalman filter. This can reduce the influence of an error in the initial value in a state space model such as a Kalman filter, and can improve the state estimation accuracy.

[0058] 9 is a flowchart showing an example of processing in the object detection device of the embodiment. In step S101, the image acquisition unit 101 acquires a captured image 50 captured by the imaging device 14. In step S102, the representative point estimation unit 102 sets an attention area 52 surrounding a person 54 in the captured image 50. At this time, if the captured image 50 does not include an image of the person 54, that is, if no person is present within the imaging range of the imaging device 14, the attention area 52 is not set.

[0059] In step S103, the representative point estimation unit 102 determines whether or not there is a region of interest 52 in the captured image 50. If it is determined in step S103 that there is no region of interest 52 (S103: No), this routine ends. If it is determined in step S103 that there is a region of interest 52 (S103: Yes), in step S104, the representative point estimation unit 102 inputs the region of interest 52 into the trained model and estimates the representative point P.

[0060] In step S105, the coordinate calculation unit 103 calculates a three-dimensional relative coordinate system based on the estimation result by the representative point estimation unit 102 and the internal parameters of the image capture device 14.<X’,Y’,Z’> In step S106, the coordinate conversion unit 104 converts the relative coordinates (x', y', z') into the three-dimensional absolute coordinate system<X,Y,Z> The coordinates are converted into absolute coordinates (x, y, z) at (see Figure 6).

[0061] In step S107, the state estimation unit 105 inputs the absolute coordinates to a state space model and estimates the state around the vehicle 10. In step S108, the output unit 106 outputs the estimation result by the state estimation unit 105 to a predetermined system such as the driving support unit 32. Note that, when multiple attention areas 52 exist, the above process is repeatedly executed for each attention area 52.

[0062] As described above, according to the object detection device of this embodiment, the state of the surroundings of the vehicle 10 is estimated based on absolute coordinates in a fixed three-dimensional absolute coordinate system regardless of the movement of the vehicle 10 or a specific object (such as the person 54) to be detected (tracked). This makes it possible to estimate the state with high accuracy even when the vehicle 10 or the object is moving at high speed.

[0063] In the above embodiment, the image analysis process for estimating the representative point P includes a process for setting a region of interest 52 surrounding an area in the captured image 50 where a person 54 is deemed to exist, and a process for estimating the representative point P by inputting the region of interest 52 into the trained model. This makes it possible to limit the processing by the trained model to processing on the region of interest 52, thereby reducing the computational load and the computational resources.

[0064] Furthermore, in the above embodiment, when the representative point P is not acquired in a certain period, the state estimation unit 105 continues to estimate the state for a predetermined period after the certain period based on the representative point P acquired before the certain period. This makes it possible to continue tracking the person 54 even in a situation where the representative point P cannot be acquired temporarily (for example, when the person 54 is temporarily invisible due to an obstacle, etc.).

[0065] In the above embodiment, the state estimation unit 105 executes state estimation using a statistical value calculated based on at least three absolute coordinates as an initial value, thereby reducing the influence of an error in the initial value in a state space model such as a Kalman filter, and improving the accuracy of state estimation.

[0066] A program for causing a computer (e.g., CPU 24a, etc.) to realize the functions of the object detection device as described above may be configured to be provided by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

[0067] The program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program may be provided or distributed via a network such as the Internet.

[0068] Although the embodiment of the present invention has been described above, the above-mentioned embodiment and its modifications are merely examples and are not intended to limit the scope of the invention. The above-mentioned novel embodiment and modifications can be implemented in various forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The above-mentioned embodiment and modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0069] 10...vehicle, 14...imaging device, 50...captured image, 52...area of ​​interest, 54...person, 101...image acquisition unit, 102...representative point estimation unit, 103...coordinate calculation unit, 104...coordinate conversion unit, 105...state estimation unit, 106...output unit, P...representative point

Claims

1. an image acquisition unit that acquires an image of the surroundings of the vehicle captured by an imaging device; a representative point estimation unit that estimates a representative point that can be regarded as a ground position of a specific object existing around the vehicle in the captured image by image analysis processing of the captured image; a coordinate calculation unit that calculates relative coordinates, which are coordinates of the representative point in a three-dimensional relative coordinate system corresponding to a space based on a specific position of the vehicle, based on an estimation result by the representative point estimation unit and internal parameters of the imaging device; a coordinate conversion unit that converts the relative coordinates into absolute coordinates, which are coordinates in a three-dimensional absolute coordinate system corresponding to a space based on a specific position on the earth, based on movement information indicating a movement trajectory of the vehicle; a state estimation unit that estimates a state around the vehicle by inputting the absolute coordinates into a state space model; an output unit that outputs an estimation result by the state estimation unit; An object detection device comprising:

2. The image analysis process includes a process of setting a region of interest surrounding an area in the captured image where the object is considered to exist, and a process of estimating the representative point by inputting the region of interest into a trained model. The object detection device according to claim 1 .

3. when the representative point is not acquired in a certain period, the state estimation unit continues to estimate the state for a predetermined period after the certain period based on the representative point acquired before the certain period. The object detection device according to claim 1 or 2.

4. the state estimation unit executes the state estimation using a statistical value calculated based on at least three of the absolute coordinates as an initial value. The object detection device according to claim 1 or 2.

Images