Object detection device

Abstract

To provide an object detection device capable of detecting the position of a person with high accuracy.SOLUTION: An object detection device includes an image acquisition part for acquiring a picked-up image obtained by imaging a situation around a vehicle by an imaging device, a representative point estimation part for estimating a representative point regarded as a ground position of a person existing around the vehicle in the picked-up image by image analysis processing on the picked-up image, a coordinate calculation part for calculating representative point coordinates obtained by converting the coordinates of the representative point in the picked-up image into coordinates in a three-dimensional coordinate system corresponding to a space around the vehicle on the basis of an estimation result obtained by the representative point estimation part and an internal parameter of the imaging device, and an output part for outputting position information showing the position of the person based on the representative point coordinates. The image analysis processing includes first determination processing for determining whether the legs of the person are concealed by an obstacle in the picked-up image, and the output part does not output the position information in the case that, in the first determination processing, it is determined that the legs of the person are concealed by the obstacle.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an object detection device. [Background technology]

[0002] Conventionally, as a technology for detecting objects around a vehicle, an object detection device has been proposed that performs object detection (for example, detection of obstacles such as people and other vehicles) on images captured by an on-board camera. The object detection device can detect the presence or absence of an object, detect the distance to the object, calculate the three-dimensional coordinates of the object, and if the object is moving, obtain the object's moving speed and moving direction, etc., and these detection results can be used for vehicle control. In such object detection devices, there is a method of detecting a position (ground contact position) where a person can be considered to be in contact with the road surface as the position of the object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6891954 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, when detecting the ground contact position as the position of a person, if an image of the vicinity of the feet of the person is not properly acquired, it becomes difficult to accurately detect the position of the person.

[0005] The present invention has been made in view of the above, and provides an object detection device capable of detecting the position of a person with high accuracy. [Means for solving the problem]

[0006] An object detection device as one aspect of the present invention includes an image acquisition unit that acquires an image of the situation around the vehicle captured by an imaging device, a representative point estimation unit that estimates a representative point that can be regarded as the ground contact position of a person present around the vehicle in the captured image by image analysis processing of the captured image, a coordinate calculation unit that calculates representative point coordinates by converting the coordinates of the representative point in the captured image into coordinates in a three-dimensional coordinate system corresponding to the space around the vehicle based on the estimation result by the representative point estimation unit and internal parameters of the imaging device, and an output unit that outputs position information indicating the position of the person based on the representative point coordinates, wherein the image analysis processing includes a first determination processing that determines whether the person's feet are hidden by an obstacle in the captured image, and the output unit does not output the position information if the first determination processing determines that the person's feet are hidden by an obstacle. Effect of the Invention

[0007] According to the present invention, it is possible to provide an object detection device capable of detecting the position of a person with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of a vehicle on which an object detection device according to a first embodiment is mounted. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of a control system including the object detection device mounted on the vehicle of the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional configuration of the object detection device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a captured image and a region of interest according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of representative points estimated from the region of interest in the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the first determination process according to the first embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of processing in the object detection device of the first embodiment. [Figure 8]FIG. 8 is a flowchart showing an example of processing in the object detection device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The configuration of the embodiment described below and the actions and effects brought about by the configuration are merely examples, and the present invention is not limited to the following description.

[0010] (First embodiment) The object detection device of this embodiment generates position information indicating the position of an object on a road surface that is included in an image captured by, for example, a wide-angle camera (for example, a fish-eye camera).

[0011] FIG. 1 is a diagram showing an example of a vehicle 10 equipped with an object detection device of the first embodiment. The vehicle 10 may be, for example, an automobile using an internal combustion engine as a drive source (internal combustion engine automobile), an automobile using an electric motor as a drive source (electric automobile, fuel cell automobile, etc.), or an automobile using both of them as drive sources (hybrid automobile). The vehicle 10 may be equipped with various transmissions, and may be equipped with various devices (systems, parts, etc.) required to drive the internal combustion engine or the electric motor. The type, number, layout, etc. of devices related to the drive of the wheels 12 (front wheels 12F, rear wheels 12R) of the vehicle 10 may be set in various ways.

[0012] As illustrated in FIG. 1, the vehicle 10 is provided with a plurality of imaging devices 14, for example, four imaging devices 14a to 14d. The imaging devices 14 are digital cameras incorporating imaging elements such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor). The imaging devices 14 can output image data (video data) at a predetermined frame rate. Each of the imaging devices 14 has a wide-angle lens (for example, a fisheye lens) and can capture an image in a range of, for example, 140° to 220° in the horizontal direction. In addition, for example, the optical axis of the imaging devices 14 (14a to 14d) arranged on the outer periphery of the vehicle 10 may be set to face diagonally downward. The imaging devices 14 (14a to 14d) sequentially capture images of the situation around the vehicle 10 and output image data of images captured of the road surface and objects on the road surface. The objects on the road surface may include, for example, people (pedestrians), other vehicles, buildings, road structures, road markings, etc.

[0013] The imaging device 14a is provided, for example, on the front side of the vehicle 10, i.e., on the front side in the vehicle longitudinal direction and at an end portion substantially in the center in the vehicle width direction, such as the front bumper 10a or the front grille, and can capture a forward image including the front end portion of the vehicle 10 (for example, the front bumper 10a). The imaging device 14b is provided, for example, on the rear side of the vehicle 10, i.e., on the rear side in the vehicle longitudinal direction and at an end portion substantially in the center in the vehicle width direction, such as an upper position of the rear bumper 10b, and can capture an image of the rear area including the rear end portion of the vehicle 10 (for example, the rear bumper 10b). The imaging device 14c is provided, for example, on the right end portion of the vehicle 10, such as the right door mirror 10c, and can capture a right side image including an area centered on the right side of the vehicle 10 (for example, an area from the right front to the right rear). The imaging device 14d is provided, for example, on the left end of the vehicle 10, such as the left door mirror 10d, and is capable of capturing a left side image including an area centered on the left side of the vehicle 10 (for example, the area from the left front to the left rear).

[0014] For example, by performing arithmetic processing or image processing on each image data obtained by the imaging devices 14a to 14d, it is possible to display images in each direction around the vehicle 10 and perform periphery monitoring. Furthermore, by performing arithmetic processing or image processing based on each image data, it is possible to generate images with a wider viewing angle, generate and display virtual images (bird's-eye view images (planar images), side-view images, front-view images, etc.) of the vehicle 10 viewed from above, in front, to the side, etc., and perform periphery monitoring.

[0015] As described above, image data captured by each imaging device 14 is displayed on a display device in the vehicle cabin to provide a user, such as a driver, with information about the situation around the vehicle 10. The image data is also used to perform various detection operations, such as detecting objects (e.g., obstacles such as people or other vehicles), identifying their positions, and measuring distances, and position information of detected objects can be used to control the vehicle 10.

[0016] FIG. 2 is a diagram showing an example of the configuration of a control system 100 including an object detection device mounted on a vehicle 10 of the first embodiment. A display device 16 and an audio output device 18 are provided in the cabin of the vehicle 10. The display device 16 is, for example, a liquid crystal display (LCD) or an organic electroluminescent display (OLED). The audio output device 18 is, for example, a speaker. The display device 16 is covered with a transparent operation input unit 20 such as a touch panel. A user (for example, a driver) can visually recognize an image displayed on a display screen of the display device 16 through the operation input unit 20. The user can also perform an operation input by touching, pressing, or moving the operation input unit 20 with a finger or the like at a position corresponding to an image displayed on the display screen of the display device 16. The display device 16, the audio output device 18, the operation input unit 20, and the like are provided in a monitor device 22 located, for example, in the center of the dashboard of the vehicle 10 in the vehicle width direction, i.e., the left-right direction. The monitor device 22 may have an appropriate operation input unit such as a switch, a dial, a joystick, a push button, etc. The monitor device 22 may also function as, for example, a navigation system or an audio system.

[0017] As illustrated in FIG. 2, the control system 100 (including the object detection device) includes an ECU 24 (Electronic Control Unit), a wheel speed sensor 26, a steering angle sensor 28, a shift sensor 30, a driving support unit 32, and the like, in addition to the imaging device 14 (14a to 14d) and the monitor device 22. In the control system 100, the ECU 24, the monitor device 22, the wheel speed sensor 26, the steering angle sensor 28, the shift sensor 30, the driving support unit 32, and the like are electrically connected via an in-vehicle network 34 as an electric communication line. The in-vehicle network 34 is configured as, for example, a CAN (Controller Area Network). The ECU 24 can control various systems by sending control signals through the in-vehicle network 34. The ECU 24 can also receive operation signals from the operation input unit 20 and various switches, and detection signals from various sensors such as the wheel speed sensor 26, the steering angle sensor 28, and the shift sensor 30, and the like, via the in-vehicle network 34. It should be noted that various systems (such as a steering system, a braking system, a drive system, etc.) and various sensors for driving the vehicle 10 are connected to the in-vehicle network 34, but in FIG. 2, configurations that are less relevant to the object detection device of this embodiment are not shown in the figure and will not be described.

[0018] The ECU 24 is configured with a computer or the like, and the hardware and software work together to perform at least a part of the control of the vehicle 10. Specifically, the ECU 24 includes a CPU (Central Processing Unit) 24a, a ROM (Read Only Memory) 24b, a RAM (Random Access Memory) 24c, a display control unit 24d, an audio control unit 24e, and an SSD (Solid State Drive) 24f.

[0019] The CPU 24a reads out a program stored (installed) in a non-volatile storage device such as the ROM 24b, and executes arithmetic processing according to the program. The CPU 24a can, for example, perform image analysis processing on the captured image captured by the imaging device 14, execute object position detection to obtain the three-dimensional coordinates of the object, and estimate the location of the object, the distance to the object, and if the object is moving, the moving speed and moving direction, etc. Then, for example, the CPU 24a can provide information required for controlling and operating the steering system, the brake system, the drive system, etc.

[0020] The ROM 24b stores each program and parameters required for executing the program. The RAM 24c is used as a work area when the CPU 24a executes various arithmetic processing, and is also used as a temporary storage area for various data (such as image data captured sequentially (in time series) by the imaging device 14) used in the calculations by the CPU 24a. The display control unit 24d mainly executes, among the arithmetic processing in the ECU 24, processing of image data acquired from the imaging device 14 and output to the CPU 24a, and conversion of image data acquired from the CPU 24a into image data for display on the display device 16. The audio control unit 24e mainly executes, among the arithmetic processing in the ECU 24, processing of audio acquired from the CPU 24a and output to the audio output device 18. The SSD 24f is a rewritable non-volatile storage unit, and continues to store data acquired from the CPU 24a even when the power supply of the ECU 24 is turned off. The CPU 24a, the ROM 24b, the RAM 24c, and the like can be integrated in the same package. The ECU 24 may be configured to use other processors or logic circuits such as a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 24a. A HDD (Hard Disk Drive) may be provided instead of or in addition to the SSD 24f, or the SSD 24f and the HDD may be provided separately from the ECU 24.

[0021] The wheel speed sensor 26 is a sensor that detects the amount of rotation of the wheel 12 and the number of rotations per unit time. The wheel speed sensor 26 is arranged on each wheel 12, and outputs the number of wheel speed pulses indicating the number of rotations detected at each wheel 12 as a sensor value. The wheel speed sensor 26 may be configured using, for example, a Hall element. The CPU 24a calculates the vehicle speed, acceleration, etc. of the vehicle 10 based on the detection value acquired from the wheel speed sensor 26, and executes various controls.

[0022] The steering angle sensor 28 is a sensor that detects the amount of steering of a steering part such as a steering wheel. The steering angle sensor 28 can be configured using, for example, a Hall element, etc. The CPU 24a acquires the amount of steering of the steering part by the driver, the amount of steering of the front wheels 12F during automatic steering when performing parking assistance, etc. from the steering angle sensor 28 and performs various controls.

[0023] The shift sensor 30 is a sensor that detects the position of the movable parts (bar, arm, button, etc.) of the gear shift operating unit, and detects information indicating the operating state of the transmission, the state of the gear stage, and the direction in which the vehicle 10 can travel (D range: forward direction, R range: reverse direction), etc.

[0024] The driving support unit 32 provides control information to the steering system, the brake system, the drive system, etc., in order to realize driving support for moving the vehicle 10 based on the moving route calculated by the control system 100 or the moving route provided from the outside. For example, the driving support unit 32 executes fully automatic control for automatically controlling all of the steering system, the brake system, the drive system, etc., or executes semi-automatic control for automatically controlling part of the steering system, the brake system, the drive system, etc. In addition, the driving support unit 32 may execute manual control for causing the driver to perform driving operations by providing operation guides for the steering system, the brake system, the drive system, etc. to the driver so that the vehicle 10 can move along the moving route. In this case, the driving support unit 32 may provide operation information to the display device 16 and the audio output device 18. In addition, even when executing semi-automatic control, the driving support unit 32 can provide information on the driver to operate, for example, accelerator operation, etc., via the display device 16 and the audio output device 18.

[0025] FIG. 3 is a diagram showing an example of the functional configuration of the object detection device of the first embodiment. Here, a case where the functional configuration of the object detection device is realized by the CPU 24a is illustrated. The CPU 24a executes a program read from the ROM 24b to realize modules such as an image acquisition unit 101, a representative point estimation unit 102, a coordinate calculation unit 103, and an output unit 104, as shown in FIG. 3. Note that the manner for realizing such a functional configuration is not limited to this. For example, at least a part of the functional units 101 to 104 may be configured by dedicated hardware (circuits, etc.), or may be configured using a processor other than the CPU 24a. Furthermore, the CPU 24a may realize various modules necessary for controlling the vehicle 10 other than the functional units 101 to 104.

[0026] The image acquisition unit 101 acquires captured images showing the surroundings of the vehicle 10 including the road surface on which the vehicle 10 is located, captured by the imaging devices 14. The image acquisition unit 101 may sequentially acquire captured images captured by each imaging device 14 (14a to 14d) and provide them to the representative point estimation unit 102. The image acquisition unit 101 may selectively acquire captured images of possible travel directions so that object detection in the possible travel directions can be performed based on information on the possible travel directions (forward direction or backward direction) of the vehicle 10 that can be acquired from the shift sensor 30 and information on the turning direction of the vehicle 10 that can be acquired from the steering angle sensor 28.

[0027] The representative point estimation unit 102 estimates a representative point that can be regarded as a grounding position of a person existing around the vehicle 10 in the captured image by performing image analysis processing on the captured image acquired by the image acquisition unit 101. A specific method for estimating the representative point from the captured image may be appropriately determined according to the specifications of the imaging device 14 and the vehicle 10, and for example, the representative point can be estimated using a trained model generated by machine learning (deep learning) performed in advance. More specifically, for example, the representative point may be estimated by setting a region of interest that surrounds an area in the captured image where a person can be considered to exist, and inputting the region of interest into the trained model.

[0028] Fig. 4 is a diagram showing an example of a captured image 50 and a region of interest 52 in the first embodiment. Fig. 4 illustrates a captured image 50 that is a wide-angle image captured by an imaging device 14d installed on the left side of a vehicle 10. The captured image 50 includes an image of the side of the vehicle 10, an image of a plurality of people 54 present on the road, and a plurality of regions of interest 52 set to surround each of the people 54.

[0029] The attention area 52 exemplified here is a rectangular area (bounding box) set so as to surround a person, and the size (width x height) of each attention area 52 is set according to the size of an object regarded as a person 54 in the captured image 50 by an appropriate learned model or algorithm. The attention area 52 may be defined, for example, by the center coordinates (x, y), width (w), height (h), etc. of the attention area 52 in the captured image 50. Note that, although the attention area 52 set so as to surround a person is mentioned here, the attention area may be set so as to surround a predetermined object other than a person (for example, another vehicle, a road structure, etc.).

[0030] FIG. 5 is a diagram showing an example of a representative point 71 estimated from the attention area 52 in the first embodiment. FIG. 5 shows that an estimation result (representative point estimation result) of the representative point 71 that can be regarded as the grounding position of the person 54 is obtained by inputting the attention area 52 to the trained model. Note that "inputting the attention area 52 to the trained model" can mean, for example, directly inputting image data corresponding to the attention area 52 extracted from the image data of the captured image 50 to the trained model, or inputting the feature amount (for example, values ​​indicating brightness, color, edges, etc.) of the attention area 52 to the trained model. In addition, when the imaging device 14 is a wide-angle camera or the like, the image of the person 54 in the attention area 52 is distorted due to the influence of a wide-angle lens, etc., but here, for convenience of explanation, an image in a state where the distortion is removed is illustrated.

[0031] The representative point 71 illustrated here is a midpoint between two points 61A, 61B corresponding to each foot of person 54. By determining representative point 71 in this manner, the position on the road surface corresponding to the approximate center (center of gravity) of the body of person 54 can be treated as the position of person 54, and information that appropriately indicates the position of person 54 can be obtained.

[0032] In the present embodiment, as shown in FIG. 5, an enlarged attention area 56 (image data, feature values, etc. corresponding to the enlarged attention area 56) wider than the attention area 52 set to surround the person 54 is input to the trained model to estimate the representative point 71. The enlarged attention area 56 is set to widen the attention area 52, that is, to form a predetermined margin 57 between the attention area 52 and the enlarged attention area 56. The attention area 52 may be set based on, for example, an inference result using a trained model. In such a case, the attention area 52 may vary, and the attention area 52 that does not include the feet of the person 54 may be set. Therefore, by using the enlarged attention area 56 obtained by widening the attention area 52 as an input to the trained model for estimating the representative point 71 as described above, it is possible to prevent the representative point 71 from being estimated based on data that does not include the feet of the person 54, and improve the estimation accuracy of the representative point 71.

[0033] Note that the above-mentioned method of estimating the representative point 71 is merely an example, and is not limited to the above-mentioned method of estimating the representative point 71. For example, the representative point 71 may be estimated using a trained model capable of directly estimating the representative point 71 from the captured image 50, without setting the attention area 52 or the enlarged attention area 56 as described above.

[0034] Moreover, the image analysis process executed in the representative point estimation unit 102 of this embodiment includes a first determination process for determining whether or not the feet of the person 54 in the captured image 50 are hidden by an obstacle (e.g., a wall, a hedge, another vehicle, etc.).

[0035] FIG. 6 is a diagram showing an example of the first determination process of the first embodiment. In FIG. 6, the attention area 52 in a state in which the feet of the person 54 are hidden by an obstacle 75 is illustrated. In this embodiment, such an attention area 52 or an enlarged attention area 56 obtained by enlarging the attention area 52 is input to the trained model to obtain an obstacle presence / absence determination result indicating whether the feet of the person 54 in the attention area 52 are hidden by the obstacle 75. In the example shown in FIG. 6, an obstacle presence / absence determination result indicating the presence of an obstacle 75 hiding the feet of the person 54 is output from the trained model. Note that the trained model that outputs the obstacle presence / absence determination result and the trained model that outputs the representative point estimation result described above may be the same, may be provided separately, or may be linked to each other. In addition, the obstacle presence / absence determination result may be obtained by using an appropriate algorithm other than the trained model.

[0036] Returning to Fig. 3, the coordinate calculation unit 103 calculates representative point coordinates by converting the coordinates of the representative point 71 in the captured image 50 into coordinates in a three-dimensional coordinate system corresponding to the space around the vehicle 10, based on the estimation result by the representative point estimation unit 102 (representative point estimation result) and the internal parameters of the imaging device 14. The representative point estimation result is information indicating the position of the representative point 71 in the captured image 50, and may be, for example, two-dimensional coordinate information indicating the position of a pixel corresponding to the representative point 71 among a plurality of pixels constituting the captured image 50. The internal parameters may include, for example, a focal length, a lens distortion coefficient, a pixel size, a camera position, a camera attitude, etc.

[0037] The output unit 104 outputs position information indicating the position of the person 54 based on the representative point coordinates calculated by the coordinate calculation unit 103. The position information may include, for example, the distance from the vehicle 10 to the person 54, the positional relationship between the vehicle 10 and the person 54, and the relative moving speed and moving direction of the person 54 with respect to the vehicle 10. The position information is used, for example, for various driving support controls by the driving support unit 32.

[0038] Furthermore, the output unit 104 of this embodiment does not output position information when it is determined by the first determination process included in the image analysis process performed by the representative point estimation unit 102 that the feet of the person 54 in the captured image 50 are hidden by the obstacle 75. This makes it possible to suppress the output of low-precision position information.

[0039] 7 is a flowchart showing an example of processing in the object detection device of the first embodiment. In step S101, the image acquisition unit 101 acquires a captured image 50 captured by the imaging device 14. In step S102, the representative point estimation unit 102 sets an attention area 52 surrounding a person 54 in the captured image 50. At this time, if the captured image 50 does not include an image of the person 54, that is, if no person is present within the imaging range of the imaging device 14, the attention area 52 is not set.

[0040] In step S103, the representative point estimation unit 102 determines whether or not there is an attention area 52 in the captured image 50. If it is determined in step S103 that there is no attention area 52 (S103: No), this routine ends without the coordinate calculation unit 103 calculating the representative point coordinates. Note that, as described above, the representative point coordinates are obtained by converting the coordinates (e.g., two-dimensional coordinates) of the representative point 71 in the captured image 50 into coordinates (e.g., three-dimensional coordinates) in a three-dimensional coordinate system corresponding to the space around the vehicle 10.

[0041] In step S103, if it is determined that the attention region 52 exists (S103: Yes), in step S104, the representative point estimation unit 102 inputs an enlarged attention region 56 obtained by enlarging the attention region 52 to the trained model. At this time, the attention region 52 may be input to the trained model.

[0042] In step S105, the representative point estimation unit 102 acquires a representative point estimation result and an obstacle presence / absence determination result as a result of inputting the enlarged attention area 56 into the trained model. As described above, the representative point estimation result is information (e.g., two-dimensional coordinates, etc.) that specifies the position of the representative point 71 in the captured image 50. As described above, the obstacle presence / absence determination result is information that indicates the presence or absence of an obstacle 75 that hides the feet of the person 54 in the attention area 52. At this time, the trained model that outputs the representative point setting result and the trained model that outputs the obstacle presence / absence determination result may be the same or different.

[0043] In step S106, the coordinate calculation unit 103 determines whether or not there is an obstacle 75 hiding the feet of the person 54 based on the obstacle presence / absence determination result. If it is determined in step S106 that there is no obstacle 75 (S106: No), in step S107, the coordinate calculation unit 103 calculates representative point coordinates based on the representative point estimation result and internal parameters of the imaging device 14. Note that information other than the representative point estimation result and the internal parameters may be used to calculate the representative point coordinates.

[0044] Thereafter, in step S108, the coordinate calculation unit 103 determines whether the process for calculating the representative point coordinates for all of the attention areas 52 (enlarged attention areas 56) has been completed. In step S108, if the process has not been completed for all of the attention areas 52 (S108: No), the process from step S104 onwards is executed again for the enlarged attention areas 56 corresponding to the unprocessed attention areas 52. In step S108, if the process has been completed for all of the attention areas 52 (S108: Yes), in step S110, the output unit 104 outputs position information of the person 54 based on the calculated representative point coordinates to a predetermined mechanism such as the driving support unit 32.

[0045] If it is determined in step S106 that an obstacle 75 hiding the feet of person 54 is present (S106: Yes), in step S109, coordinate calculation unit 103 stops calculating the representative point coordinates for the attention area 52 currently being processed. Then, in step S108, it is determined whether or not the processing has been completed for all attention areas 52. That is, if it is determined by the first determination process that the feet of person 54 in attention area 52 are hidden by obstacle 75, output unit 104 does not output position information indicating the position of person 54 whose feet are hidden.

[0046] As described above, according to the object detection device of the present embodiment, the representative point 71 that can be regarded as the ground position of the person 54 in the captured image 50 is estimated by the image analysis process on the captured image 50, the representative point coordinates are calculated by converting the representative point 71 into three-dimensional coordinates, and the position information based on the representative point coordinates is output. At this time, if the feet of the person 54 are not clearly shown in the captured image 50, the representative point 71 as the ground position of the person 54 cannot be estimated with high accuracy, and the accuracy of the position information may decrease. Therefore, in the present embodiment, the image analysis process includes a first determination process for determining whether the feet of the person 54 are hidden by an obstacle 75 in the captured image 50, and if the first determination process determines that the feet of the person 54 are hidden by the obstacle 75, the output of the position information is prohibited. As a result, the output of low-accuracy position information is prohibited, and the position of the person can be detected with high accuracy.

[0047] In the above embodiment, the image analysis process includes a process of setting a region of interest 52 surrounding an area in the captured image 50 where a person 54 is deemed to exist, and a process of estimating a representative point 71 by inputting the region of interest 52 (enlarged region of interest 56) to the trained model. This makes it possible to limit the processing by the trained model to processing on the region of interest 52 (enlarged region of interest 56), thereby reducing the computational load and the computational resources.

[0048] In the above embodiment, the representative point 71 is the midpoint between the two points 61A, 61B corresponding to the feet of the person 54. This makes it possible to treat the position of the road surface corresponding to approximately the center of the body of the person 54 as the position of the person, and to obtain information that appropriately indicates the position of the person 54.

[0049] In the above embodiment, the configuration is exemplified in which the representative point 71 is estimated by setting the attention area 52 in the captured image 50 and inputting the attention area 52 or the enlarged attention area 56 obtained by enlarging the attention area 52 to the trained model, but the method of estimating the representative point 71 is not limited to this. For example, the representative point 71 may be estimated using a trained model capable of estimating the representative point 71 directly from the captured image 50. In this case, the process of setting the attention area 52 in the captured image 50 can be omitted.

[0050] Other embodiments will be described below, but descriptions of parts that are the same as or similar to the first embodiment will be omitted as appropriate.

[0051] Second embodiment The object detection device of the second embodiment includes, in its image analysis processing, a second judgment processing for judging whether or not a person 54 is present within a region of interest 52 set within a captured image 50, and has a function of not outputting location information if the second judgment processing determines that no person is present within the region of interest 52.

[0052] 8 is a flowchart showing an example of processing in the object detection device of the second embodiment. In step S201, the image acquisition unit 101 acquires a captured image 50 captured by the imaging device 14. In step S202, the representative point estimation unit 102 sets an attention area 52 surrounding a person 54 in the captured image 50. At this time, if the captured image 50 does not include an image of the person 54, that is, if no person is present within the imaging range of the imaging device 14, the attention area 52 is not set.

[0053] In step S203, the representative point estimation unit 102 determines whether or not there is a region of interest 52 in the captured image 50. If it is determined in step S203 that there is no region of interest 52 (S203: No), the coordinate calculation unit 103 does not calculate the representative point coordinates, and this routine ends.

[0054] In step S203, if it is determined that the attention region 52 exists (S203: Yes), in step S204, the representative point estimation unit 102 inputs an enlarged attention region 56 obtained by enlarging the attention region 52 to the trained model. At this time, the attention region 52 may be input to the trained model.

[0055] In step S205, the representative point estimation unit 102 acquires a representative point estimation result, an obstacle presence / absence determination result, and a person presence / absence determination result as a result of inputting the enlarged attention area 56 to the trained model. The person presence / absence determination result is information indicating whether or not a person 54 exists in the attention area 52 set in the captured image 51 in step S202. At this time, the trained model that outputs the representative point setting result, the trained model that outputs the obstacle presence / absence determination result, and the trained model that outputs the person presence / absence determination result may be the same or different. Furthermore, the person presence / absence determination result may be acquired using the trained model as described above, or may be acquired using an appropriate algorithm other than the trained model.

[0056] In step S206, the coordinate calculation unit 103 determines whether or not the person 54 exists within the attention area 52 based on the person presence / absence determination result. If it is determined in step S206 that the person 54 exists within the attention area 52 (S206: Yes), in step S207, the coordinate calculation unit 103 determines whether or not an obstacle 75 hiding the feet of the person 54 exists based on the obstacle presence / absence determination result. If it is determined in step S207 that an obstacle 75 does not exist (S207: No), in step S208, the coordinate calculation unit 103 calculates representative point coordinates based on the representative point estimation result and internal parameters of the imaging device 14. Note that information other than the representative point estimation result and the internal parameters may be used to calculate the representative point coordinates.

[0057] Thereafter, in step S209, the coordinate calculation unit 103 determines whether the process for calculating the representative point coordinates for all attention areas 52 (enlarged attention areas 56) has been completed. In step S209, if the process has not been completed for all attention areas 52 (S209: No), the process from step S204 onwards is executed again for the enlarged attention areas 56 corresponding to the unprocessed attention areas 52. In step S209, if the process has been completed for all attention areas 52 (S209: Yes), in step S211, the output unit 104 outputs position information of the person 54 based on the calculated representative point coordinates to a predetermined mechanism such as the driving support unit 32.

[0058] If it is determined in step S206 that the person 54 does not exist (S206: No), or if it is determined in step S207 that an obstacle 75 hiding the feet of the person 54 exists (S207: Yes), in step S210, the coordinate calculation unit 103 stops calculating the representative point coordinates for the attention area 52 currently being processed. Thereafter, in step S209, it is determined whether or not the processing has been completed for all attention areas 52. That is, if it is determined by the second determination process that the person 54 does not exist in the attention area 52, or if it is determined by the first determination process that the feet of the person 54 in the attention area 52 are hidden by the obstacle 75, the output unit 104 does not output the position information. Thereby, for example, after the attention area 52 is set in the captured image 50, when the person 54 who existed at the time of setting moves outside the imaging range of the imaging device 14, or when the person 54 is hidden by an obstacle such as another vehicle, it is possible to prevent the position information from being output.

[0059] As described above, in the object detection device of this embodiment, in addition to the first determination process, the image analysis process includes a second determination process for determining whether or not a person 54 exists within the attention area 52 set in the captured image 50. Then, when it is determined by the second determination process that a person 54 does not exist within the attention area 52, output of the position information is also prohibited. This makes it possible to further improve the accuracy of the position information.

[0060] A program for causing a computer (e.g., CPU 24a, etc.) to realize the functions of the object detection device as described above may be configured to be provided by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

[0061] The program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program may be provided or distributed via a network such as the Internet.

[0062] Although the embodiment of the present invention has been described above, the above-mentioned embodiment and its modifications are merely examples and are not intended to limit the scope of the invention. The above-mentioned novel embodiment and modifications can be implemented in various forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The above-mentioned embodiment and modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0063] 10 vehicle, 14 imaging device, 50 captured image, 52 area of ​​interest, 54 person, 56 enlarged area of ​​interest, 71 representative point, 75 obstacle, 101 image acquisition unit, 102 representative point estimation unit, 103 coordinate calculation unit, 104 output unit

Claims

1. an image acquisition unit that acquires an image of the surroundings of the vehicle captured by an imaging device; a representative point estimation unit that estimates a representative point that can be regarded as a grounding position of a person existing around the vehicle in the captured image by image analysis processing of the captured image; a coordinate calculation unit that calculates representative point coordinates by converting coordinates of the representative point in the captured image into coordinates in a three-dimensional coordinate system corresponding to a space around the vehicle, based on an estimation result by the representative point estimation unit and internal parameters of the imaging device; an output unit that outputs position information indicating a position of the person based on the representative point coordinates; Equipped with the image analysis process includes a first determination process of determining whether or not a foot of the person is hidden by an obstacle in the captured image, the output unit does not output the position information when it is determined that the person's feet are hidden by an obstacle in the first determination process. Object detection device.

2. The image analysis process includes a process of setting a region of interest surrounding an area in the captured image where a person is considered to exist, and a process of estimating the representative point by inputting the region of interest into a trained model. The object detection device according to claim 1 .

3. the image analysis process includes a second determination process of determining whether or not a person is present within the attention area set in the captured image, When it is determined that no person is present in the attention area by the second determination process, the output unit does not output the position information. The object detection device according to claim 2 .

4. The representative point is a midpoint between two points corresponding to each of the person's feet. The object detection device according to any one of claims 1 to 3.

Images