Object position detection device

Abstract

To provide an object position detection device that can reduce a processing load and computational resource while utilizing images captured by a wide-angle camera.SOLUTION: An object position detection device includes: an image acquisition unit which acquires imaging data of wide-angle images obtained by imaging surroundings of a vehicle cabin by a wide-angle camera; a region setting unit which sets a region of interest that encloses a region in the wide-angle image where it is assumed that an object exists; a candidate point setting unit which sets multiple candidate points that could serve as candidates of the existence position of the object on or near a boundary line that defines the region of interest; a representative point selection unit which defines a reference point at a prescribed position within the region of interest, applies distortion correction to the reference point and the candidate points, and selects a representative point that is assumed as a ground contact position of the object from the candidate points after distortion correction; a coordinate acquisition unit which acquires the three-dimensional coordinates of the representative point; and an output unit which outputs position information of the ground contact position of the object on the basis of the three-dimensional coordinates.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an object position detection device. [Background technology]

[0002] Conventionally, various object position detection devices have been proposed as a technology for detecting objects around a vehicle, which detect objects (for example, detect obstacles such as people or other vehicles) from images captured by an on-board camera. The object position detection device can detect the presence or absence of an object, detect the distance to the object, calculate the three-dimensional coordinates of the object, and, if the object is moving, obtain the object's moving speed and moving direction, etc., and these detection results can be used for vehicle control. It is desirable for the on-board cameras used in such on-board object position detection devices to obtain as wide a range of information (images) as possible with a small number of cameras, and therefore wide-angle cameras (fisheye cameras) equipped with wide-angle lenses (for example, fisheye lenses) are often used. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-155102 [Patent Document 2] Patent No. 6891954 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional technology, images captured by wide-angle cameras (fisheye cameras) used in object position detection devices tend to become more distorted as they approach the periphery. Therefore, in order to accurately detect the position of an object, it was necessary to perform object detection processing after correcting the distortion of the captured image. Therefore, the processing load of the object position detection device is large, and there is room for improvement in in-vehicle devices with limited computing resources.

[0005] Therefore, one of the objectives of the present disclosure is to provide an object position detection device that is capable of reducing the processing load and calculation resources while using an image captured by a wide-angle camera (fisheye camera). [Means for solving the problem]

[0006] An object position detection as an example of the present disclosure includes an image acquisition unit that acquires imaging data of a wide-angle image captured by a wide-angle camera of the surroundings of a vehicle cabin, an area setting unit that sets an attention area surrounding an area in which an object is considered to exist in the wide-angle image, a candidate point setting unit that sets a plurality of candidate points that can be candidates for the presence position of the object on a boundary line that defines the attention area or in a position near the boundary line, a representative point selection unit that sets a reference point at a predetermined position within the attention area, performs distortion correction on the reference point and the candidate point, and selects a representative point that can be considered as the ground position of the object from the candidate points after distortion correction, a coordinate acquisition unit that acquires the three-dimensional coordinates of the representative point, and an output unit that outputs position information of the ground position of the object based on the three-dimensional coordinates. According to this configuration, for example, processing such as distortion correction is performed only on the reference point and the correction point (including the representative point) rather than on the entire wide-angle image, so that the processing load and calculation resources can be reduced.

[0007] In addition, the representative point selection unit of the object position detection device may select the candidate point that is closest to the reference point in a direction perpendicular to a normal line that extends vertically downward from the reference point as the representative point. With this configuration, for example, the selection of the representative point can be performed more accurately and easily.

[0008] In addition, the region setting unit of the object position detection device may set the region of interest as a rectangle surrounding the object, and the candidate point setting unit may set a plurality of candidate points at approximately equal intervals on a boundary line of the region of interest. With this configuration, for example, candidate points that can be representative points can be efficiently set.

[0009] Furthermore, the representative point selection unit of the object position detection device may, for example, assume that the center position of the object is located at the center of the attention area, and set the reference point at the center of the attention area. With this configuration, for example, it is possible to easily and appropriately set the reference point used to select the representative point. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an exemplary schematic plan view showing a vehicle on which an object position detection device according to an embodiment can be mounted. [Diagram 2] FIG. 2 is an exemplary schematic block diagram showing a configuration of a control system including an object position detection device according to an embodiment. [Diagram 3] FIG. 3 is an exemplary schematic block diagram showing a configuration in which the object position detection device (object position detection unit) according to the embodiment is realized by a CPU. [Figure 4] FIG. 4 is an exemplary schematic diagram showing a setting state of a region of interest in a wide-angle image used by the object position detection device according to the embodiment. [Diagram 5] FIG. 5 is an exemplary schematic diagram showing a state in which reference points and candidate points are set in a region of interest used when detecting an object position in an object position detection device according to an embodiment. [Figure 6] FIG. 6 is an exemplary schematic diagram showing in detail how candidate points are set in the object position detection device according to the embodiment. [Figure 7] FIG. 7 is an exemplary schematic diagram showing an image of the panoramic transformation process used when selecting a representative point in the object position detection device according to the embodiment. [Figure 8] FIG. 8 is an exemplary schematic image diagram showing that a wide-angle image used by an object position detection device according to an embodiment includes distortion in the vertical direction. [Figure 9] FIG. 9 is an exemplary schematic image diagram showing a state in which vertical distortion of a wide-angle image used by an object position detection device according to an embodiment has been removed by panoramic transformation using equirectangular projection. [Figure 10] FIG. 10 is an exemplary flowchart showing the flow of an object position detection process in the object position detection device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments and modifications of the present disclosure will be described with reference to the drawings. The configurations of the embodiments and modifications described below, as well as the actions and effects brought about by the configurations, are merely examples and are not limited to the contents described below.

[0012] The object position detection device of this embodiment acquires specific pixels to be used to identify, for example, the foot position of an object in a region of interest (for example, a bounding box) that is recognized as a region where an object exists on a road surface (ground) included in an image captured by a wide-angle camera (for example, a fisheye camera).Then, by performing processing such as distortion correction on the identified pixels, the object's foot position is detected with low processing load and low resource processing, and is output as object position information.

[0013] 1 is an exemplary schematic plan view of a vehicle 10 equipped with an object position detection device according to the present embodiment. The vehicle 10 may be, for example, an automobile (internal combustion engine automobile) using an internal combustion engine (engine, not shown) as a drive source, an automobile (electric automobile, fuel cell automobile, etc.) using an electric motor (motor, not shown) as a drive source, or an automobile (hybrid automobile) using both of them as drive sources. The vehicle 10 may be equipped with various transmissions and various devices (systems, parts, etc.) required to drive the internal combustion engine and the electric motor. The type, number, layout, etc. of devices related to the drive of the wheels 12 (front wheels 12F, rear wheels 12R) in the vehicle 10 may be set in various ways.

[0014] As illustrated in FIG. 1, the vehicle 10 is provided with a plurality of imaging units 14, for example, four imaging units 14a to 14d. The imaging units 14 are digital cameras incorporating imaging elements such as a CCD (Charge Coupled Device) or a CIS (CMOS image sensor). The imaging units 14 can output video data (captured image data) at a predetermined frame rate. Each of the imaging units 14 has a wide-angle lens (for example, a fisheye lens) and can capture images in a range of, for example, 140° to 220° in the horizontal direction. In addition, for example, the optical axes of the imaging units 14 (14a to 14d) arranged on the outer periphery of the vehicle 10 may be set to face obliquely downward. Therefore, the imaging unit 14 (14a to 14d) sequentially captures images of the external surroundings of the vehicle 10, including the road surface (ground) on which the vehicle 10 can move, marks on the road surface (arrows, dividing lines, parking frames indicating parking spaces, lane dividers, etc.), and objects present on the road surface (obstacles such as pedestrians, other vehicles, fixed objects, etc.), and outputs the captured image data.

[0015] The imaging unit 14a is provided, for example, on the front side of the vehicle 10, i.e., on the front side in the vehicle longitudinal direction and at an end portion substantially in the center in the vehicle width direction, such as the front bumper 10a or the front grille, and can capture a forward image including the front end portion of the vehicle 10 (for example, the front bumper 10a). The imaging unit 14b is provided, for example, on the rear side of the vehicle 10, i.e., on the rear side in the vehicle longitudinal direction and at an end portion substantially in the center in the vehicle width direction, such as an upper position of the rear bumper 10b, and can capture an image of the rear region including the rear end portion of the vehicle 10 (for example, the rear bumper 10b). The imaging unit 14c is provided, for example, on the right end portion of the vehicle 10, such as the right door mirror 10c, and can capture a right side image including an area centered on the right side of the vehicle 10 (for example, an area from the right front to the right rear). The imaging unit 14d is provided, for example, on the left end of the vehicle 10, such as the left door mirror 10d, and is capable of capturing a left side image including an area centered on the left side of the vehicle 10 (for example, the area from the left front to the left rear).

[0016] For example, by performing arithmetic processing or image processing on each piece of captured image data obtained by the imaging units 14a to 14d, it is possible to display images in each direction around the vehicle 10 and perform periphery monitoring. Furthermore, by performing arithmetic processing or image processing based on each piece of captured image data, it is possible to generate images with a wider viewing angle, generate and display virtual images (bird's-eye view images (planar images), side-view images, front-view images, etc.) of the vehicle 10 viewed from above, in front, to the side, etc., and perform periphery monitoring.

[0017] As described above, the captured image data captured by each imaging unit 14 is displayed on a display device in the vehicle cabin to provide a user, such as a driver, with information about the situation around the vehicle 10. The captured image data is also used to perform various detection operations, such as detecting objects (e.g., obstacles such as other vehicles and pedestrians), identifying their positions, and measuring distances, and the position information of the detected objects can be used to control the vehicle 10.

[0018] FIG. 2 is an exemplary schematic block diagram of a configuration of a control system 100 including an object detection device mounted on a vehicle 10. A display device 16 and an audio output device 18 are provided in the cabin of the vehicle 10. The display device 16 is, for example, a liquid crystal display (LCD) or an organic electroluminescent display (OLED). The audio output device 18 is, for example, a speaker. The display device 16 is covered with a transparent operation input unit 20 such as a touch panel. A user (for example, a driver) can visually recognize an image displayed on a display screen of the display device 16 through the operation input unit 20. The user can also perform an operation input by touching, pressing, or moving the operation input unit 20 with a finger or the like at a position corresponding to an image displayed on the display screen of the display device 16. The display device 16, the audio output device 18, the operation input unit 20, and the like are provided in a monitor device 22 located, for example, in the center of the dashboard of the vehicle 10 in the vehicle width direction, i.e., the left-right direction. The monitor device 22 may have an operation input unit (not shown) such as a switch, a dial, a joystick, a push button, etc. The monitor device 22 may also be used as, for example, a navigation system or an audio system.

[0019] As illustrated in FIG. 2, the control system 100 (including the object detection device) includes an ECU (electronic control unit) 24, a wheel speed sensor 26, a steering angle sensor 28, a shift sensor 30, a driving support unit 32, and the like, in addition to the imaging unit 14 (14a to 14d) and the monitor device 22. In the control system 100, the ECU 24, the monitor device 22, the wheel speed sensor 26, the steering angle sensor 28, the shift sensor 30, the driving support unit 32, and the like are electrically connected via an in-vehicle network 34 as an electric communication line. The in-vehicle network 34 is configured as, for example, a CAN (controller area network). The ECU 24 can control various systems by sending control signals through the in-vehicle network 34. The ECU 24 can also receive operation signals from the operation input unit 20 and various switches, and detection signals from various sensors such as the wheel speed sensor 26, the steering angle sensor 28, and the shift sensor 30, and the like, via the in-vehicle network 34. It should be noted that various systems (such as a steering system, a braking system, a drive system, etc.) for driving the vehicle 10 and various sensors are connected to the in-vehicle network 34. However, in FIG. 2, configurations that are less relevant to the object position detection device of this embodiment are not shown in the figure, and their description will be omitted.

[0020] The ECU 24 is configured with a computer or the like, and the hardware and software work together to perform overall control of the vehicle 10. Specifically, the ECU 24 includes a CPU (Central Processing Unit) 24a, a ROM (Read Only Memory) 24b, a RAM (Random Access Memory) 24c, a display control unit 24d, an audio control unit 24e, and an SSD (Solid State Drive) 24f.

[0021] The CPU 24a reads out a program stored (installed) in a non-volatile storage device such as the ROM 24b, and executes arithmetic processing according to the program. The CPU 24a can, for example, perform image processing on the captured image captured by the imaging unit 14, execute object position detection (recognition) to obtain the three-dimensional coordinates of the object, and estimate the location of the object, the distance to the object, and if the object is moving, the moving speed and moving direction, etc. Then, for example, the CPU 24a can provide information required for controlling and operating the steering system, the brake system, the drive system, etc.

[0022] The ROM 24b stores each program and parameters required for executing the program. The RAM 24c is used as a work area when the CPU 24a executes the object position detection process, and is also used as a temporary storage area for various data (image data captured sequentially (in time series) by the imaging unit 14, etc.) used in the calculations of the CPU 24a. The display control unit 24d mainly executes image processing of image data acquired from the imaging unit 14 and output to the CPU 24a, and conversion of image data acquired from the CPU 24a into image data for display on the display device 16, among the calculations of the ECU 24. The audio control unit 24e mainly executes audio processing of the calculations of the ECU 24, which is acquired from the CPU 24a and output to the audio output device 18. The SSD 24f is a rewritable non-volatile storage unit, and continues to store data acquired from the CPU 24a even when the power of the ECU 24 is turned off. The CPU 24a, the ROM 24b, the RAM 24c, etc. can be integrated in the same package. The ECU 24 may be configured to use other logic operation processors, such as a digital signal processor (DSP), logic circuits, etc., instead of the CPU 24a. A hard disk drive (HDD) may be provided instead of the SSD 24f, or the SSD 24f and the HDD may be provided separately from the ECU 24.

[0023] The wheel speed sensor 26 is a sensor that detects the amount of rotation of the wheel 12 and the number of rotations per unit time. The wheel speed sensor 26 is arranged on each wheel 12, and outputs the number of wheel speed pulses indicating the number of rotations detected at each wheel 12 as a sensor value. The wheel speed sensor 26 can be configured using, for example, a Hall element. The CPU 24a calculates the vehicle speed, acceleration, etc. of the vehicle 10 based on the detection value acquired from the wheel speed sensor 26, and executes various controls.

[0024] The steering angle sensor 28 is a sensor that detects the amount of steering of a steering part such as a steering wheel. The steering angle sensor 28 is configured using, for example, a Hall element. The CPU 24a acquires the amount of steering of the steering part by the driver, the amount of steering of the front wheels 12F during automatic steering when performing parking assistance, and the like from the steering angle sensor 28 to perform various controls.

[0025] The shift sensor 30 is a sensor that detects the position of the movable parts (bar, arm, button, etc.) of the gear shift operating unit, and detects information indicating the operating state of the transmission, the state of the gear stage, the direction in which the vehicle 10 can travel (D range: forward direction, R range: reverse direction), etc.

[0026] The driving support unit 32 provides control information to the steering system, the brake system, the drive system, etc., in order to realize driving support for moving the vehicle 10 based on the moving route calculated by the control system 100 or the moving route provided from the outside. For example, the driving support unit 32 executes fully automatic control for automatically controlling all of the steering system, the brake system, the drive system, etc., or executes semi-automatic control for automatically controlling part of the steering system, the brake system, the drive system, etc. In addition, the driving support unit 32 may execute manual control for causing the driver to perform driving operations by providing the driver with operation guides for the steering system, the brake system, the drive system, etc., so that the vehicle 10 can move along the moving route. In this case, the driving support unit 32 may provide operation information to the display device 16 and the audio output device 18. In addition, even when executing semi-automatic control, the driving support unit 32 can provide information on the driver to operate, for example, accelerator operation, etc., via the display device 16 and the audio output device 18.

[0027] FIG. 3 is a block diagram showing an example and schematic configuration of an object position detection device (object position detection unit 36) according to the embodiment realized by a CPU 24a. The CPU 24a executes an object position detection program read from the ROM 24b to realize modules such as an image acquisition unit 38, an area setting unit 40, a candidate point setting unit 42, a representative point selection unit 44, a coordinate acquisition unit 46, and an output unit 48, as shown in FIG. 3. Note that the image acquisition unit 38, the area setting unit 40, the candidate point setting unit 42, the representative point selection unit 44, the coordinate acquisition unit 46, and the output unit 48 may be partially or entirely configured by hardware such as a circuit. Although not shown in FIG. 3, the CPU 24a may also realize various modules necessary for the running of the vehicle 10. Although FIG. 2 shows the CPU 24a that mainly executes the object position detection process, the CPU 24a may be provided with a CPU for realizing various modules necessary for the running of the vehicle 10, or may be provided with an ECU separate from the ECU 24.

[0028] The image acquisition unit 38 acquires a captured image (wide-angle (fisheye) image) showing the surroundings of the vehicle 10 including the road surface (ground) on which the vehicle 10 is located, captured by the imaging unit 14 (wide-angle camera, fisheye camera, etc.), and provides the image to the region setting unit 40. The image acquisition unit 38 may sequentially acquire captured images captured by each imaging unit 14 (14a to 14d) and provide the images to the region setting unit 40. In another example, the image acquisition unit 38 may selectively acquire captured images in a possible travel direction and provide the images to the region setting unit 40 so that object detection in the possible travel direction can be performed based on information on the possible travel direction (forward direction or backward direction) of the vehicle 10 that can be acquired from the shift sensor 30 and information on the turning direction of the vehicle 10 that can be acquired from the steering angle sensor 28.

[0029] The region setting unit 40 sets a rectangular region of interest (e.g., a rectangular bounding box) for selecting a predetermined target object (e.g., an object corresponding to a person or another vehicle) from among the objects included in the wide-angle image acquired by the image acquisition unit 38. That is, a region that surrounds an area in which an object to be processed in the wide-angle image is considered to exist is set. The object surrounded by the region of interest is an object that may exist on the road surface (ground) around the vehicle 10, and may include, for example, a movable object corresponding to a person or another vehicle (including a bicycle, etc.), as well as a stationary object corresponding to a fence, a utility pole, a roadside tree, a flower bed, etc. The object surrounded by the region of interest is an object that can be extracted by referring to a model that has been trained in advance, for example, by machine learning, etc. The region setting unit 40 applies a model trained according to a known technique to the wide-angle image, and detects where a known object exists on the wide-angle image by, for example, determining the degree of agreement between the feature points of the object as the model and the feature points of the image included in the wide-angle image. Then, a region of interest (bounding box) of, for example, a substantially rectangular shape is set so as to surround the detected object.

[0030] FIG. 4 is an exemplary schematic diagram showing a setting state of attention areas 52 (52a to 54d) in a wide-angle image 50 used by the object position detection device (object position detection unit 36). In the case of FIG. 4, four people 54 (54a to 54d) are present in the left front of the vehicle 10, and attention areas 52 are set for each of them. As described above, the attention areas 52 are set so as to surround the people 54 extracted by comparison with a model trained according to a known technique. Therefore, each attention area 52 is set to a size (horizontal length x vertical length) according to the size of an object regarded as the extracted person 54, etc. The attention area 52 is defined, for example, by center coordinates, horizontal length, and vertical length (x, y, w, h) on the wide-angle image 50.

[0031] The candidate point setting unit 42 selects one of the attention areas 52 set on the wide-angle image 50, and sets a candidate point that may indicate the feet of the person 54. Fig. 5 is an exemplary schematic diagram showing a setting state of a reference point 56 and candidate points 58 used when detecting an object position in the object position detection unit 36 ​​(object position detection device). The candidate point setting unit 42 sets a plurality of candidate points 58 that may be candidates for the presence position (e.g., feet) of an object (e.g., person 54) on a boundary line 60 that defines the selected attention area 52 or in a position near the boundary line 60.

[0032] As described above, the attention area 52 is a substantially rectangular area that is set to surround the person 54. In addition, as shown in FIG. 4, the attention area 52 has, for example, four boundary lines 60 that are parallel to the horizontal frame line 50w and the vertical frame line 50h of the wide-angle image 50. When an object (e.g., a person 54) is surrounded by the attention area 52, the attention area 52 is set with the object (e.g., a person 54) at the center. In this case, if the person 54 is standing on the road surface (ground), when the wide-angle image 50 is divided into left and right at the center, if the person 54 is on the right side, the feet of the person 54 will be present on the side of the boundary line 60 that forms the left corner 52L of the attention area 52. Conversely, if the person 54 is on the left side of the wide-angle image 50, the feet of the person 54 will be present on the side of the boundary line 60 that forms the right corner 52R of the attention area 52. Therefore, when setting candidate point 58, candidate point setting unit 42 sets candidate point 58 on the side where an object (e.g., person 54) exists in the left or right region when wide-angle image 50 is divided into left and right at the center. As a result, it is possible to efficiently limit and set positions where the feet (grounding position) of an object (e.g., person 54) may exist. For example, in the case of FIG. 4, since the object (e.g., person 54) exists on the right side of wide-angle image 50, the feet of person 54 will exist on the side of boundary line 60 forming left corner 52L of attention area 52.

[0033] 5 shows the attention area 52a tilted so that an image recognized as a person 54a included in the attention area 52a is illustrated as standing approximately upright. As shown in FIG. 5, the candidate point setting unit 42 sets a plurality of candidate points 58 on each of the boundary lines 60a and 60b that form the corner portion 52L. In the case of FIG. 5, for the sake of simplicity, two candidate points 58 are shown on the boundary line 60a and three candidate points 58 are shown on the boundary line 60b, but in reality, for example, about 20 candidate points 58 can be set on each line.

[0034] FIG. 6 is an exemplary and schematic diagram showing in detail the setting of the candidate points 58. In FIG. 6, three candidate points 58 are shown on each of the boundary lines 60a and 60b for the sake of simplicity, but in reality, for example, about 20 points can be set on each of the boundary lines 60a and 60b, as described above. As shown in FIG. 6, each candidate point 58 is arranged at equal intervals on the boundary line 60 in a rectangular attention area 52a surrounding a person 54 (not shown), which is an object. As shown in FIG. 5, the attention area 52a is set so as to surround the person 54a (almost the entire person), so that the feet (ground position) of the person 54a are present on the corner 52L side. Therefore, in the attention area 52a, the area where the feet of the person 54a may be present can be considered to be the side of the boundary line 60 closer to the corner 52L. Therefore, the candidate point setting unit 42 sets the setting range of the candidate points 58 to, for example, a range of 1 / 2 of the boundary line 60 from the corner 52L. For example, in the case of the boundary line 60a, when the total length is h, the candidate points 58 are set at approximately equal intervals in the range of h / 2. Similarly, in the case of the boundary line 60b, when the total length is w, the candidate points 58 are set at approximately equal intervals in the range of w / 2. The intervals between the candidate points 58 on the boundary line 60a and the intervals between the candidate points 58 on the boundary line 60b may be the same or different. Also, as shown in FIG. 5, the position of the feet of the person 54a is often not necessarily at a position that coincides with the corner 52L. Rather, it is often at a position that is shifted from the corner 52L to the inner region of the attention region 52a or a position that is biased toward the boundary line 60a or the boundary line 60b. Therefore, if the candidate points 58 are set at the position of the corner 52L or at a position very close to the corner 52L, as described later, the representative point selection unit 44 may select an undesirable candidate point 58 when selecting a representative point 62 from among the candidate points 58. In order to avoid such inconvenience, the candidate point setting unit 42 provides a non-setting area for the candidate point 58. For example, a removal rate α is determined in advance in a test or the like to define an area where the candidate point 58 is unlikely to exist. In other words, the candidate point setting unit 42 does not set the candidate point 58 in an area defined by, for example, the boundary line length w of the horizontal frame of the attention area 52 × the removal rate α and the boundary line length h of the vertical frame × the removal rate α.In this way, by providing a region in which no candidate points 58 are set, it is possible to prevent inappropriate candidate points 58 from being used in the object position detection process.

[0035] 6 shows an example in which the candidate point 58 is set on the boundary line 60 of the region of interest 52, but the candidate point 58 does not necessarily have to be on the boundary line 60. For example, the candidate point 58 may be set near the boundary line 60 in an area inside the region of interest 52. Alternatively, the candidate point 58 may be set near the boundary line 60 in an area outside the region of interest 52.

[0036] Next, the representative point selection unit 44 determines a reference point 56 at a predetermined position within the region of interest 52, performs distortion correction on the reference point 56 and the candidate points 58, and selects a representative point 62 (see Figure 5) that can be regarded as the ground contact position (foot position) of an object (e.g., a person 54) from among the candidate points 58 after distortion correction.

[0037] First, the representative point selection unit 44 determines a reference point 56 that is a reference for object position detection (position detection of the person 54) in the attention area 52 selected as the processing target. As shown in FIG. 5, for example, when the person 54 is standing upright on the road surface (ground), there is a high possibility that the feet FO are present at a position where a normal line PL is drawn from the abdomen to the road surface. As described above, the attention area 52 is set so as to surround the entire object (e.g., the person 54) with the object (e.g., the abdomen S) as the center. That is, it can be considered that the center (e.g., the abdomen S) of the object (person 54) is present at the center of the attention area 52. Therefore, the representative point selection unit 44 considers that the center (e.g., the abdomen S) of the person 54 surrounded by the attention area 52 is present at the center (coordinates) of the attention area 52. That is, the center coordinates of the attention area 52 are set as the position (coordinates) of the reference point 56.

[0038] Next, the representative point selection unit 44 selects a representative point 62 from among a plurality of candidate points 58 set in the attention area 52. First, an example of panoramic transformation of the coordinates of processing target points (e.g., reference point 56 and candidate points 58 including representative point 62) by removing distortion from the wide-angle image 50 will be described with reference to Fig. 7. Fig. 7 is a diagram showing, as an example, a procedure M of panoramic transformation (conversion to polar coordinates) using equirectangular projection.

[0039] First, in step M1, the target point is transformed from the wide-angle (fisheye) image coordinates (u,v) to the perspective projection image coordinates (u′,v′). In other words, distortion is removed. Then, in step M2, the perspective projection image coordinates (u′,v′) are transformed to the camera coordinate system (x c ,y c ,z c ) The line of sight vector in the world coordinate system is a vector directed from the center of the camera coordinate system toward the target point T. Note that the diagram in step M2 is an image of the camera coordinate system space.

[0040] Next, in step M3, the camera coordinate system (x c ,y c ,z c ) to the world coordinate system (x w ,y w ,z w ). The line of sight vector in the world coordinate system is calculated by applying only the rotation matrix, since the origin is the center of the camera. At this point, the world coordinates are horizontal to the ground surface. This process ensures linearity in the vertical direction during panoramic conversion. Then, in step M4, the world coordinate system (x w ,y w ,z w ) to panoramic coordinates (Φ w ,θ w In this case, the azimuth and elevation angles in the world coordinate system are calculated and are used as panoramic coordinates in the equirectangular projection. w and θ w can be calculated using the following formula:

number

number

[0041] As described above, in the coordinate system transformed into the panoramic coordinate system, the panoramic coordinate (Φ w ,θ w 5, the feet FO of the person 54 are likely to be on the normal line PL drawn from the target point 52a (52). Therefore, the representative point selection unit 44 selects the panoramic coordinates (Φ w ,θ w )'s Φ w The closest Φ to the coordinates w A candidate point 58 having coordinates is selected as a representative point 62 .

[0042] FIG. 8 is an exemplary and schematic diagram showing a distorted image of a wide-angle image, which shows that distortion occurs in the radial direction of the wide-angle image 50 (fisheye image) by showing auxiliary lines 64 in the vertical direction of the wide-angle image 50. FIG. 8 shows that the auxiliary lines 64 are more distorted as they approach the periphery of the wide-angle image 50 (for example, in the left-right direction). On the wide-angle image 50, the feet of a person 54a standing upright on the road surface are displayed as if they are shifted in a direction parallel to the road surface. In FIG. 8, a plurality of candidate points 58 are shown as a candidate point group 58L. On the other hand, FIG. 9 is an exemplary and schematic diagram showing an image in which distortion in the vertical direction has been removed in the process of panoramic conversion using equirectangular projection. That is, each auxiliary line 64 is oriented in the vertical direction to the road surface. In this way, when distortion in the vertical direction has been removed, the feet FO of the person 54 should be on or near the normal line PL drawn from the reference point 56. In other words, if the horizontal direction (Φ w The candidate point 58 (closest Φ w It can be estimated that the candidate point 58 having coordinates is the foot FO (representative point 62). In FIG. 9, the multiple candidate points 58 are shown as a candidate point group 58L.

[0043] In this way, by identifying a minimum number of pixels (e.g., reference point 56, candidate point 58) from the wide-angle image 50 and performing panoramic transformation including distortion correction only on that image, it is possible to accurately obtain the position of the feet FO of an object (e.g., person 54) without normalizing the wide-angle image 50 itself. In other words, compared to performing panoramic transformation including distortion correction on the entire wide-angle image 50 (all images), it is possible to select the feet FO of the object (person 54), i.e., the representative point 62, while contributing to a reduction in the processing load and the computational resources.

[0044] Returning to Fig. 3, the coordinate acquisition unit 46 performs three-dimensional conversion on the coordinates of the selected representative point 62 using a known conversion method to acquire three-dimensional coordinates. Then, the output unit 48 outputs the acquired three-dimensional coordinates to another control system of the vehicle 10 as position information of an object (e.g., person 54) that is the target of position detection. For example, in an automatic driving system, the acquired three-dimensional coordinates are provided as object information for avoiding contact with the object.

[0045] The flow of object position detection processing by the object position detection device (object position detection unit 36) configured in this manner will be described with reference to the flowchart of FIG. 10, which shows an example.

[0046] First, the image acquisition unit 38 acquires a wide-angle image 50 from the imaging unit 14 (wide-angle camera) (S100). The image acquisition unit 38 may acquire the wide-angle image 50 including the traveling direction of the vehicle 10 based on, for example, the detection results of the shift sensor 30 and the steering angle sensor 28. Next, the region setting unit 40 sets an attention region 52 (bounding box) for an area including an image considered to be an object (for example, a person 54 or another vehicle) in the acquired wide-angle image 50 so as to surround the object individually (S102). If the attention region 52 is not set in the process of S102 (No in S104), that is, if it can be determined that no object considered to be a processing target exists in the wide-angle image 50, this flow is temporarily terminated.

[0047] When the attention area 52 is set in the process of S104 (Yes in S104), the candidate point setting unit 42 selects one of the attention areas 52 in the wide-angle image 50 (S106). For example, when there are multiple attention areas 52 for which the object position detection process has not been performed, the attention area 52 is selected according to a predetermined priority order. For example, the attention area 52 is selected in order of proximity to the vehicle 10.

[0048] When the attention area 52 is selected in the process of S106, the candidate point setting unit 42 sets a plurality of candidate points 58 at positions where the feet FO of an object (e.g., person 54) may be present, as described in FIG. 5 and FIG. 6 (S108). Next, the representative point selection unit 44 sets a reference point 56 at a predetermined position (e.g., center position) within the attention area 52 selected in S106, performs distortion correction on the reference point 56 and the candidate points 58, and selects a representative point 62 that can be regarded as the grounding position (feet FO) of the object (e.g., person 54) from among the candidate points 58 after distortion correction (S110). At this time, the representative point selection unit 44 performs panoramic transformation including distortion correction only on the candidate points 58 including the reference point 56 and the representative point 62, as described above. Then, the coordinate acquisition unit 46 performs three-dimensional transformation on the coordinates of the selected representative point 62 using a known transformation method to acquire three-dimensional coordinates (S112). Then, the output unit 48 outputs the acquired three-dimensional coordinates as position information of the object (for example, the person 54) that is the target of position detection, for example, to another control system of the vehicle 10 (S114).

[0049] When the output of the position information of the attention region 52 is completed in the process of S114, if the candidate point setting unit 42 has completed position detection for all attention regions 52 set in the wide-angle image 50 being processed (Yes in S116), the candidate point setting unit 42 temporarily ends this flow, waits for the timing of the next object position detection process, and repeats the process from S100. Also, if it is determined in the process of S116 that position detection for all attention regions 52 has not been completed (No in S116), the candidate point setting unit 42 proceeds to the process of S106, selects the next attention region 52 to be processed, and continues the process from S108 onwards.

[0050] In this way, the object detection device of this embodiment identifies a minimum number of pixels (e.g., reference point 56, candidate point 58) from the wide-angle image 50, and performs panoramic conversion including distortion correction only on that image. In other words, by not performing calculation processing on areas not involved in position detection, it is possible to efficiently detect the position of an object (person 54) while contributing to a reduction in the processing load and the computational resources.

[0051] The object position detection program for the object position detection process realized by the CPU 24a of this embodiment may be configured to be provided by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

[0052] Furthermore, the object location detection program for executing the object location detection process of this embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the posture estimation program executed in this embodiment may be provided or distributed via a network such as the Internet.

[0053] Although the present embodiment and the modified examples have been described above, the above-mentioned embodiment and modified examples are merely examples and are not intended to limit the scope of the invention. The above-mentioned novel embodiment and modified examples can be implemented in various forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The above-mentioned embodiment and modified examples are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0054] 10...vehicle, 14...imaging unit, 24...ECU, 24a...CPU, 36...object position detection unit, 38...image acquisition unit, 40...area setting unit, 42...candidate point setting unit, 44...representative point selection unit, 46...coordinate acquisition unit, 48...output unit, 50...wide-angle image, 52...area of ​​interest, 54...person, 56...reference point, 58...candidate point, 60...boundary line, 62...representative point, FO...feet, PL...normal.

Claims

1. an image acquisition unit that acquires imaging data of a wide-angle image captured by a wide-angle camera of the surrounding environment of the vehicle interior; a region setting unit that sets a region of interest surrounding a region in which an object is considered to exist in the wide-angle image; a candidate point setting unit that sets a plurality of candidate points that can be candidates for the location of the object on or near a boundary line that defines the region of interest; a representative point selection unit that determines a reference point at a predetermined position within the region of interest, performs distortion correction on the reference point and the candidate points, and selects a representative point that can be regarded as a grounding position of the object from among the candidate points after distortion correction; A coordinate acquisition unit that acquires three-dimensional coordinates of the representative point; an output unit that outputs position information of the grounding position of the object based on the three-dimensional coordinates; An object position detection device comprising:

2. The object position detection device according to claim 1 , wherein the representative point selection unit selects, as the representative point, the candidate point that is closest to a normal line that extends vertically downward from the reference point in a direction perpendicular to the normal line.

3. the region setting unit sets the rectangular region of interest surrounding the object; The object position detection device according to claim 1 , wherein the candidate point setting section sets a plurality of the candidate points at substantially equal intervals on a boundary line of the region of interest.

4. The object position detection device according to claim 1 , wherein the representative point selection section assumes that a central position of the object exists at a center of the region of interest, and sets the reference point at the center of the region of interest.

Images