Object position detection device

Abstract

The invention provides an object position detection device which can reduce processing load and computing resources even if an image captured by a wide-angle camera is used. The object position detection includes: an image acquisition unit that acquires imaging data of a wide-angle image obtained by imaging a situation around a vehicle cabin with a wide-angle camera; a region setting unit that sets a region of interest surrounding a region in the wide-angle image in which an object is deemed to be present; a candidate point setting unit that sets a plurality of candidate points that can be candidates for the presence position of the object on or near a boundary line defining the region of interest; a representative point selection unit that specifies a reference point at a predetermined position within the region of interest, performs distortion correction on the reference point and the candidate points, and selects a representative point that can be regarded as a ground contact position of the object from among the distortion-corrected candidate points; a coordinate acquisition unit that acquires three-dimensional coordinates of the representative point; and an output unit that outputs position information of the ground contact position of the object on the basis of the three-dimensional coordinates.

Description

Technical Field

[0001] The invention relates to an object position detection device. Background Art

[0002] In the past, various object position detection devices have been proposed as technologies for detecting objects around a vehicle, which detect objects (e.g., detection of obstacles such as people and other vehicles) in images taken by a vehicle-mounted camera. The object position detection device can detect the presence or absence of an object, detect the distance of the object, calculate the three-dimensional coordinates of the object, and obtain the moving speed and direction of the object when the object is moving, and can use these detection results for vehicle control. The vehicle-mounted camera used in such a vehicle-mounted object position detection device is expected to be able to obtain information (images) of the largest possible range with a small number of cameras, and a wide-angle camera (fisheye camera) equipped with a wide-angle lens (e.g., a fisheye lens) is often used.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2022-155102

[0004] Patent Document 2: Japanese Patent No. 6891954

[0005] In the prior art, the image captured by the wide-angle camera (fisheye camera) used in the object position detection device tends to be more distorted toward the peripheral portion. Therefore, in order to detect the correct position of the object, it is necessary to perform object detection processing after performing distortion correction on the acquired image. Therefore, there is room for improvement in the vehicle-mounted device where the processing load of the object position detection device is large and the computing resources are limited. Summary of the invention

[0006] Therefore, one of the objects of the present disclosure is to provide an object position detection device that can reduce processing load and calculation resources even when using an image captured by a wide-angle camera (fisheye camera).

[0007] An object position detection device as an example of the present disclosure includes: an image acquisition unit that acquires shooting data of a wide-angle image obtained by shooting the surrounding conditions of a vehicle cabin with a wide-angle camera; an area setting unit that sets a focus area that surrounds a region that can be considered as the existence of an object in the wide-angle image; a candidate point setting unit that sets a plurality of candidate points that can be candidates for the existence position of the object on or near a boundary line that defines the focus area; a representative point selection unit that determines a reference point at a predetermined position within the focus area, performs distortion correction on the reference point and the candidate points, and selects a representative point that can be considered as the grounding position of the object from the candidate points after the distortion correction; a coordinate acquisition unit that acquires the three-dimensional coordinates of the representative point; and an output unit that outputs position information of the grounding position of the object based on the three-dimensional coordinates. According to this configuration, for example, a distortion correction process is not performed on the entire wide-angle image, but only on the reference point and the candidate points (including the representative point), so that the processing load and computing resources can be reduced.

[0008] In addition, the representative point selection unit of the object position detection device may set the candidate point closest to the normal line drawn downward in the vertical direction relative to the reference point as the representative point. According to this structure, for example, the selection of representative points can be implemented more accurately and easily.

[0009] Alternatively, the region setting unit of the object position detection device may set the rectangular region of interest surrounding the object, and the candidate point setting unit may set a plurality of candidate points at approximately equal intervals on a boundary line of the region of interest. According to this configuration, for example, candidate points that can become representative points can be efficiently set.

[0010] In addition, the representative point selection unit of the object position detection device may set the reference point at the center of the focus area by considering that the center position of the object exists at the center of the focus area. According to this structure, for example, the reference point for selecting the representative point can be set more easily and appropriately. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is an illustrative and schematic plan view showing a vehicle on which the object position detection device according to the embodiment can be mounted.

[0012] Figure 2 This is an illustrative and schematic block diagram showing the configuration of a control system including the object position detection device according to the embodiment.

[0013] Figure 3This is an illustrative and schematic block diagram showing a configuration in which the object position detection device (object position detection unit) according to the embodiment is realized by a CPU.

[0014] Figure 4 This is an illustrative and schematic diagram showing a setting state of a region of interest in a wide-angle image used by the object position detection device according to the embodiment.

[0015] Figure 5 This is an illustrative and schematic diagram showing a state in which reference points and candidate points are set in a region of interest used when detecting an object position in the object position detection device according to the embodiment.

[0016] Figure 6 It is an illustrative and schematic diagram showing in detail the setting of candidate points in the object position detection device according to the embodiment.

[0017] Figure 7 This is an illustrative and schematic diagram of an image showing a panoramic conversion process used when selecting a representative point in the object position detection device according to the embodiment.

[0018] Figure 8 This is an illustrative and schematic image diagram showing that a wide-angle image used by the object position detection device according to the embodiment includes distortion in the vertical direction.

[0019] Fig. 9 This is an illustrative and schematic image diagram showing a state in which vertical distortion of a wide-angle image used by the object position detection device according to the embodiment is removed by panoramic conversion based on equidistant cylindrical projection.

[0020] Fig.10 This is an exemplary flowchart showing the flow of an object position detection process in the object position detection device according to the embodiment.

[0021] Description of Reference Numerals

[0022] 10... vehicle; 14... photographing unit; 24... ECU; 24a... CPU; 36... object position detecting unit; 38... image acquiring unit; 40... region setting unit; 42... candidate point setting unit; 44... representative point selecting unit; 46... coordinate acquiring unit; 48... output unit; 50... wide-angle image; 52... region of interest; 54... person; 56... reference point; 58... candidate point; 60... dividing line; 62... representative point; FO... under the foot; PL... normal line. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments and modified examples of the present disclosure will be described based on the drawings. The structures of the embodiments and modified examples described below and the functions and effects brought about by the structures are merely examples, and are not limited to the following descriptions.

[0024] The object position detection device of this embodiment obtains, for example, certain pixels for determining the foot position of the object in a focus area (for example, a bounding box) that is recognized as an area where an object exists on a road surface (ground) included in a captured image captured by a wide-angle camera (for example, a fisheye camera). Then, by performing distortion correction and other processing on the determined pixels, the foot position of the object is detected with processing under low processing load and low resources, and the detected pixels are output as the position information of the object.

[0025] Figure 1 1 is an illustrative and schematic top view of a vehicle 10 equipped with the object position detection device of the present embodiment. The vehicle 10 may be, for example, a car (internal combustion engine car) with an internal combustion engine (engine, not shown) as a driving source, or a car (electric car, fuel cell car, etc.) with an electric motor (motor, not shown) as a driving source, or a car (hybrid vehicle) with both of them as driving sources. In addition, the vehicle 10 may be equipped with various transmission devices, and may be equipped with various devices (systems, components, etc.) required to drive the internal combustion engine and the electric motor. In addition, the method, number, and layout of the devices related to the driving of the wheels 12 (front wheels 12F, rear wheels 12R) in the vehicle 10 may be set in a variety of ways.

[0026] like Figure 1 As shown in the example, four imaging units 14a to 14d are provided as the plurality of imaging units 14 in the vehicle 10. The imaging unit 14 is, for example, a digital camera having an imaging element such as a CCD (Charge Coupled Device) or a CIS (CMOS image sensor). The imaging unit 14 can output video data (captured image data) at a predetermined frame rate. Each of the imaging units 14 has a wide-angle lens (for example, a fisheye lens) and can capture a range of, for example, 140° to 220° in the horizontal direction. In addition, for example, the optical axis of the imaging unit 14 (14a to 14d) disposed on the outer periphery of the vehicle 10 can also be set to face obliquely downward. Therefore, the imaging unit 14 (14a to 14d) sequentially captures the peripheral conditions of the outside of the vehicle 10 including the road surface (ground) on which the vehicle 10 can move, marks (arrows, dividing lines, parking frames indicating parking spaces, vehicle line segments, etc.) marked on the road surface, and objects (for example, pedestrians, other vehicles, fixed objects, etc. as obstacles) existing on the road surface, and outputs them as captured image data.

[0027] The camera unit 14a is provided, for example, at the front side of the vehicle 10, that is, at the front side in the front-rear direction of the vehicle, at an end portion substantially in the center in the vehicle width direction, such as the front bumper 10a, the front grille, etc., so that a front image including the front end portion (e.g., the front bumper 10a) of the vehicle 10 can be captured. In addition, the camera unit 14b is provided, for example, at the rear side of the vehicle 10, that is, at the rear side in the front-rear direction of the vehicle, at an end portion substantially in the center in the vehicle width direction, such as a position above the rear bumper 10b, so that a rear area including the rear end portion (e.g., the rear bumper 10b) of the vehicle 10 can be captured. In addition, the camera unit 14c is provided, for example, at the end portion on the right side of the vehicle 10, such as the right rearview mirror 10c, so that a right side image including an area centered on the right side of the vehicle 10 (e.g., an area from the right front to the right rear) can be captured. The imaging unit 14d is provided at the left end of the vehicle 10, such as the left rearview mirror 10d, and can capture a left side image including an area centered on the left side of the vehicle 10 (eg, an area from the left front to the left rear).

[0028] For example, by performing calculation processing and image processing on each captured image data obtained by the imaging units 14a to 14d, images in various directions around the vehicle 10 can be displayed, or surrounding monitoring can be performed. In addition, by performing calculation processing and image processing based on each captured image data, an image with a wider viewing angle can be generated, or a virtual image (a bird's-eye view image (top view image), a side view image, a front view image, etc.) obtained by observing the vehicle 10 from above, in front, or to the side can be generated and displayed, or surrounding monitoring can be performed.

[0029] As described above, the captured image data captured by each camera unit 14 is displayed on a display device in the vehicle cabin to provide the driver and other users with the surrounding conditions of the vehicle 10. In addition, the captured image data is used to perform various detections such as detection of objects (for example, obstacles such as other vehicles and pedestrians), determination of positions, and measurement of distances, and the position information of the detected objects can be used to control the vehicle 10.

[0030] Figure 2This is an illustrative and schematic block diagram of the structure of a control system 100 including an object detection device mounted on a vehicle 10. A display device 16 and a sound output device 18 are provided in the interior of the vehicle 10. The display device 16 is, for example, an LCD (liquid crystal display), an OELD (organic electroluminescent display), etc. The sound output device 18 is, for example, a speaker. In addition, the display device 16 is covered by a transparent operation input unit 20 such as a touch panel. A user (for example, a driver) can visually confirm the image displayed on the display screen of the display device 16 via the operation input unit 20. In addition, the user can perform operation input by touching, pressing or moving the operation input unit 20 with a finger or the like at a position corresponding to the image displayed on the display screen of the display device 16. These display device 16, the sound output device 18, the operation input unit 20, etc. are provided in, for example, a monitoring device 22 located in, for example, the center of the vehicle width direction, i.e., the left-right direction, of the instrument panel of the vehicle 10. The monitoring device 22 can include an operation input unit (not shown) such as a switch, a dial, a dial, and a button. The monitoring device 22 can also be used as a navigation system or an audio system, for example.

[0031] In addition, if Figure 2 As shown in the example, the control system 100 (including the object position detection device) includes an ECU 24 (electronic control unit), a wheel speed sensor 26, a steering angle sensor 28, a shift sensor 30, a driving assist unit 32, etc. in addition to the imaging unit 14 (14a to 14d) and the monitoring device 22. In the control system 100, the ECU 24, the monitoring device 22, the wheel speed sensor 26, the steering angle sensor 28, the shift sensor 30, and the driving assist unit 32 are electrically connected via an in-vehicle network 34 as an electrical communication line. The in-vehicle network 34 is configured as a CAN (controller area network). The ECU 24 transmits control signals through the in-vehicle network 34, thereby being able to perform various system controls. In addition, the ECU 24 receives operation signals of the operation input unit 20 and various switches, and detection signals of various sensors such as the wheel speed sensor 26, the steering angle sensor 28, and the shift sensor 30, etc., via the in-vehicle network 34. In addition, although various systems (steering system, brake system, drive system, etc.) and various sensors for driving the vehicle 10 are connected to the in-vehicle network 34, Figure 2 In the description, structures that have little relevance to the object position detection device of this embodiment and their descriptions are omitted.

[0032] The ECU 24 is composed of a computer and the like, and manages the overall control of the vehicle 10 through the cooperation of hardware and software. Specifically, the ECU 24 includes a CPU (Central Processing Unit) 24a, a ROM (Read Only Memory) 24b, a RAM (Random Access Memory) 24c, a display control unit 24d, a sound control unit 24e, and an SSD (Solid State Drive) 24f.

[0033] The CPU 24a reads a program stored (installed) in a non-volatile storage device such as the ROM 24b, and performs calculation processing according to the program. For example, the CPU 24a performs image processing on the captured image captured by the camera 14, performs object position detection (recognition) to obtain the three-dimensional coordinates of the object, and can infer the position where the object exists, the distance to the object, and the moving speed and direction of the object when the object moves. In addition, for example, it can provide information required for controlling and operating the steering system, the brake system, the drive system, etc.

[0034] ROM24b stores various programs and parameters required for the execution of the programs, etc. RAM24c is used as a work area when CPU24a performs object position detection processing, and is used as a temporary storage area for various data (such as captured image data captured sequentially (in time series) by the camera 14) used in the calculation based on CPU24a. Display control unit 24d mainly performs image processing of image data obtained from the camera 14 and output to CPU24a, conversion of image data obtained from CPU24a to image data for display displayed on display device 16, etc. in the calculation processing based on ECU24. Sound control unit 24e mainly performs sound processing obtained from CPU24a and output by sound output device 18 in the calculation processing based on ECU24. SSD24f is a rewritable non-volatile storage unit that can continue to store data obtained from CPU24a even when the power of ECU24 is turned off. In addition, CPU24a, ROM24b, RAM24c, etc. can be integrated in the same package. In addition, the ECU 24 may use a DSP (digital signal processor) or other logic operation processor, logic circuit, etc. instead of the CPU 24a. In addition, a HDD (hard disk drive) may be provided instead of the SSD 24f, and the SSD 24f and the HDD may be provided separately from the ECU 24.

[0035] The wheel speed sensor 26 is a sensor that detects the rotation amount and the rotation speed per unit time of the wheel 12. The wheel speed sensor 26 is disposed on each wheel 12, and outputs the number of wheel speed pulses indicating the rotation speed detected by each wheel 12 as a sensor value. The wheel speed sensor 26 can be configured using, for example, a Hall element or the like. The CPU 24a calculates the vehicle speed, acceleration, etc. of the vehicle 10 based on the detection value obtained from the wheel speed sensor 26, and performs various controls.

[0036] The steering angle sensor 28 is a sensor that detects the steering amount of a steering control unit such as a steering wheel, for example. The steering angle sensor 28 is configured using, for example, a Hall element, etc. The CPU 24a obtains the steering amount of the steering control unit by the driver, for example, the steering amount of the front wheel 12F when performing automatic steering during parking assistance, from the steering angle sensor 28, and performs various controls.

[0037] The shift sensor 30 is a sensor that detects the position of the movable part (lever, arm, button, etc.) of the shift operating part, and detects information indicating the operating state of the transmission, the state of the gear stage, the drivable direction of the vehicle 10 (D gear: forward direction, R gear: reverse direction), etc.

[0038] The driving assist unit 32 provides control information to the steering system, the brake system, the drive system, etc. based on the moving path calculated by the control system 100 or the moving path provided from the outside, so as to realize driving assistance for moving the vehicle 10. The driving assist unit 32 performs, for example, fully automatic control for automatically controlling the steering system, the brake system, the drive system, etc., or performs semi-automatic control for automatically controlling a part of the steering system, the brake system, the drive system, etc. In addition, the driving assist unit 32 may also perform manual control for providing the driver with operation guidance for the steering system, the brake system, the drive system, etc., so that the driver performs the operation operation, so that the vehicle 10 can move along the moving path. In this case, the driving assist unit 32 may also provide operation information to the display device 16 and the sound output device 18. In addition, even when the driving assist unit 32 performs semi-automatic control, it can provide information related to the driver's operation, such as accelerator operation, via the display device 16 and the sound output device 18.

[0039] Figure 3 2 is a schematic block diagram showing an exemplary structure of the object position detection device (object position detection unit 36) according to the embodiment implemented by CPU 24a. CPU 24a executes the object position detection program read from ROM 24b, such as Figure 3As shown, the image acquisition unit 38, the region setting unit 40, the candidate point setting unit 42, the representative point selection unit 44, the coordinate acquisition unit 46, and the output unit 48 are implemented. In addition, the image acquisition unit 38, the region setting unit 40, the candidate point setting unit 42, the representative point selection unit 44, the coordinate acquisition unit 46, and the output unit 48 may be partially or entirely constituted by hardware such as circuits. Figure 3 Although not shown in the figure, the CPU 24a can also realize various modules required for the driving of the vehicle 10. Figure 2 , the CPU 24 a that mainly executes the object position detection process is shown, but a CPU for realizing various modules required for the driving of the vehicle 10 may be provided, and an ECU different from the ECU 24 may be provided.

[0040] The image acquisition unit 38 acquires a captured image (wide-angle (fish-eye) image) representing the surrounding conditions of the vehicle 10 including the road surface (ground) on which the vehicle 10 is located, captured by the image capturing unit 14 (wide-angle camera, fish-eye camera, etc.), and supplies the captured image to the area setting unit 40. In addition, the image acquisition unit 38 may sequentially acquire the captured images captured by each image capturing unit 14 (14a to 14d) and supply the captured images to the area setting unit 40. In another example, the image acquisition unit 38 may selectively acquire a captured image of a traversable direction and supply the captured image to the area setting unit 40 so that object detection in the traversable direction can be performed based on the traversable direction (forward direction or backward direction) of the vehicle 10 that can be acquired from the shift sensor 30 and the information related to the turning direction of the vehicle 10 that can be acquired from the steering angle sensor 28.

[0041] The region setting unit 40 sets a rectangular region of interest (e.g., a rectangular bounding box) for selecting a predetermined target object (e.g., an object corresponding to a person or other vehicle) from among the objects contained in the wide-angle image acquired by the image acquisition unit 38. That is, a region is set to enclose a region in which the object to be processed can be considered to exist in the wide-angle image. The object enclosed by the region of interest is an object that can exist on the road surface (ground) around the vehicle 10, and for example, in addition to movable objects corresponding to a person or other vehicle (including bicycles, etc.), it can also include stationary objects corresponding to fences, electric poles, roadside trees, flower beds, etc. The object enclosed by the region of interest is an object that can be extracted by referring to a model that has been learned in advance, for example, by machine learning. The region setting unit 40 applies a model learned according to a known technique to the wide-angle image, for example, obtains the degree of consistency between the feature points of the object as the model and the feature points of the image contained in the wide-angle image, and detects where the known object exists on the wide-angle image. Then, a region of interest (bounding box) of, for example, a roughly rectangular shape is set to enclose the detected object.

[0042] Figure 4 1 is an illustrative and schematic diagram showing the setting state of the attention area 52 (52a to 52d) in the wide-angle image 50 used by the object position detection device (object position detection unit 36). Figure 4 In the case of , there are four people 54 (54a to 54d) in the left front of the vehicle 10, and the region of interest 52 is set for each of them. As described above, the region of interest 52 is set to surround the person 54 extracted by comparison with the model learned according to the known technology. Therefore, each region of interest 52 is set to a size (horizontal length × vertical length) corresponding to the size of an object that can be regarded as the extracted person 54. The region of interest 52 is defined by the central coordinates, horizontal length, and vertical length (x, y, w, h) on the wide-angle image 50, for example.

[0043] The candidate point setting unit 42 selects one from the target area 52 set on the wide-angle image 50 and sets a candidate point that can represent the foot of the person 54 . Figure 5 This is an illustrative and schematic diagram showing the setting state of the reference point 56 and the candidate point 58 used when detecting the object position in the object position detection unit 36 ​​(object position detection device). The candidate point setting unit 42 sets a plurality of candidate points 58 that can be candidates for the existence position (e.g., under the feet) of the object (e.g., person 54) on or near the boundary line 60 defining the selected target area 52.

[0044] As described above, the region of interest 52 is a substantially rectangular region that surrounds the person 54. Figure 4 As shown, for the region of interest 52, for example, four dividing lines 60 are set so as to be parallel to the horizontal frame line 50w and the vertical frame line 50h of the wide-angle image 50. When an object (for example, a person 54) is surrounded in such a region of interest 52, the region of interest 52 is set with the object (for example, a person 54) as the center. In this case, when considering that the person 54 is standing on the road surface (ground), when the wide-angle image 50 is divided into left and right at the center and the person 54 is on the right side, the foot of the person 54 is on the side of the dividing line 60 forming the corner 52L on the left side of the region of interest 52. On the contrary, when the person 54 is on the left side of the wide-angle image 50, the foot of the person 54 is on the side of the dividing line 60 forming the corner 52R on the right side of the region of interest 52. Therefore, when setting the candidate point 58, the candidate point setting unit 42 sets the candidate point 58 on the side where the object (e.g., person 54) exists in the left area or the right area when the wide-angle image 50 is divided into left and right areas at the center. As a result, the position where the feet (ground contact position) of the object (e.g., person 54) can exist can be limited and set efficiently. For example, Figure 4In the case where the object (eg, person 54 ) exists on the right side of the wide-angle image 50 , the foot of person 54 exists on the side of the boundary line 60 forming the corner 52L on the left side of the focus area 52 .

[0045] Figure 5 The region of interest 52a is shown tilted so that the image of the person 54a considered to be included in the region of interest 52a is shown in a substantially upright manner. Figure 5 As shown in FIG. 5 , the candidate point setting unit 42 forms the corner 52L and sets a plurality of candidate points 58 on each of the boundary line 60a and the boundary line 60b. Figure 5 In order to simplify the drawing, two candidate points 58 are shown on the boundary line 60a and three candidate points 58 are shown on the boundary line 60b, but in reality, for example, about 20 candidate points 58 can be set for each.

[0046] Figure 6 Detailed description of the setting of the candidate point 58 is provided in an exemplary and schematic manner. Figure 6 In order to simplify the drawing, three candidate points 58 are shown on the boundary line 60a and the boundary line 60b, respectively. However, in reality, as described above, for example, about 20 candidate points can be set. Figure 6 As shown, a plurality of candidate points 58 are arranged at equal intervals on a boundary line 60 in a rectangular region of interest 52a surrounding a person 54 (not shown) as an object. Figure 5 As shown, the focus area 52a is set to surround the person 54a (roughly in its entirety), so the feet of the person 54a (the ground contact position) exist on the side of the corner 52L. Therefore, in the focus area 52a, a certain area where the feet of the person 54a may exist can be regarded as being on the side of the boundary 60 close to the corner 52L. Therefore, the candidate point setting unit 42 can set the setting range of the candidate point 58 to a range from the corner 52L, for example, to 1 / 2 of the boundary 60. For example, in the case of the boundary 60a, when the total length is set to h, the candidate points 58 are set at approximately equal intervals within the range of h / 2. Similarly, in the case of the boundary 60b, when the total length is set to w, the candidate points 58 are set at approximately equal intervals within the range of w / 2. In addition, the intervals between the candidate points 58 at the boundary 60a and the intervals between the candidate points 58 at the boundary 60b may be the same as or different from the intervals between the candidate points 58 at the boundary 60b. In addition, as Figure 5As shown, the position of the foot of the person 54a is often not necessarily the same as the position of the corner 52L. On the contrary, it is often the position offset from the corner 52L to the inner area of ​​the region of interest 52a, or the position deviated from the boundary line 60a and the boundary line 60b. Therefore, if the candidate point 58 is set at the position of the set corner 52L or the position very close to the corner 52L, as described later, the representative point selection unit 44 may select an unpreferable candidate point 58 when selecting the representative point 62 from the candidate points 58. In order to avoid such inconvenience, the candidate point setting unit 42 sets a non-setting area for the candidate point 58. For example, the removal rate α for specifying an area where the possibility of the candidate point 58 existing in an experiment is low is predetermined. That is, the candidate point setting unit 42 does not set the candidate point 58 in the area specified by the boundary line length w of the horizontal frame of the region of interest 52×the removal rate α and the boundary line length h of the vertical frame×the removal rate α. In this way, by providing the non-setting area for the candidate points 58 , it is possible to suppress inappropriate candidate points 58 from being used in the object position detection process.

[0047] In addition, Figure 6 In the example of FIG. 5 , the candidate point 58 is set on the boundary line 60 of the region of interest 52, but the candidate point 58 is not necessarily on the boundary line 60. For example, the candidate point 58 may be set near the boundary line 60 in the region inside the region of interest 52. In addition, the candidate point 58 may be set near the boundary line 60 in the region outside the region of interest 52.

[0048] Next, the representative point selection unit 44 determines a reference point 56 at a predetermined position within the region of interest 52, performs distortion correction on the reference point 56 and the candidate points 58, and selects a representative point 62 (see FIG. 6 ) that can be regarded as a ground contact position (foot position) of an object (e.g., person 54) from among the candidate points 58 after distortion correction. Figure 5 ).

[0049] First, the representative point selection unit 44 determines the reference point 56 that serves as the reference for object position detection (position detection of the person 54) in the target area 52 selected as the processing target. Figure 5 As shown, for example, when a person 54 is standing upright on the road surface (ground), there is a high possibility that the foot FO is located at a position where the normal line PL is drawn from the abdomen to the road surface. Moreover, as described above, the area of ​​interest 52 is set to surround the entire object (e.g., person 54) with the object (e.g., person 54) as the center. That is, it can be considered that the center of the object (person 54) (e.g., abdomen S) exists in the center of the area of ​​interest 52. Therefore, the representative point selection unit 44 considers that the center (coordinates) of the person 54 surrounded by the area of ​​interest 52 exists in the center (coordinates) of the area of ​​interest 52. That is, the center coordinates of the area of ​​interest 52 are set as the position (coordinates) of the reference point 56.

[0050] Next, the representative point selection unit 44 selects a representative point 62 from the plurality of candidate points 58 set in the target region 52. Figure 7 An example of removing distortion from the wide-angle image 50 and performing panoramic conversion on the coordinates of processing target points (for example, candidate points 58 including the reference point 56 and the representative point 62 ) will be described. Figure 7 As an example, this is a diagram showing step M of panoramic conversion (conversion into polar coordinates) based on equidistant cylindrical projection.

[0051] First, in step M1, the object point is converted from the wide-angle (fisheye) image coordinates (u, v) to the perspective projection image coordinates (u', v'). That is, the distortion is removed. Next, in step M2, the perspective projection image coordinates (u', v') are converted to the camera coordinate system (x c ,y c 、z c ). The sight line vector in the world coordinate system is a vector pointing from the camera coordinate center toward the object point T. In addition, the graph of step M2 is an image of the camera coordinate system space.

[0052] Next, in step M3, the camera coordinate system (xc, yc, zc) is converted to the world coordinate system (xw, yw, zw). The sight line vector in the world coordinate system takes the center of the camera as the origin, so it is only calculated by applying the rotation matrix. At this moment, it becomes a world coordinate horizontal to the ground (ground). Through this process, the linearity of the vertical direction during the panoramic transformation is ensured. Then, in step M4, the world coordinate system (xw, yw, zw) is changed to the panoramic coordinate system (Φw, θw). In this case, the azimuth and elevation in the world coordinate system are calculated and set as the panoramic coordinates in the equidistant cylindrical projection. In addition, Φw and θw can be calculated by the following formula.

[0053]

Number 1

[0054] 0warcsin(w)

[0055]

Number 2

[0056]

[0057] As described above, in the coordinate system converted into the panoramic coordinates, it is highly likely that the foot FO of the person 54 exists on the normal line PL drawn downward from the panoramic coordinates (Φw, θw) corresponding to the reference point 56 (u, v). Figure 5 As shown, the representative point selection unit 44 selects, as the representative point 62 , the candidate point 58 having the Φw coordinate closest to the Φw coordinate of the panoramic coordinates (Φw, θw) corresponding to the reference point 56 (u, v) in the target area 52 a ( 52 ).

[0058] Figure 8 6 is an illustrative and schematic diagram showing a distorted image of a wide-angle image, and shows that distortion occurs in the radial direction of the wide-angle image 50 (fisheye image) by showing auxiliary lines 64 in the vertical direction of the wide-angle image 50. Figure 8 , the auxiliary line 64 is shown to be more distorted as it moves toward the periphery (for example, left and right) of the wide-angle image 50. On the wide-angle image 50, the feet of the person 54a who is shown standing upright relative to the road surface are offset in a direction parallel to the road surface. Figure 8 In FIG. 5 , a plurality of candidate points 58 are shown as a candidate point group 58L. On the other hand, Fig. 9 6 is an illustrative and schematic diagram showing an image in which the distortion in the vertical direction is removed during the panoramic conversion based on the equidistant cylindrical projection. That is, each auxiliary line 64 is oriented in the vertical direction relative to the road surface. In this way, when the distortion in the vertical direction is removed, the foot FO of the person 54 should be on or near the normal line PL drawn from the reference point 56. In other words, it can be inferred that the foot FO exists in the direction closest to the horizontal direction (Φ) relative to the normal line PL. w coordinates) of the position of the candidate point 58 (with the closest Φ w The candidate point 58 of the coordinates is FO under the foot (representative point 62). Fig. 9 , a plurality of candidate points 58 are shown as a candidate point group 58L.

[0059] In this way, by determining the minimum number of pixels (e.g., reference point 56, candidate point 58) from wide-angle image 50 and performing panoramic conversion including distortion correction only on the image, the position of the foot FO of the object (e.g., person 54) can be accurately obtained without normalizing the wide-angle image 50 itself. That is, compared with the case where panoramic conversion including distortion correction is performed on the entire wide-angle image 50 (the entire image), it contributes to the reduction of processing load and calculation resources, and the foot FO of the object (person 54), that is, the selection of representative point 62 can be performed.

[0060] Return to Figure 3 The coordinate acquisition unit 46 performs three-dimensional conversion on the coordinates of the selected representative point 62 using a known conversion method to obtain three-dimensional coordinates. Then, the output unit 48 outputs the obtained three-dimensional coordinates as position information of an object (e.g., person 54) to be detected, and outputs it to other control systems of the vehicle 10. For example, in an automatic driving system, it is provided as object information for avoiding contact with an object.

[0061] Using illustrative Fig.10 The flow chart of the object position detection process based on the object position detection device (object position detection unit 36) configured in this way will be described.

[0062] First, the image acquisition unit 38 acquires the wide-angle image 50 from the imaging unit 14 (wide-angle camera) (S100). The image acquisition unit 38 may acquire the wide-angle image 50 including the traveling direction of the vehicle 10, for example, based on the detection results of the shift sensor 30 and the steering angle sensor 28. Next, the region setting unit 40 sets the focus region 52 (bounding box) (S102) for the region of the image that can be regarded as an object (e.g., a person 54, another vehicle, etc.) in the acquired wide-angle image 50, so as to surround the object alone. In the process of S102, if the focus region 52 is not set (No in S104), that is, if it can be determined that there is no object that can be regarded as the processing object in the wide-angle image 50, the process is temporarily terminated.

[0063] In the process of S102, when the attention area 52 is set (Yes in S104), the candidate point setting unit 42 selects any one of the attention areas 52 in the wide-angle image 50 (S106). For example, when there are a plurality of attention areas 52 for which the object position detection process is not performed, the attention area 52 is selected according to a predetermined priority order. For example, the attention area 52 is selected in the order of approaching the vehicle 10.

[0064] In the process of S106, when the region of interest 52 is selected, as shown in FIG. Figure 5 , Figure 6 As described above, the candidate point setting unit 42 sets a plurality of candidate points 58 at positions where the FO under the feet of an object (e.g., a person 54) may exist (S108). Next, the representative point selection unit 44 sets a reference point 56 at a predetermined position (e.g., a central position) within the region of interest 52 selected in S106, and performs distortion correction on the reference point 56 and the candidate point 58, and selects a representative point 62 that can be regarded as the ground contact position (FO under the feet) of the object (e.g., a person 54) from the candidate points 58 after distortion correction (S110). As described above, at this time, the representative point selection unit 44 performs panoramic conversion including distortion correction only for the candidate points 58 including the reference point 56 and the representative point 62. Then, the coordinate acquisition unit 46 performs three-dimensional conversion on the coordinates of the selected representative point 62 using a known conversion method, thereby acquiring three-dimensional coordinates (S112). Then, the output unit 48 outputs the acquired three-dimensional coordinates as position information of an object (eg, person 54 ) to be detected, for example, to another control system of the vehicle 10 ( S114 ).

[0065] In the process of S114, when the output of the position information of the attention area 52 is completed, the candidate point setting unit 42 temporarily ends the process when the position detection of all the attention areas 52 set in the wide-angle image 50 being processed is completed (Yes in S116), waits for the opportunity of the next object position detection process, and repeats the process from S100. In addition, in the process of S116, when it is determined that the position detection of all the attention areas 52 is not completed (No in S116), the candidate point setting unit 42 moves to the process of S106, selects the attention area 52 of the next processing object, and continues the process after S108.

[0066] Thus, in the object position detection device of this embodiment, the minimum number of pixels (e.g., reference point 56, candidate point 58) is determined from the wide-angle image 50, and the panoramic conversion including distortion correction is performed only on the image. That is, the operation processing of the area not involved in the position detection is not performed, thereby contributing to the reduction of the processing load and the reduction of the computing resources, and the position detection of the object (person 54) can be performed efficiently.

[0067] In addition, the object position detection program for object position detection processing implemented by the CPU 24a of this embodiment can also be provided by recording it in a computer-readable recording medium such as a CD-ROM, a floppy disk (FD), a CD-R, or a DVD (Digital Versatile Disk) in an installable or executable form.

[0068] Furthermore, the object position detection program for executing the object position detection processing of the present embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading the program via the network. In addition, the posture estimation program executed in the present embodiment may be provided or distributed via a network such as the Internet.

[0069] Although the present embodiment and variations are described above, the above embodiment and variations are only examples and are not intended to limit the scope of the invention. The above new embodiment and variations can be implemented in various ways, and various omissions, substitutions, and changes can be made without departing from the scope of the subject matter of the invention. The above embodiment and variations are included in the scope and subject matter of the invention, and are included in the invention described in the claims and the scope of its equivalent.

Claims

1. An object position detection device, wherein: include: an image acquisition unit that acquires captured data of a wide-angle image captured by a wide-angle camera of a condition surrounding the vehicle interior; an area setting unit that sets an area of ​​interest surrounding an area where an object can be considered to exist in the wide-angle image; A candidate point setting unit that sets a plurality of candidate points that can be candidates for the existence position of the object on a boundary line that defines the target area or at a position near the boundary line; a representative point selection unit that determines a reference point at a predetermined position within the region of interest, performs distortion correction on the reference point and the candidate points, and selects a representative point that can be regarded as a ground contact position of the object from among the candidate points after the distortion correction; A coordinate acquisition unit, which acquires the three-dimensional coordinates of the representative point; as well as The output unit outputs position information of a ground contact position of the object based on the three-dimensional coordinates.

2. The object position detection device according to claim 1, wherein: The representative point selection unit sets the candidate point at the closest position to a normal line drawn vertically downward from the reference point in a direction perpendicular to the normal line as the representative point.

3. The object position detection device according to claim 1, wherein: The region setting unit sets the rectangular region of interest surrounding the object. The candidate point setting unit sets the plurality of candidate points at substantially equal intervals on the boundary line of the region of interest.

4. The object position detection device according to claim 1, wherein: The representative point selection unit considers that the center position of the object exists at the center of the target area, The reference point is set at the center of the region of interest.

Images