Information processing device, information processing method, and information processing program

Abstract

To accurately identify a three-dimensional position of an obstacle existing in the vicinity of a vehicle.SOLUTION: An information processing apparatus includes: an acquisition unit that acquires an image captured by an imaging unit provided on a hinge door of a vehicle and a door open angle indicating an angle at which the hinged door is opened from a closed state when the image is captured; a correction unit that corrects an error of the door open angle acquired by the acquisition unit; and an identification unit that identifies a three-dimensional position of an obstacle with respect to the vehicle using a corresponding point of the obstacle existing around the hinge door determined based on the plurality of images captured by the imaging unit at a plurality of viewpoints with the different door open angles acquired by the acquisition unit, and the corrected door open angles which are associated with the respective plurality of images and are corrected by the correction unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Patent Document 1 discloses a technique in which a power door control unit opens a power hinge door when a vehicle user approaches the power hinge door. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-147856 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, there is a conventional technique for automatically opening a hinged door of a vehicle without human intervention. When automatically opening a hinged door in a situation where there are obstacles such as walls and other vehicles around the vehicle, the door must be opened within a range that does not come into contact with the obstacles. To open the door without coming into contact with the obstacles, it is necessary to accurately identify the three-dimensional position of the obstacle relative to the vehicle, but there is still room for improvement in the method for identifying the three-dimensional position of the obstacle.

[0005] Therefore, an object of the present invention is to provide an information processing device, an information processing method, and an information processing program that can accurately identify the three-dimensional position of an obstacle present around a vehicle. [Means for solving the problem]

[0006] The information processing device of the first aspect includes an acquisition unit that acquires an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle that indicates the angle by which the hinged door was opened from a closed state when the image was captured; a correction unit that corrects an error in the door opening angle acquired by the acquisition unit; and an identification unit that identifies the three-dimensional position of the obstacle relative to the vehicle using corresponding points of an obstacle present around the hinged door, determined based on a plurality of images captured by the imaging unit from a plurality of viewpoints with different door opening angles, and the corrected door opening angle corrected by the correction unit that is linked to each of the plurality of images.

[0007] The information processing device of the second aspect is the information processing device of the first aspect, and is equipped with a determination unit that determines the maximum door opening angle at which the hinged door does not come into contact with the obstacle, using the three-dimensional position of the obstacle identified by the identification unit and door information regarding the shape and dimensions of the hinged door.

[0008] The information processing device of the third aspect is the information processing device of the second aspect, and is equipped with a control unit that controls the hinged door to open to the maximum door opening angle determined by the determination unit so as not to come into contact with the obstacle.

[0009] An information processing device of a fourth aspect is the information processing device of the third aspect, wherein the control unit determines the door opening angle when the imaging unit captures the image, and controls the hinged door to open up to the determined door opening angle.

[0010] An information processing device of a fifth aspect is the information processing device of the third aspect, wherein, after the determination unit determines the maximum door opening angle at which the hinged door will not come into contact with the obstacle, the control unit controls the opening of the hinged door to a predetermined angle within the range of the maximum door opening angle at which the image is not captured by the imaging unit, and the identification unit re-identifies a three-dimensional position of the obstacle relative to the vehicle using a corresponding point of the obstacle determined based on a plurality of images captured by the imaging unit from a viewpoint of the predetermined angle and a viewpoint of another door opening angle, and the corrected door opening angle corrected by the correction unit associated with each of the plurality of images, and the determination unit re-determines the maximum door opening angle at which the hinged door will not come into contact with the obstacle using the three-dimensional position of the obstacle re-identified by the identification unit and the door information.

[0011] An information processing device of a sixth aspect is the information processing device of the fifth aspect, wherein the control unit determines the specified angle according to the maximum door opening angle at which the door does not come into contact with the obstacle, as determined by the determination unit.

[0012] An information processing device of a seventh aspect is an information processing device of the fifth aspect, and is provided with a receiving unit that receives input of the number of times the determination unit determines the maximum door opening angle at which the hinged door does not contact the obstacle.

[0013] An information processing device of an eighth aspect is the information processing device of the third aspect, wherein the control unit performs control to prohibit the hinge door from opening based on a detection result of a distance measurement sensor provided on the hinge door.

[0014] An information processing device of a ninth aspect is the information processing device of the first aspect, wherein the correction unit corrects the image captured by the imaging unit using an internal parameter of the imaging unit.

[0015] In a tenth aspect of the information processing method, a computer executes a process to acquire an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating the angle at which the hinged door was opened from a closed state when the image was captured, correct an error in the acquired door opening angle, and identify the three-dimensional position of the obstacle relative to the vehicle using corresponding points of an obstacle present around the hinged door, determined based on a plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the corrected door opening angle associated with each of the plurality of images.

[0016] An information processing program of an eleventh aspect causes a computer to execute a process of acquiring an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating the angle by which the hinged door was opened from a closed state when the image was captured, correcting an error in the acquired door opening angle, and identifying the three-dimensional position of the obstacle relative to the vehicle using corresponding points of an obstacle present around the hinged door determined based on a plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the corrected door opening angle associated with each of the plurality of images. [Effects of the Invention]

[0017] The information processing device, the information processing method, and the information processing program according to the present invention can accurately identify the three-dimensional position of an obstacle present around a vehicle. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a first block diagram showing a hardware configuration of a vehicle. [Figure 2] FIG. 2 is a first block diagram showing an example of the functional configuration of the vehicle-mounted device. [Figure 3] 10 is a first flowchart showing the flow of a release process. [Figure 4] FIG. 1 is a first explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle. [Figure 5] FIG. 10 is a second explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle. [Figure 6] FIG. 2 is a second block diagram showing an example of the functional configuration of the vehicle-mounted device. [Figure 7] 10 is a second flowchart showing the flow of the release process. [Figure 8] FIG. 2 is a second block diagram showing the hardware configuration of the vehicle. [Figure 9] 10 is a third flowchart showing the flow of the release process. DETAILED DESCRIPTION OF THE INVENTION

[0019] The vehicle 20 according to this embodiment will be described below. (First embodiment) FIG. 1 is a first block diagram showing the hardware configuration of a vehicle 20. The vehicle 20 may be any of a gasoline vehicle, a hybrid vehicle, and an electric vehicle, but in the first embodiment, as an example, the vehicle 20 is a gasoline vehicle. The vehicle 20 also has a driver's door on the driver's side, a passenger door on the passenger's side, and a rear door at the rear of the vehicle 20. In the first embodiment, the driver's door of the vehicle 20 is configured as a hinged door whose rotation axis in the vehicle body coordinate system is known. The driver's door is an example of a "hinged door."

[0020] 1, the vehicle 20 includes an on-board device 15, a door ECU (Electronic Control Unit) 30, an actuator 31, an angle sensor 32, a microphone 40, a camera 41, an input switch 42, a monitor 43, a speaker 44, and a GPS device 45. The on-board device 15 is an example of an "information processing device."

[0021] The vehicle-mounted device 15 includes a CPU 21 (Central Processing Unit), a ROM 22 (Read Only Memory), a RAM 23 (Random Access Memory), a storage unit 24, an in-vehicle communication I / F (Interface) 25, an input / output I / F 26, and a wireless communication I / F 27. The CPU 21, the ROM 22, the RAM 23, the storage unit 24, the in-vehicle communication I / F 25, the input / output I / F 26, and the wireless communication I / F 27 are connected to each other via an internal bus 28 so as to be able to communicate with each other.

[0022] The CPU 21 is a central processing unit that executes various programs and controls each part. That is, the CPU 21 reads programs from the ROM 22 or the storage unit 24 and executes the programs using the RAM 23 as a work area. The CPU 21 controls each of the above components and performs various arithmetic processing in accordance with the programs recorded in the ROM 22 or the storage unit 24.

[0023] The ROM 22 stores various programs and various data. The RAM 23 serves as a working area for temporarily storing programs or data.

[0024] The storage unit 24 is configured with a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and stores various programs and various data. The storage unit 24 stores at least an information processing program for executing the release process described below.

[0025] The in-vehicle communication I / F 25 is an interface for connecting to the door ECU 30. The interface uses a communication standard based on the CAN protocol. The in-vehicle communication I / F 25 is connected to an external bus 29.

[0026] In the first embodiment, the ECU provided is a door ECU 30. Although not shown in the figure, a plurality of ECUs are provided for each function of the vehicle 20, and the vehicle 20 is configured to include ECUs other than the door ECU 30. The door ECU 30 is connected to an actuator 31 and an angle sensor 32 .

[0027] The actuator 31 automatically drives and opens / closes at least the driver's door among the doors of the vehicle 20. In the first embodiment, the door ECU 30 drives the actuator 31 based on the control of the in-vehicle device 15, so that the driver's door can be automatically opened / closed without an occupant performing an opening / closing operation of the driver's door.

[0028] The angle sensor 32 is provided on at least the driver's door among the doors of the vehicle 20, and is a sensor for detecting the door opening angle when the driver's door is in a closed state, i.e., the angle by which the door is opened from the time the door is closed. The door opening angle detected by the angle sensor 32 is stored in the memory unit 24.

[0029] The input / output I / F 26 is an interface for communicating with a microphone 40 , a camera 41 , an input switch 42 , a monitor 43 , a speaker 44 , and a GPS device 45 that are mounted on the vehicle 20 .

[0030] The microphone 40 is a device that is provided on the front pillar or dashboard of the vehicle 20 and collects sounds made by the user of the vehicle 20 .

[0031] The camera 41 includes, for example, a solid-state imaging element such as a charge coupled device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) image sensor. For example, the camera 41 is provided at least on the door mirror 33 of the driver's door of the vehicle 20 (see FIGS. 4 and 5), and captures images of the sides of the vehicle. The images captured by the camera 41 are associated with the door opening angle at the time when each image was captured and stored in the memory unit 24. The camera 41 may be connected to the in-vehicle device 15 via an ECU (for example, a camera ECU). The camera 41 is an example of an "imaging unit."

[0032] Furthermore, the orientation of the camera 41 in the vehicle body coordinate system when the driver's door is closed is known, and information relating to this orientation is stored in the storage unit 24.

[0033] The input switch 42 is provided on an instrument panel, a center console, a steering wheel, or the like, and is a switch that inputs an operation by the driver's finger. For example, a push-button numeric keypad, a touchpad, or the like can be used as the input switch 42. In the first embodiment, at least an opening switch for opening the driver's door is provided as the input switch 42. In the first embodiment, the driver's door can be automatically opened by operating the opening switch while the vehicle 20 is stopped or parked.

[0034] The monitor 43 is a liquid crystal monitor provided on an instrument panel, a meter panel, or the like, for displaying images related to operation suggestions and explanations of functions of the vehicle 20. The monitor 43 may be provided as a touch panel that also serves as the input switch 42.

[0035] The speaker 44 is provided on the instrument panel, the center console, the front pillar, the dashboard, or the like, and is a device for outputting audio suggesting the operation of functions of the vehicle 20 and explaining the functions. The speaker 44 may also be provided on the monitor 43.

[0036] The GPS device 45 is a device that measures the current position of the vehicle 20. The GPS device 45 includes an antenna (not shown) that receives signals from GPS satellites. The GPS device 45 may be connected to the vehicle-mounted device 15 via a car navigation system that is connected to an ECU (for example, a multimedia ECU).

[0037] The wireless communication I / F 27 is a wireless communication module for communicating with other devices, and uses communication standards such as 5G, LTE, and Wi-Fi (registered trademark).

[0038] Next, the functional configuration of the vehicle-mounted device 15 will be described. FIG. 2 is a first block diagram showing an example of the functional configuration of the vehicle-mounted device 15. As shown in FIG.

[0039] 2, the CPU 21 of the vehicle-mounted device 15 has, as functional components, an acquisition unit 21A, a correction unit 21B, an identification unit 21C, a determination unit 21D, and a control unit 21E. Each functional component is realized by the CPU 21 reading and executing an information processing program stored in the storage unit 24.

[0040] The acquisition unit 21A acquires an image captured by the camera 41 and the door opening angle at which the image was captured. In the first embodiment, the acquisition unit 21A acquires a plurality of images captured by the camera 41 from a plurality of viewpoints with different door opening angles, and the door opening angle associated with each of the plurality of images.

[0041] The correction unit 21B corrects an error in the door opening angle acquired by the acquisition unit 21A. The door opening angle corrected by the correction unit 21B is stored in the memory unit 24. Here, the door opening angle detected by the angle sensor 32 may contain an error due to a measurement error dependent on the angle sensor 32 (e.g., a sensor mounting error or a sampling error) and a measurement error dependent on the driver's door (e.g., an error due to door deflection). Therefore, in the first embodiment, it is assumed that the door opening angle detected by the angle sensor 32 contains an error, and the error is corrected by the correction unit 21B.

[0042] The identification unit 21C identifies the three-dimensional position of the obstacle relative to the vehicle 20 using corresponding points of the obstacle present around the driver's door, which are determined based on the multiple images captured by the camera 41 from multiple viewpoints with different door opening angles and acquired by the acquisition unit 21A, and the corrected door opening angles corrected by the correction unit 21B associated with each of the multiple images. The corresponding points of the obstacle are determined by performing a process of extracting known image feature points on the multiple images captured by the camera 41 from multiple viewpoints with different door opening angles. The identification unit 21C also identifies the three-dimensional position of the obstacle using a multi-view stereo (MVS) method, which is a technology for reconstructing the three-dimensional shape of an object using multiple images captured from different viewpoints.

[0043] The determination unit 21D determines the maximum door opening angle at which the driver's door does not come into contact with the obstacle (hereinafter referred to as the "maximum opening angle"), using the three-dimensional position of the obstacle identified by the identification unit 21C and door information related to the shape and dimensions of the driver's door. The door information is stored in advance in the storage unit 24.

[0044] The control unit 21E determines the door opening angle when the camera 41 captures an image, and performs control to open the driver's door to the determined door opening angle. In the first embodiment, as an example, the acquisition unit 21A determines the door opening angle for the first time when the camera 41 captures an image to be "0 degrees" and the door opening angle for the second time to be "7 degrees."

[0045] Furthermore, the control unit 21E performs control to open the driver's door to the maximum opening angle determined by the determination unit 21D.

[0046] 3 is a first flowchart showing the flow of an opening process for determining a maximum opening angle and opening the driver's door to the determined maximum opening angle. The opening process is performed by the CPU 21 reading an information processing program from the storage unit 24, loading it into the RAM 23, and executing it. Note that the opening process is started, for example, when the opening switch is operated while the vehicle 20 is stopped or parked.

[0047] 3, CPU 21 acquires an image captured by camera 41 and the door opening angle at which the image was captured. As an example, CPU 21 acquires an image captured by camera 41 at a door opening angle of "0 degrees" and an image captured by camera 41 at a door opening angle of "7 degrees." Then, the process proceeds to step S11.

[0048] In step S11, the CPU 21 identifies the three-dimensional position of the obstacle relative to the vehicle 20 using the corresponding points of the obstacle determined based on the multiple images captured by the camera 41 from multiple viewpoints with different door opening angles acquired in step S10, and the corrected door opening angles associated with each of the multiple images. Here, the CPU 21 corrects an error in the door opening angle acquired in step S10. Then, the process proceeds to step S12. A method for identifying the three-dimensional position of the obstacle, including a method for correcting an error in the door opening angle, will be described later.

[0049] In step S12, the CPU 21 determines the maximum opening angle using the three-dimensional position of the obstacle identified in step S11 and the door information of the driver's door. Then, the process proceeds to step S13. The method for determining the maximum opening angle will be described later.

[0050] In step S13, the CPU 21 opens the driver's door to the maximum opening angle determined in step S12, and then ends the opening process.

[0051] Next, a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle will be described with reference to FIGS.

[0052] Fig. 4 is a first explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle. Fig. 4 shows a reference coordinate system and also shows a state in which the door mirror 33 of the driver's door is viewed from above. In the first embodiment, the reference coordinate system is set so that the hinge 34, which is the rotation axis of the driver's door, coincides with the Y axis, and X indicates the direction toward the rear of the vehicle, Y indicates the direction toward the bottom of the vehicle, and Z indicates the direction toward the right of the vehicle. The origin is the point where the Y axis intersects with the ground.

[0053] As shown in FIG. 4, the initial position of the camera 41 in the reference coordinate system when the door is closed is T c0 _w=(X c0 _w,Y c0 _w,Z c0 _w) T , the radius of rotation of the camera 41 with respect to the hinge 34 is L c , the door opening angle is α, and the position of the camera 41 in the reference coordinate system when the door opening angle is α is T cα _w=(X cα _w,Y cα _w,Z cα _w) T T represents a transposition. As shown in FIG. 4, the camera 41 at the initial position can be considered to have rotated by α0 around the hinge 34. At this time, the CPU 21 calculates X c0 _w and Z c0 _w to L c and α0 are calculated.

[0054]

number

[0055]

number

[0056] Furthermore, the CPU 21 calculates (X cα _w,Y cα _w,Z cα _w) is calculated.

[0057]

number

[0058] As shown in FIG. 4, the orientation of the camera 41 when the door is closed and when the door is opened at an angle α is expressed by a rotation matrix in the reference coordinate system, and is denoted by R c0 _w and R cα Let _w be the rotation matrix representing the rotation around the Y axis by an angle θ. Y _w(θ), R c0 _w and R cα The relationship between _w is shown in the following equation (4).

[0059]

number

[0060] From the above, the position and orientation of the camera 41 when the door is opened at an angle α can be determined using the position and orientation of the camera 41 when the door is closed and the door opened at an angle α.

[0061] Next, the relationship between the camera coordinate system and the reference coordinate system, and the coordinates of the obstacle to be measured are shown in FIG. FIG. 5 is a second explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle.

[0062] As shown in Figure 5, the coordinates of the obstacle in the reference coordinate system corresponding to the corresponding point of the obstacle are P_w=(X_w, Y_w, Z_w) T The coordinates of the obstacle in the camera coordinate system when the door is closed are P_c0=(X_c0, Y_c0, Z_c0) T The coordinates of the obstacle in the camera coordinate system at the door opening angle α are P_cα=(X_cα,Y_cα,Z_cα) T In addition, the image coordinates of the obstacle when the door is closed and when the door is open at an angle α are I_i0=(x_i0, y_i0). T , and I_iα=(x_iα,y_iα) T Let's say.

[0063] Here, P_w, P_c0, and P_cα represent the same object, that is, an obstacle, in different coordinate systems, and have the relationship shown in the following equations (5) and (6).

[0064]

number

[0065]

number

[0066] The internal parameters of the camera 41 are the focal length in pixels and the image center, respectively. c _i=(x c _i,y c _i) T In this case, the projection formula of the image is expressed by the following formulas (7) and (8).

[0067]

number

[0068]

number

[0069] From the above equations (7) and (8), the following equations (9), (10), (11) and (12) are obtained.

[0070]

number

[0071]

number

[0072]

number

[0073]

number

[0074] By substituting the formulas (9) and (10) into the formula (5) above, the following formula (13) is obtained.

[0075]

number

[0076] By substituting the formulas (11) and (12) into the formula (6) above, the following formula (14) is obtained.

[0077]

number

[0078] Here, in order to simplify the following equations, the constant terms in the above equations (13) and (14) are replaced by the following equations (15) and (16).

[0079]

number

[0080]

number

[0081] Here, if the error in the door opening angle detected by the angle sensor 32 (hereinafter also referred to as "door angle error") is ε and it is assumed that ε is small, the rotation matrix is ​​expressed by the following equation (17).

[0082]

number

[0083] Door angle error rotation matrix R ε By substituting the above, the following equations (18) and (19) are obtained.

[0084]

number

[0085]

number

[0086] By modifying the above formula (19), the following formula (20) is obtained.

[0087]

number

[0088] By setting Z=εZ_cα and solving the above equation (20), we obtain the following equation (21).

[0089]

number

[0090] From Z=εZ_cα, the following equation (22) is obtained.

[0091]

number

[0092] Here, when the value of ε is calculated at a plurality of corresponding points, the value may vary. However, since ε is common to the same image pair, for example, the CPU 21 may calculate the average value of ε calculated at a plurality of corresponding points as - Let ε be - Using ε, R -ε and A is calculated using the following formula (23): α Correcting B α Let's say.

[0093]

number

[0094] The CPU 21 calculates Z_c0 and Z_cα using the following equation (24).

[0095]

number

[0096] Here, X_c0 and Y_c0 are calculated by substituting the value of Z_c0 into the above formulas (9) and (10), and P_c0=(X_c0, Y_c0, Z_c0) is added to the above formula (5). T Similarly, P_w is calculated by substituting the value of Z_cα into the above equations (11) and (12), and then substituting it into the above equation (6).

[0097] At this time, since P_w calculated from the above formula (5) and P_w calculated from the above formula (6) usually do not match, in the first embodiment, the CPU 21 takes the average of the two P_w to determine a final solution, that is, the three-dimensional position of the obstacle relative to the vehicle 20. However, without being limited to this, the CPU 21 may adopt a predetermined one of the two P_w values ​​or a weighted average of the two P_w values ​​as the final solution.

[0098] Then, the CPU 21 performs the above-described identification of the three-dimensional position of the obstacle at a plurality of locations on the obstacle, and calculates the three-dimensional positions of N locations on the obstacle.

[0099] Here, the shape of the driver's door is known, and the height Y h Radius L of the driver's door at h The CPU 21 calculates the three-dimensional points P_w of all the calculated obstacles. n =(X_w n ,Y_w n ,Z_w n) T , (n=0~N-1), perform the following calculation.

[0100] First, the CPU 21 calculates the height Y_w of the obstacle. n The height Y is equal to h Radius L of the driver's door at h Then, the CPU 21 calculates the radius L of the driver's door. h If the relationship shown in the following formula (25) is satisfied, there is a possibility of contact with an obstacle. Therefore, the angle θ n Calculate.

[0101]

number

[0102]

number

[0103] On the other hand, CPU21 is the radius of the driver's door L h If the relationship shown in the following formula (27) is satisfied, the angle θ n Then, the CPU 21 does not calculate all the calculated angles θ n The smallest one among these is determined as the maximum opening angle.

[0104]

number

[0105] As described above, in the first embodiment, the CPU 21 acquires an image captured by the camera 41 and the door opening angle at which the image was captured. The CPU 21 also corrects an error in the acquired door opening angle. The CPU 21 then identifies the three-dimensional position of the obstacle relative to the vehicle 20 using corresponding points of the obstacle determined based on multiple images captured by the camera 41 from multiple viewpoints with different acquired door opening angles and the corrected door opening angles associated with each of the multiple images. As described above, the door opening angle detected by the angle sensor 32 may contain errors due to measurement errors dependent on the angle sensor 32 (e.g., sensor mounting error, sampling error) and measurement errors dependent on the driver's door (e.g., error due to door deflection). Therefore, in the first embodiment, the door opening angle detected by the angle sensor 32 is assumed to contain an error, and the error is corrected, thereby enabling the three-dimensional position of an obstacle present around the vehicle 20, specifically, around the driver's door, to be identified with high accuracy.

[0106] Here, when identifying the three-dimensional position of an obstacle using the Multi-View Stereo method, it is necessary to calculate the position and orientation of the camera in each image captured from multiple viewpoints. Conventionally, this calculation has been difficult, and therefore the accuracy of estimating the position and orientation of the camera has been insufficient. Furthermore, conventionally, there has been a problem in that a large number of images are required because the accuracy of estimating the position and orientation of the camera has not been sufficient. Note that, for example, literature on the Multi-View Stereo method includes "A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, CVPR2006."

[0107] In contrast to this, in the first embodiment, the movement of the camera 41 is restricted by the hinge 34 being a common rotation axis, and therefore the position and orientation of the camera in each of the images captured from multiple viewpoints can be estimated with high accuracy by using information indicating the coordinates of the hinge 34. Therefore, according to the first embodiment, the three-dimensional position of an obstacle can be identified with fewer images than when the three-dimensional position of an obstacle is identified by the conventional Multi-View Stereo method.

[0108] In addition, in the first embodiment, the three-dimensional position of an obstacle is identified using a door camera 41, which is a camera installed in a door mirror that is installed in many vehicles, and an angle sensor 32, so there is no need to add dedicated parts for this identification.

[0109] In the first embodiment, the CPU 21 determines the maximum opening angle using the identified three-dimensional position of the obstacle and the door information. As a result, according to the first embodiment, when the driver's door is opened in a situation where an obstacle is present around the vehicle 20, specifically around the driver's door, the driver's door can be opened to the maximum extent possible without coming into contact with the obstacle.

[0110] In the first embodiment, the CPU 21 controls the driver's door to open to the determined maximum opening angle. As a result, according to the first embodiment, in a situation where an obstacle is present around the driver's door, the driver's door can be automatically opened to the maximum extent possible without coming into contact with the obstacle, even if an occupant does not perform an opening operation of the driver's door.

[0111] In the first embodiment, the CPU 21 determines the door opening angle when the camera 41 captures an image, and controls the driver's door to open to the determined door opening angle. As a result, according to the first embodiment, in a situation where an obstacle is present around the driver's door, the driver's door can be automatically opened within a range that will not come into contact with the obstacle, even if the occupant does not perform an opening operation of the driver's door.

[0112] In the first embodiment, even while the CPU 21 is automatically opening the driver's door, the passenger can manually open and close the driver's door.

[0113] (Second embodiment) Next, a second embodiment will be described while omitting or simplifying parts that overlap with other embodiments.

[0114] FIG. 6 is a second block diagram showing an example of the functional configuration of the vehicle-mounted device 15. As shown in FIG. 6, the CPU 21 of the vehicle-mounted device 15 has, as functional components, an acquisition unit 21A, a correction unit 21B, an identification unit 21C, a determination unit 21D, a control unit 21E, and a reception unit 21F. Each functional component is realized by the CPU 21 reading and executing an information processing program stored in the storage unit 24.

[0115] After determination unit 21D determines the maximum opening angle, control unit 21E in the second embodiment performs control to open the driver's door to a predetermined angle within the range of the maximum opening angle at which no image is captured by camera 41. At this time, control unit 21E determines the predetermined angle according to the maximum opening angle determined by determination unit 21D. As an example, control unit 21E basically updates the predetermined angle in increments of 10 degrees, but if the maximum opening angle determined by determination unit 21D is greater than a specific angle (e.g., 70 degrees), it updates the predetermined angle in increments of 20 degrees.

[0116] The identification unit 21C in the second embodiment re-identifies the three-dimensional position of the obstacle relative to the vehicle 20 by using the corresponding points of the obstacle determined based on a plurality of images captured by the camera 41 from a viewpoint at a predetermined angle and a viewpoint at another door opening angle, and the corrected door opening angles corrected by the correction unit 21B associated with each of the plurality of images. The above-mentioned "another door opening angle" may be a door opening angle of "0 degrees" or a door opening angle other than "0 degrees", i.e., a door opening angle of "1 degree" or more.

[0117] The determination unit 21D in the second embodiment determines the maximum opening angle again using the three-dimensional position of the obstacle identified again by the identification unit 21C and the door information.

[0118] Receiving unit 21F receives an input of the number of times the maximum opening angle is to be determined by determining unit 21D (hereinafter referred to as the "determined number of times"). As an example, receiving unit 21F receives a value designated by the occupant operating monitor 43 as the determined number of times.

[0119] FIG. 7 is a second flowchart showing the flow of the release process. 7, the CPU 21 receives an input of the number of determinations, and then proceeds to step S21. As an example, it is assumed that the number of determinations received as input by the CPU 21 is two.

[0120] In step S21, CPU 21 acquires an image captured by camera 41 and the door opening angle at which the image was captured. As an example, in the first execution of step S21, CPU 21 acquires an image captured by camera 41 at a door opening angle of "0 degrees" and an image captured by camera 41 at a door opening angle of "7 degrees." In addition, in the second execution of step S21, CPU 21 acquires an image captured by camera 41 at a door opening angle of "17 degrees," which is the above-mentioned predetermined angle. Then, the process proceeds to step S22.

[0121] In step S22, the CPU 21 identifies the three-dimensional position of the obstacle relative to the vehicle 20 using the corresponding points of the obstacle determined based on the multiple images captured by the camera 41 from multiple viewpoints with different door opening angles acquired in step S21 and the corrected door opening angles associated with each of the multiple images. Here, the CPU 21 corrects an error in the door opening angle acquired in step S21. As an example, in the first iteration of step S22, the CPU 21 determines the corresponding points of the obstacle based on images captured by the camera 41 from a viewpoint where the door opening angle is "0 degrees" and a viewpoint where the door opening angle is "7 degrees". In addition, in the second iteration of step S22, the CPU 21 determines the corresponding points of the obstacle based on images captured by the camera 41 from a viewpoint where the door opening angle is "17 degrees" and a viewpoint where the door opening angle is "0 degrees". Then, the process proceeds to step S23.

[0122] In step S23, the CPU 21 determines the maximum opening angle using the three-dimensional position of the obstacle identified in step S22 and the door information of the driver's door, and then proceeds to step S24.

[0123] In step S24, CPU 21 determines whether the number of times the maximum opening angle was determined in step S23 has reached the number of times the input was accepted in step S20, and if it is determined that the number of times has been reached (step S24: YES), the process proceeds to step S25. On the other hand, if CPU 21 does not determine that the number of times has been reached (step S24: NO), the process returns to step S21.

[0124] In step S25, the CPU 21 opens the driver's door to the maximum opening angle determined in the immediately preceding step S23, and then ends the opening process.

[0125] As described above, in the second embodiment, the CPU 21 determines the maximum opening angle once, and then controls the driver's door to open to a predetermined angle within the range of the maximum opening angle at which no image is captured by the camera 41. The CPU 21 also re-specifies the three-dimensional position of the obstacle relative to the vehicle 20 using corresponding points of the obstacle determined based on multiple images captured by the camera 41 from a viewpoint at a predetermined angle and viewpoints at other door opening angles, and the corrected door opening angles associated with each of the multiple images. The CPU 21 then re-determines the maximum opening angle using the re-specified three-dimensional position of the obstacle and the door information. Thus, according to the second embodiment, by re-determining the maximum opening angle, the accuracy of the determined maximum opening angle can be improved compared to a configuration in which the maximum opening angle is determined only once.

[0126] Furthermore, in the second embodiment, the CPU 21 determines the predetermined angle according to the determined maximum opening angle. Here, when the distance to the obstacle is far (a predetermined distance or more), the three-dimensional position of the obstacle can be identified with higher accuracy by using an image with a larger door opening angle rather than an image with a smaller door opening angle. Therefore, according to the second embodiment, as an example, when the determined maximum opening angle is larger than a specific angle, the three-dimensional position of the obstacle can be identified with high accuracy by determining the predetermined angle to be larger than usual.

[0127] In the second embodiment, the CPU 21 receives an input of the number of determinations. Therefore, according to the second embodiment, for example, when the occupant has ample time, the determination of the maximum opening angle can be repeated many times to determine the maximum opening angle at which the door can be opened close to an obstacle. Also, according to the second embodiment, when the occupant does not have ample time, the determination of the maximum opening angle can be completed after a small number of repetitions, allowing the driver's door to be opened quickly.

[0128] (Third embodiment) Next, a third embodiment will be described while omitting or simplifying parts that overlap with other embodiments.

[0129] FIG. 8 is a second block diagram showing the hardware configuration of vehicle 20. As shown in FIG. As shown in FIG. 8, the vehicle 20 in the third embodiment includes an on-board device 15, a door ECU 30, an actuator 31, an angle sensor 32, a microphone 40, a camera 41, an input switch 42, a monitor 43, a speaker 44, a GPS device 45, and a sonar sensor 46.

[0130] The sonar sensor 46 is provided at least on the driver's door and is a device that uses ultrasonic waves to detect the distance to an obstacle approaching the side of the vehicle. The sonar sensor 46 is an example of a "distance sensor."

[0131] An example of the functional configuration of the vehicle-mounted device 15 in the third embodiment is similar to the example of the functional configuration of the vehicle-mounted device 15 in the second embodiment shown in FIG.

[0132] Correction unit 21B in the third embodiment corrects an image captured by camera 41 using internal parameters of camera 41. As an example, correction unit 21B corrects the image by performing distortion correction. At this time, correction unit 21B uses parameters for correcting optical distortion for each camera model, focal length, etc. as internal parameters of camera 41. These internal parameters are stored in storage unit 24 in advance.

[0133] As an example, distortion correction by the correction unit 21B is performed using the following method: Scaramuzza, D., A. Martinelli, and R. Siegwart, "A Toolbox for Easy Calibrating Omnidirectional Cameras," Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), October 7-15, 2006.

[0134] The control unit 21E in the third embodiment performs control to prohibit the opening of the driver's door based on the detection result of the sonar sensor 46 provided on the driver's door. Specifically, when the sonar sensor 46 detects an obstacle that is close to or approaching the driver's door, the control unit 21E performs control to prohibit the opening of the driver's door.

[0135] FIG. 9 is a third flowchart showing the flow of the release process. 9, the CPU 21 accepts input of the number of determinations, and then proceeds to step S31.

[0136] In step S31, the CPU 21 acquires an image captured by the camera 41 and the door opening angle at the time the image was captured, and then proceeds to step S32.

[0137] In step S32, CPU 21 corrects the image acquired in step S31 using the internal parameters of camera 41. Then, the process proceeds to step S33.

[0138] In step S33, the CPU 21 identifies the three-dimensional position of the obstacle relative to the vehicle 20 by using the corresponding points of the obstacle determined based on the multiple images corrected in step S32 and the corrected door opening angles associated with each of the multiple images. Here, the CPU 21 corrects the error in the door opening angle acquired in step S31. Then, the process proceeds to step S34.

[0139] In step S34, the CPU 21 determines the maximum opening angle using the three-dimensional position of the obstacle identified in step S33 and the door information of the driver's door, and then proceeds to step S35.

[0140] In step S35, CPU 21 determines whether the number of times the maximum opening angle was determined in step S34 has reached the number of times the input was accepted in step S30, and if it is determined that the number of times has been reached (step S35: YES), the process proceeds to step S36. On the other hand, if CPU 21 does not determine that the number of times has been reached (step S35: NO), the process returns to step S31.

[0141] In step S36, the CPU 21 opens the driver's door to the maximum opening angle determined in the immediately preceding step S34, and then ends the opening process.

[0142] As described above, in the third embodiment, the CPU 21 performs control to prohibit the opening of the driver's door based on the detection result of the sonar sensor 46 provided on the driver's door. As a result, according to the third embodiment, for example, the opening of the driver's door can be prohibited when the sonar sensor 46 detects an obstacle that is close to or approaching the driver's door.

[0143] Furthermore, in the third embodiment, CPU 21 corrects the image captured by camera 41 using the internal parameters of camera 41. As a result, according to the third embodiment, the three-dimensional position of the obstacle is identified using the corrected image, and therefore the three-dimensional position of the obstacle can be identified with higher accuracy than in a configuration in which image correction is not performed.

[0144] (others) In the above embodiment, the driver's door of the vehicle 20 is an example of a "hinge door." However, instead of or in addition to this, at least one of the passenger door and the rear door may be an example of a "hinge door." When at least one of the passenger door and the rear door is an example of a "hinge door," the vehicle 20 is equipped with actuators that automatically drive these doors to open and close, angle sensors that detect the door opening angles of these doors, and cameras that are provided on these doors and that capture images of the sides of the vehicle. Furthermore, when at least one of the passenger door and the rear door is an example of a "hinge door," a sonar sensor may be provided on these doors.

[0145] In the above embodiment, an example has been described in which the opening process is initiated when the occupant is inside the vehicle 20, but this is not limiting, and the opening process may be initiated when the occupant is outside the vehicle 20. As an example, the opening process may be initiated when an electronic key corresponding to the vehicle 20 is detected when the occupant is outside the vehicle 20.

[0146] In the above embodiment, the camera 41 is provided on the door mirror 33 of the driver's door of the vehicle 20, but this is not limiting, and the camera 41 may be provided on the driver's door itself.

[0147] In the above embodiment, the obstacle present around the driver's door may be any object that appears in the image captured by camera 41, and may be present in a position that will come into contact with the driver's door when the driver's door is opened, or may be present in a position that will not come into contact with the driver's door.

[0148] In the above embodiment, the in-vehicle device 15 is an example of the "information processing device," but the present invention is not limited to this, and an external device such as a server connectable to the vehicle 20 may also be an example of the "information processing device." In this case, for example, the external device may be configured to have the functions of the acquisition unit 21A, the correction unit 21B, the identification unit 21C, and the determination unit 21D described in the above embodiment, and the vehicle 20 may be configured to have the functions of the control unit 21E and the reception unit 21F.

[0149] In the above embodiment, the release process executed by the CPU 21 after reading the software (program) may be executed by various processors other than a CPU. Examples of such processors include programmable logic devices (PLDs) whose circuit configuration can be changed after fabrication, such as field-programmable gate arrays (FPGAs), and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. The release process may be executed by one of these processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0150] In the above embodiment, the information processing program is pre-stored (installed) in the storage unit 24, but the present invention is not limited to this. The information processing program may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program may also be downloaded from an external device via a network.

[0151] The present disclosure may employ the following aspects. (1) an acquisition unit that acquires an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle that indicates an angle at which the hinged door is opened from a closed state when the image is captured; a correction unit that corrects an error in the door opening angle acquired by the acquisition unit; an identification unit that identifies a three-dimensional position of the obstacle relative to the vehicle using a corresponding point of the obstacle present around the hinged door, which is determined based on the plurality of images captured by the imaging unit from a plurality of viewpoints with different door opening angles and acquired by the acquisition unit, and the door opening angle corrected by the correction unit that is associated with each of the plurality of images; An information processing device comprising:

[0152] (2) a determination unit that determines the maximum door opening angle at which the hinge door does not come into contact with the obstacle, using the three-dimensional position of the obstacle identified by the identification unit and door information related to the shape and dimensions of the hinge door; The information processing device described in (1).

[0153] (3) a control unit that controls the hinge door to open to the maximum door opening angle at which the hinge door does not come into contact with the obstacle, the maximum door opening angle being determined by the determination unit; (2) An information processing device according to the present invention.

[0154] (4) The control unit determining the door opening angle when the imaging unit captures the image; performing control to open the hinge door to the determined door opening angle; (3) An information processing device according to the present invention.

[0155] (5) The control unit After the determination unit determines the maximum door opening angle at which the door will not come into contact with the obstacle, the determination unit controls the hinge door to open to a predetermined angle at which the image is not captured by the imaging unit within the range of the maximum door opening angle at which the door will not come into contact with the obstacle, The identification unit re-identifying a three-dimensional position of the obstacle relative to the vehicle using a corresponding point of the obstacle determined based on a plurality of the images captured by the imaging unit from a viewpoint at the predetermined angle and a viewpoint at another door opening angle, and the door opening angle corrected by the correction unit associated with each of the plurality of the images; The determination unit using the three-dimensional position of the obstacle identified again by the identification unit and the door information, again determining the maximum door opening angle at which the hinge door does not come into contact with the obstacle; The information processing device according to (3) or (4).

[0156] (6) The control unit determining the predetermined angle according to the maximum door opening angle at which the door does not come into contact with the obstacle, as determined by the determination unit; (5) An information processing device according to the present invention.

[0157] (7) a receiving unit that receives an input of the number of times that the determination unit determines the maximum door opening angle at which the hinge door does not come into contact with the obstacle, An information processing device according to (5) or (6).

[0158] (8) The control unit a control for prohibiting the hinge door from opening based on a detection result of a distance measuring sensor provided on the hinge door; An information processing device according to any one of (3) to (7).

[0159] (9) The correction unit correcting the image captured by the imaging unit using internal parameters of the imaging unit; An information processing device according to any one of (1) to (8).

[0160] (10) Acquire an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating an angle by which the hinged door was opened from a closed state when the image was captured; Correcting an error in the acquired door opening angle; identifying a three-dimensional position of the obstacle relative to the vehicle using corresponding points of the obstacle present around the hinged door, which are determined based on the plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the corrected door opening angles associated with each of the plurality of images; An information processing method in which processing is performed by a computer.

[0161] (11) On the computer, Acquire an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating an angle by which the hinged door was opened from a closed state when the image was captured; Correcting an error in the acquired door opening angle; identifying a three-dimensional position of the obstacle relative to the vehicle using corresponding points of the obstacle present around the hinged door, which are determined based on the plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the corrected door opening angles associated with each of the plurality of images; An information processing program for executing processing. [Explanation of symbols]

[0162] 15. In-vehicle device (an example of an information processing device) 20 vehicles 21A Acquisition Department 21B Correction section 21C Specific part 41 Camera (an example of an imaging unit)

Claims

1. an acquisition unit that acquires an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle that indicates an angle at which the hinged door is opened from a closed state when the image is captured; a correction unit that corrects an error in the door opening angle acquired by the acquisition unit; an identification unit that identifies a three-dimensional position of the obstacle relative to the vehicle using a corresponding point of the obstacle present around the hinged door, which is determined based on the plurality of images captured by the imaging unit from a plurality of viewpoints with different door opening angles and acquired by the acquisition unit, and the door opening angle corrected by the correction unit that is associated with each of the plurality of images; An information processing device comprising:

2. a determination unit that determines the maximum door opening angle at which the hinge door does not come into contact with the obstacle, using the three-dimensional position of the obstacle identified by the identification unit and door information related to the shape and dimensions of the hinge door; The information processing device according to claim 1 .

3. a control unit that controls the hinge door to open to the maximum door opening angle at which the hinge door does not come into contact with the obstacle, the maximum door opening angle being determined by the determination unit; The information processing device according to claim 2 .

4. The control unit determining the door opening angle when the imaging unit captures the image; performing control to open the hinge door to the determined door opening angle; The information processing device according to claim 3 .

5. The control unit After the determination unit determines the maximum door opening angle at which the door will not come into contact with the obstacle, the determination unit controls the hinge door to open to a predetermined angle at which the image is not captured by the imaging unit within the range of the maximum door opening angle at which the door will not come into contact with the obstacle, The identification unit re-identifying a three-dimensional position of the obstacle relative to the vehicle using a corresponding point of the obstacle determined based on a plurality of images captured by the imaging unit from a viewpoint at the predetermined angle and a viewpoint at another door opening angle, and the door opening angle corrected by the correction unit associated with each of the plurality of images; The determination unit using the three-dimensional position of the obstacle identified again by the identification unit and the door information, again determining the maximum door opening angle at which the hinge door does not come into contact with the obstacle; The information processing device according to claim 3 .

6. The control unit determining the predetermined angle according to the maximum door opening angle at which the door does not come into contact with the obstacle, as determined by the determination unit; The information processing device according to claim 5 .

7. a receiving unit that receives an input of the number of times that the determination unit determines the maximum door opening angle at which the hinge door does not come into contact with the obstacle, The information processing device according to claim 5 .

8. The control unit a control for prohibiting the hinge door from opening based on a detection result of a distance measuring sensor provided on the hinge door; The information processing device according to claim 3 .

9. The correction unit correcting the image captured by the imaging unit using internal parameters of the imaging unit; The information processing device according to claim 1 .

10. Acquire an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating an angle by which the hinged door was opened from a closed state when the image was captured; Correcting an error in the acquired door opening angle; identifying a three-dimensional position of the obstacle relative to the vehicle using corresponding points of the obstacle present around the hinged door, which are determined based on the plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the corrected door opening angles associated with each of the plurality of images; An information processing method in which processing is performed by a computer.

11. On the computer, Acquire an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating an angle by which the hinged door was opened from a closed state when the image was captured; Correcting an error in the acquired door opening angle; identifying a three-dimensional position of the obstacle relative to the vehicle using corresponding points of the obstacle present around the hinged door, which are determined based on the plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the corrected door opening angles associated with each of the plurality of images; An information processing program for executing processing.

Images