Information processing device, information processing method, and information processing program

Abstract

To enable a hinge door to be opened to a maximum range that does not contact an obstacle in a case where the hinge door is automatically opened in such a situation that the obstacle exists in the periphery of a vehicle.SOLUTION: An information processing device comprises: an acquisition unit which acquires an image captured by an imaging unit provided in a hinge door of a vehicle and a door opening angle indicating an angle at which the hinge door is opened from the close state when the image is captured; a specification unit which specifies a three-dimensional position of an obstacle with respect to the vehicle by using a corresponding point of the obstacle existing in the periphery of the hinge door determined on the basis of a plurality of images captured by the imaging unit at a plurality of view points having the different door opening angles acquired by the acquisition unit and the door opening angle associated with each of the plurality of images; and a determination unit which determines the maximum door opening angle at which the hinge door does not contact the obstacle by using the three-dimensional position of the obstacle specified by the specification unit and door information about the shape and dimension of the hinge door.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Patent Document 1 discloses a technique in which a power door control unit opens a power hinge door when a vehicle user approaches the power hinge door. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-147856 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, there is a conventional technique for automatically opening a hinged door of a vehicle without human intervention. When the hinged door is automatically opened in a situation where there are obstacles such as walls and other vehicles around the vehicle, the door must be opened to the extent that it does not come into contact with the obstacles. Even in this case, it is desirable to open the door as far as possible without coming into contact with the obstacles in order to facilitate passenger entry and exit.

[0005] Therefore, the present invention aims to provide an information processing device, an information processing method, and an information processing program that, when automatically opening a hinged door in a situation where an obstacle is present around the vehicle, enable the hinged door to be opened to the maximum extent possible without coming into contact with the obstacle. [Means for solving the problem]

[0006] The information processing device of the first aspect includes: an acquisition unit that acquires an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle that indicates the angle by which the hinged door was opened from a closed state when the image was captured; an identification unit that identifies a three-dimensional position of an obstacle present around the hinged door relative to the vehicle using corresponding points of the obstacle present around the hinged door, determined based on a plurality of images captured by the imaging unit from a plurality of viewpoints with different door opening angles and the door opening angles associated with each of the plurality of images; and a determination unit that determines the maximum door opening angle at which the hinged door does not come into contact with the obstacle, using the three-dimensional position of the obstacle identified by the identification unit and door information related to the shape and dimensions of the hinged door. and a control unit that performs control to open the hinge door up to the maximum door opening angle determined by the determination unit at which the hinge door will not come into contact with the obstacle, wherein after the determination unit determines the maximum door opening angle at which the hinge door will not come into contact with the obstacle, the control unit performs control to open the hinge door up to a predetermined angle at which the image is not captured by the imaging unit within the range of the maximum door opening angle at which the door will not come into contact with the obstacle, and the identification unit re-identifies a three-dimensional position of the obstacle relative to the vehicle using a corresponding point of the obstacle determined based on a plurality of images captured by the imaging unit at a viewpoint of the predetermined angle and viewpoints of other door opening angles, and the door opening angle associated with each of the plurality of images, and the determination unit re-determines the maximum door opening angle at which the hinge door will not come into contact with the obstacle using the three-dimensional position of the obstacle re-identified by the identification unit and the door information. .

[0008] No. 2 The information processing device of the present embodiment is 1 In the information processing device of the aspect 1, the control unit determines the door opening angle when the imaging unit captures the image, and performs control to open the hinged door up to the determined door opening angle.

[0010] No. 3 The information processing device of the present embodiment is 1 In the information processing device of the aspect 1, the control unit determines the predetermined angle in accordance with the maximum door opening angle at which the door does not come into contact with the obstacle, which is determined by the determination unit.

[0011] No. 4 The information processing device of the present embodiment is 1 The information processing device of the aspect further includes a receiving unit that receives an input of the number of times the determination unit determines the maximum door opening angle at which the hinge door does not come into contact with the obstacle.

[0012] No. 5 The information processing device of the present embodiment is 1 From 4 In the information processing device of any one of the above aspects, the control unit performs control to prohibit the hinge door from opening based on a detection result of a distance measurement sensor provided on the hinge door.

[0013] No. 6 The information processing device of the aspect includes first to second 5 The information processing device according to any one of the above aspects further comprises a correction unit that corrects the image captured by the imaging unit using an internal parameter of the imaging unit.

[0014] No. 7 The information processing method of the aspect acquires an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating an angle by which the hinged door was opened from a closed state when the image was captured, identifies a three-dimensional position of the obstacle relative to the vehicle using corresponding points of an obstacle present around the hinged door, which are determined based on a plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the door opening angles associated with each of the plurality of images, and determines the maximum door opening angle at which the hinged door does not come into contact with the obstacle using the identified three-dimensional position of the obstacle and door information related to the shape and dimensions of the hinged door. and after determining the maximum door opening angle at which the hinge door will not come into contact with the obstacle, controlling the hinge door to open up to a predetermined angle at which the image is not captured by the imaging unit within the range of the maximum door opening angle at which the hinge door will not come into contact with the obstacle, re-identifying a three-dimensional position of the obstacle with respect to the vehicle using a corresponding point of the obstacle determined based on a plurality of images captured by the imaging unit at a viewpoint at the predetermined angle and viewpoints at other door opening angles, and the door opening angle associated with each of the plurality of images, and re-determining the maximum door opening angle at which the hinge door will not come into contact with the obstacle using the re-identified three-dimensional position of the obstacle and the door information, and controlling the hinge door to open up to the determined maximum door opening angle at which the hinge door will not come into contact with the obstacle. The processing is performed by a computer.

[0015] No. 8 The information processing program of the aspect causes a computer to acquire an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating an angle by which the hinged door was opened from a closed state when the image was captured, identify a three-dimensional position of the obstacle relative to the vehicle using corresponding points of an obstacle present around the hinged door, which are determined based on a plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the door opening angle associated with each of the plurality of images, and determine the maximum door opening angle at which the hinged door does not come into contact with the obstacle, using the identified three-dimensional position of the obstacle and door information related to the shape and dimensions of the hinged door. and after determining the maximum door opening angle at which the hinge door will not come into contact with the obstacle, controlling the hinge door to open up to a predetermined angle at which the image is not captured by the imaging unit within the range of the maximum door opening angle at which the hinge door will not come into contact with the obstacle, re-identifying a three-dimensional position of the obstacle with respect to the vehicle using a corresponding point of the obstacle determined based on a plurality of images captured by the imaging unit at a viewpoint at the predetermined angle and viewpoints at other door opening angles, and the door opening angle associated with each of the plurality of images, and re-determining the maximum door opening angle at which the hinge door will not come into contact with the obstacle using the re-identified three-dimensional position of the obstacle and the door information, and controlling the hinge door to open up to the determined maximum door opening angle at which the hinge door will not come into contact with the obstacle. , and execute the processing. [Effects of the Invention]

[0016] The information processing device, information processing method, and information processing program of the present invention enable the hinged door to be opened to the maximum extent possible without coming into contact with the obstacle when the hinged door is automatically opened in a situation where an obstacle is present around the vehicle. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a first block diagram showing a hardware configuration of a vehicle. [Figure 2] FIG. 2 is a first block diagram showing an example of the functional configuration of the vehicle-mounted device. [Figure 3] 10 is a first flowchart showing the flow of a release process. [Figure 4] FIG. 1 is a first explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle. [Figure 5] FIG. 10 is a second explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle. [Figure 6] FIG. 2 is a second block diagram showing an example of the functional configuration of the vehicle-mounted device. [Figure 7] 10 is a second flowchart showing the flow of the release process. [Figure 8] FIG. 2 is a second block diagram showing the hardware configuration of the vehicle. [Figure 9] FIG. 10 is a third block diagram showing an example of the functional configuration of the vehicle-mounted device. [Figure 10] 10 is a third flowchart showing the flow of the release process. DETAILED DESCRIPTION OF THE INVENTION

[0018] The vehicle 20 according to this embodiment will be described below. (First embodiment) FIG. 1 is a first block diagram showing the hardware configuration of a vehicle 20. The vehicle 20 may be any of a gasoline vehicle, a hybrid vehicle, and an electric vehicle, but in the first embodiment, as an example, the vehicle 20 is a gasoline vehicle. The vehicle 20 also has a driver's door on the driver's side, a passenger door on the passenger's side, and a rear door at the rear of the vehicle 20. In the first embodiment, the driver's door of the vehicle 20 is configured as a hinged door whose rotation axis in the vehicle body coordinate system is known. The driver's door is an example of a "hinged door."

[0019] 1, the vehicle 20 includes an on-board device 15, a door ECU (Electronic Control Unit) 30, an actuator 31, an angle sensor 32, a microphone 40, a camera 41, an input switch 42, a monitor 43, a speaker 44, and a GPS device 45. The on-board device 15 is an example of an "information processing device."

[0020] The vehicle-mounted device 15 includes a CPU 21 (Central Processing Unit), a ROM 22 (Read Only Memory), a RAM 23 (Random Access Memory), a storage unit 24, an in-vehicle communication I / F (Interface) 25, an input / output I / F 26, and a wireless communication I / F 27. The CPU 21, the ROM 22, the RAM 23, the storage unit 24, the in-vehicle communication I / F 25, the input / output I / F 26, and the wireless communication I / F 27 are connected to each other via an internal bus 28 so as to be able to communicate with each other.

[0021] The CPU 21 is a central processing unit that executes various programs and controls each part. That is, the CPU 21 reads programs from the ROM 22 or the storage unit 24 and executes the programs using the RAM 23 as a work area. The CPU 21 controls each of the above components and performs various arithmetic processing in accordance with the programs recorded in the ROM 22 or the storage unit 24.

[0022] The ROM 22 stores various programs and various data. The RAM 23 serves as a working area for temporarily storing programs or data.

[0023] The storage unit 24 is configured with a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and stores various programs and various data. The storage unit 24 stores at least an information processing program for executing the release process described below.

[0024] The in-vehicle communication I / F 25 is an interface for connecting to the door ECU 30. The interface uses a communication standard based on the CAN protocol. The in-vehicle communication I / F 25 is connected to an external bus 29.

[0025] In the first embodiment, the ECU provided is a door ECU 30. Although not shown in the figure, a plurality of ECUs are provided for each function of the vehicle 20, and the vehicle 20 is configured to include ECUs other than the door ECU 30. The door ECU 30 is connected to an actuator 31 and an angle sensor 32 .

[0026] The actuator 31 automatically drives and opens / closes at least the driver's door among the doors of the vehicle 20. In the first embodiment, the door ECU 30 drives the actuator 31 based on the control of the in-vehicle device 15, so that the driver's door can be automatically opened / closed without an occupant performing an opening / closing operation of the driver's door.

[0027] The angle sensor 32 is provided on at least the driver's door among the doors of the vehicle 20, and is a sensor for detecting the door opening angle when the driver's door is in a closed state, i.e., the angle by which the door is opened from the time the door is closed. The door opening angle detected by the angle sensor 32 is stored in the memory unit 24.

[0028] The input / output I / F 26 is an interface for communicating with a microphone 40 , a camera 41 , an input switch 42 , a monitor 43 , a speaker 44 , and a GPS device 45 that are mounted on the vehicle 20 .

[0029] The microphone 40 is a device that is provided on the front pillar or dashboard of the vehicle 20 and collects sounds made by the user of the vehicle 20 .

[0030] The camera 41 includes, for example, a solid-state imaging element such as a charge coupled device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) image sensor. For example, the camera 41 is provided at least on the door mirror 33 of the driver's door of the vehicle 20 (see FIGS. 4 and 5), and captures images of the sides of the vehicle. The images captured by the camera 41 are associated with the door opening angle at the time when each image was captured and stored in the memory unit 24. The camera 41 may be connected to the in-vehicle device 15 via an ECU (for example, a camera ECU). The camera 41 is an example of an "imaging unit."

[0031] Furthermore, the orientation of the camera 41 in the vehicle body coordinate system when the driver's door is closed is known, and information relating to this orientation is stored in the storage unit 24.

[0032] The input switch 42 is provided on an instrument panel, a center console, a steering wheel, or the like, and is a switch that inputs an operation by the driver's finger. For example, a push-button numeric keypad, a touchpad, or the like can be used as the input switch 42. In the first embodiment, at least an opening switch for opening the driver's door is provided as the input switch 42. In the first embodiment, the driver's door can be automatically opened by operating the opening switch while the vehicle 20 is stopped or parked.

[0033] The monitor 43 is a liquid crystal monitor provided on an instrument panel, a meter panel, or the like, for displaying images related to operation suggestions and explanations of functions of the vehicle 20. The monitor 43 may be provided as a touch panel that also serves as the input switch 42.

[0034] The speaker 44 is provided on the instrument panel, the center console, the front pillar, the dashboard, or the like, and is a device for outputting audio suggesting the operation of functions of the vehicle 20 and explaining the functions. The speaker 44 may also be provided on the monitor 43.

[0035] The GPS device 45 is a device that measures the current position of the vehicle 20. The GPS device 45 includes an antenna (not shown) that receives signals from GPS satellites. The GPS device 45 may be connected to the vehicle-mounted device 15 via a car navigation system that is connected to an ECU (for example, a multimedia ECU).

[0036] The wireless communication I / F 27 is a wireless communication module for communicating with other devices, and uses communication standards such as 5G, LTE, and Wi-Fi (registered trademark).

[0037] Next, the functional configuration of the vehicle-mounted device 15 will be described. FIG. 2 is a first block diagram showing an example of the functional configuration of the vehicle-mounted device 15. As shown in FIG.

[0038] 2, the CPU 21 of the vehicle-mounted device 15 has, as functional components, an acquisition unit 21A, an identification unit 21B, a determination unit 21C, and a control unit 21D. Each functional component is realized by the CPU 21 reading and executing an information processing program stored in the storage unit 24.

[0039] The acquisition unit 21A acquires an image captured by the camera 41 and the door opening angle at which the image was captured. In the first embodiment, the acquisition unit 21A acquires a plurality of images captured by the camera 41 from a plurality of viewpoints with different door opening angles, and the door opening angle associated with each of the plurality of images.

[0040] The identification unit 21B identifies the three-dimensional position of the obstacle relative to the vehicle 20 using corresponding points of the obstacle present around the driver's door, which are determined based on the multiple images captured by the camera 41 from multiple viewpoints with different door opening angles and acquired by the acquisition unit 21A, and the door opening angles associated with each of the multiple images. The corresponding points of the obstacle are determined by performing a process of extracting known image feature points on the multiple images captured by the camera 41 from multiple viewpoints with different door opening angles. The identification unit 21B also identifies the three-dimensional position of the obstacle using a multi-view stereo (MVS) method, which is a technology for reconstructing the three-dimensional shape of an object using multiple images captured from different viewpoints.

[0041] The determination unit 21C determines the maximum door opening angle at which the driver's door does not come into contact with the obstacle (hereinafter referred to as the "maximum opening angle") by using the three-dimensional position of the obstacle identified by the identification unit 21B and door information related to the shape and dimensions of the driver's door. The door information is stored in advance in the storage unit 24.

[0042] The control unit 21D determines the door opening angle when the camera 41 captures an image, and performs control to open the driver's door to the determined door opening angle. In the first embodiment, as an example, the acquisition unit 21A determines the door opening angle for the first time when the camera 41 captures an image to be "0 degrees" and the door opening angle for the second time to be "7 degrees."

[0043] Furthermore, the control unit 21D performs control to open the driver's door to the maximum opening angle determined by the determination unit 21C.

[0044] 3 is a first flowchart showing the flow of an opening process for determining a maximum opening angle and opening the driver's door to the determined maximum opening angle. The opening process is performed by the CPU 21 reading an information processing program from the storage unit 24, loading it into the RAM 23, and executing it. Note that the opening process is started, for example, when the opening switch is operated while the vehicle 20 is stopped or parked.

[0045] 3, CPU 21 acquires an image captured by camera 41 and the door opening angle at which the image was captured. As an example, CPU 21 acquires an image captured by camera 41 at a door opening angle of "0 degrees" and an image captured by camera 41 at a door opening angle of "7 degrees." Then, the process proceeds to step S11.

[0046] In step S11, the CPU 21 identifies the three-dimensional position of the obstacle relative to the vehicle 20 using the corresponding points of the obstacle determined based on the multiple images captured by the camera 41 from multiple viewpoints with different door opening angles acquired in step S10 and the door opening angles associated with each of the multiple images. Then, the process proceeds to step S12. A method for identifying the three-dimensional position of the obstacle will be described later.

[0047] In step S12, the CPU 21 determines the maximum opening angle using the three-dimensional position of the obstacle identified in step S11 and the door information of the driver's door. Then, the process proceeds to step S13. The method for determining the maximum opening angle will be described later.

[0048] In step S13, the CPU 21 opens the driver's door to the maximum opening angle determined in step S12, and then ends the opening process.

[0049] Next, a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle will be described with reference to FIGS.

[0050] Fig. 4 is a first explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle. Fig. 4 shows a reference coordinate system and also shows a state in which the door mirror 33 of the driver's door is viewed from above. In the first embodiment, the reference coordinate system is set so that the hinge 34, which is the rotation axis of the driver's door, coincides with the Y axis, and X indicates the direction toward the rear of the vehicle, Y indicates the direction toward the bottom of the vehicle, and Z indicates the direction toward the right of the vehicle. The origin is the point where the Y axis intersects with the ground.

[0051] As shown in FIG. 4, the initial position of the camera 41 in the reference coordinate system when the door is closed is T c0 _w=(X c0 _w,Y c0 _w,Z c0 _w) T , the radius of rotation of the camera 41 with respect to the hinge 34 is L c , the door opening angle is α, and the position of the camera 41 in the reference coordinate system when the door opening angle is α is T cα _w=(X cα _w,Y cα _w,Z cα _w) T T represents a transposition. As shown in FIG. 4, the camera 41 at the initial position can be considered to have rotated by α0 around the hinge 34. At this time, the CPU 21 calculates X c0 _w and Z c0 _w to L c and α0 are calculated.

[0052]

number

[0053]

number

[0054] Furthermore, the CPU 21 calculates (X cα _w,Y cα _w,Z cα _w) is calculated.

[0055]

number

[0056] As shown in FIG. 4, the orientation of the camera 41 when the door is closed and when the door is opened at an angle α is expressed by a rotation matrix in the reference coordinate system, and is denoted by R c0 _w and R cα Let _w be the rotation matrix representing the rotation around the Y axis by an angle θ. Y_w(θ), R c0 _w and R cα The relationship between _w is shown in the following equation (4).

[0057]

number

[0058] From the above, the position and orientation of the camera 41 when the door is opened at an angle α can be determined using the position and orientation of the camera 41 when the door is closed and the door opened at an angle α.

[0059] Next, the relationship between the camera coordinate system and the reference coordinate system, and the coordinates of the obstacle to be measured are shown in FIG. FIG. 5 is a second explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle.

[0060] As shown in Figure 5, the coordinates of the obstacle in the reference coordinate system corresponding to the corresponding point of the obstacle are P_w=(X_w, Y_w, Z_w) T The coordinates of the obstacle in the camera coordinate system when the door is closed are P_c0=(X_c0, Y_c0, Z_c0) T The coordinates of the obstacle in the camera coordinate system at the door opening angle α are P_cα=(X_cα,Y_cα,Z_cα) T In addition, the image coordinates of the obstacle when the door is closed and when the door is open at an angle α are I_i0=(x_i0, y_i0). T , and I_iα=(x_iα,y_iα) T Let's say.

[0061] Here, P_w, P_c0, and P_cα represent the same object, that is, an obstacle, in different coordinate systems, and have the relationship shown in the following equations (5) and (6).

[0062]

number

[0063]

number

[0064] The internal parameters of the camera 41 are the focal length in pixels and the image center, respectively. c _i=(x c _i,y c _i) T In this case, the projection formula of the image is expressed by the following formulas (7) and (8).

[0065]

number

[0066]

number

[0067] From the above equations (7) and (8), the following equations (9), (10), (11) and (12) are obtained.

[0068]

number

[0069]

number

[0070]

number

[0071]

number

[0072] By substituting the formulas (9) and (10) into the formula (5) above, the following formula (13) is obtained.

[0073]

number

[0074] By substituting the formulas (11) and (12) into the formula (6) above, the following formula (14) is obtained.

[0075]

number

[0076] Here, in order to simplify the following equations, the constant terms in the above equations (13) and (14) are replaced by the following equations (15) and (16).

[0077]

number

[0078]

number

[0079] Since the left sides of the above formulas (13) and (14) are equal, the following formulas (17), (18), and (19) can be obtained by rearranging the formulas.

[0080]

number

[0081]

number

[0082]

number

[0083] Since the above formula (19) is three linear equations with two variables (Z_c0 and Z_cα), for example, the CPU 21 calculates a solution by the least squares method as in the following formula (20).

[0084]

number

[0085] Here, X_c0 and Y_c0 are calculated by substituting the value of Z_c0 into the above formulas (9) and (10), and P_c0=(X_c0, Y_c0, Z_c0) is added to the above formula (5). T Similarly, P_w is calculated by substituting the value of Z_cα into the above equations (11) and (12), and then substituting it into the above equation (6).

[0086] At this time, since P_w calculated from the above formula (5) and P_w calculated from the above formula (6) usually do not match, in the first embodiment, the CPU 21 takes the average of the two P_w to determine a final solution, that is, the three-dimensional position of the obstacle relative to the vehicle 20. However, without being limited to this, the CPU 21 may adopt a predetermined one of the two P_w values ​​or a weighted average of the two P_w values ​​as the final solution.

[0087] Then, the CPU 21 performs the above-described identification of the three-dimensional position of the obstacle at a plurality of locations on the obstacle, and calculates the three-dimensional positions of N locations on the obstacle.

[0088] Here, the shape of the driver's door is known, and the height Y h Radius L of the driver's door at h The CPU 21 calculates the three-dimensional points P_w of all the calculated obstacles. n =(X_w n ,Y_w n ,Z_w n ) T , (n=0~N-1), perform the following calculation.

[0089] First, the CPU 21 calculates the height Y_w of the obstacle. n The height Y is equal to h Radius L of the driver's door at h Then, the CPU 21 calculates the radius L of the driver's door. h If the relationship shown in the following formula (21) is satisfied, there is a possibility of contact with an obstacle. Therefore, the angle θ n Calculate.

[0090]

number

[0091]

number

[0092] On the other hand, CPU21 is the radius of the driver's door L h If the relationship shown in the following formula (23) is satisfied, the angle θ n Then, the CPU 21 does not calculate all the calculated angles θ n The smallest one among these is determined as the maximum opening angle.

[0093]

number

[0094] As described above, in the first embodiment, the CPU 21 acquires an image captured by the camera 41 and the door opening angle at which the image was captured. The CPU 21 also identifies the three-dimensional position of the obstacle relative to the vehicle 20 using corresponding points of the obstacle determined based on multiple images captured by the camera 41 from multiple viewpoints with different acquired door opening angles and the door opening angles associated with each of the multiple images. The CPU 21 then determines the maximum opening angle using the identified three-dimensional position of the obstacle and door information about the driver's door. As a result, according to the first embodiment, when the driver's door is automatically opened in a situation where an obstacle is present around the vehicle 20, specifically around the driver's door, the driver's door can be opened to the maximum extent possible without coming into contact with the obstacle.

[0095] Here, when identifying the three-dimensional position of an obstacle using the Multi-View Stereo method, it is necessary to calculate the position and orientation of the camera in each image captured from multiple viewpoints. Conventionally, this calculation has been difficult, and therefore the accuracy of estimating the position and orientation of the camera has been insufficient. Furthermore, conventionally, there has been a problem in that a large number of images are required because the accuracy of estimating the position and orientation of the camera has not been sufficient. Note that, for example, literature on the Multi-View Stereo method includes "A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, CVPR2006."

[0096] In contrast to this, in the first embodiment, the movement of the camera 41 is restricted by the hinge 34 being a common rotation axis, and therefore the position and orientation of the camera in each of the images captured from multiple viewpoints can be estimated with high accuracy by using information indicating the coordinates of the hinge 34. Therefore, according to the first embodiment, the three-dimensional position of an obstacle can be identified with fewer images than when the three-dimensional position of an obstacle is identified by the conventional Multi-View Stereo method.

[0097] In addition, in the first embodiment, the three-dimensional position of an obstacle is identified using a door camera 41, which is a camera installed in a door mirror that is installed in many vehicles, and an angle sensor 32, so there is no need to add dedicated parts for this identification.

[0098] In the first embodiment, the CPU 21 controls the driver's door to open to the determined maximum opening angle. As a result, according to the first embodiment, in a situation where an obstacle is present around the driver's door, the driver's door can be automatically opened to the maximum extent possible without coming into contact with the obstacle, even if an occupant does not perform an opening operation of the driver's door.

[0099] In the first embodiment, the CPU 21 determines the door opening angle when the camera 41 captures an image, and controls the driver's door to open to the determined door opening angle. As a result, according to the first embodiment, in a situation where an obstacle is present around the driver's door, the driver's door can be automatically opened within a range that will not come into contact with the obstacle, even if the occupant does not perform an opening operation of the driver's door.

[0100] In the first embodiment, even while the CPU 21 is automatically opening the driver's door, the passenger can manually open and close the driver's door.

[0101] (Second embodiment) Next, a second embodiment will be described while omitting or simplifying parts that overlap with other embodiments.

[0102] FIG. 6 is a second block diagram showing an example of the functional configuration of the vehicle-mounted device 15. As shown in FIG. 6, the CPU 21 of the vehicle-mounted device 15 has, as functional components, an acquisition unit 21A, an identification unit 21B, a determination unit 21C, a control unit 21D, and a reception unit 21E. Each functional component is realized by the CPU 21 reading and executing an information processing program stored in the storage unit 24.

[0103] After determination unit 21C determines the maximum opening angle, control unit 21D in the second embodiment performs control to open the driver's door to a predetermined angle within the range of the maximum opening angle at which no image is captured by camera 41. At this time, control unit 21D determines the predetermined angle according to the maximum opening angle determined by determination unit 21C. As an example, control unit 21D basically updates the predetermined angle in increments of 10 degrees, but if the maximum opening angle determined by determination unit 21C is greater than a specific angle (e.g., 70 degrees), it updates the predetermined angle in increments of 20 degrees.

[0104] The identification unit 21B in the second embodiment re-identifies the three-dimensional position of the obstacle relative to the vehicle 20 by using the corresponding points of the obstacle determined based on multiple images captured by the camera 41 from a viewpoint at a predetermined angle and viewpoints at other door opening angles, and the door opening angles associated with each of the multiple images. The above-mentioned "other door opening angle" may be a door opening angle of "0 degrees" or a door opening angle other than "0 degrees", i.e., a door opening angle of "1 degree" or more.

[0105] The determination unit 21C in the second embodiment determines the maximum opening angle again using the three-dimensional position of the obstacle identified again by the identification unit 21B and the door information.

[0106] The receiving unit 21E receives an input of the number of times the maximum opening angle is to be determined by the determining unit 21C (hereinafter referred to as the "determined number of times"). As an example, the receiving unit 21E receives a value designated by the occupant operating the monitor 43 as the determined number of times.

[0107] FIG. 7 is a second flowchart showing the flow of the release process. 7, the CPU 21 receives an input of the number of determinations, and then proceeds to step S21. As an example, it is assumed that the number of determinations received as input by the CPU 21 is two.

[0108] In step S21, CPU 21 acquires an image captured by camera 41 and the door opening angle at which the image was captured. As an example, in the first execution of step S21, CPU 21 acquires an image captured by camera 41 at a door opening angle of "0 degrees" and an image captured by camera 41 at a door opening angle of "7 degrees." In addition, in the second execution of step S21, CPU 21 acquires an image captured by camera 41 at a door opening angle of "17 degrees," which is the above-mentioned predetermined angle. Then, the process proceeds to step S22.

[0109] In step S22, the CPU 21 identifies the three-dimensional position of the obstacle relative to the vehicle 20 using the corresponding points of the obstacle determined based on the multiple images captured by the camera 41 from multiple viewpoints with different door opening angles acquired in step S21, and the door opening angles associated with each of the multiple images. As an example, in the first iteration of step S22, the CPU 21 determines the corresponding points of the obstacle based on images captured by the camera 41 from a viewpoint where the door opening angle is "0 degrees" and a viewpoint where the door opening angle is "7 degrees". In addition, in the second iteration of step S22, the CPU 21 determines the corresponding points of the obstacle based on images captured by the camera 41 from a viewpoint where the door opening angle is "17 degrees" and a viewpoint where the door opening angle is "0 degrees". Then, the process proceeds to step S23.

[0110] In step S23, the CPU 21 determines the maximum opening angle using the three-dimensional position of the obstacle identified in step S22 and the door information of the driver's door, and then proceeds to step S24.

[0111] In step S24, CPU 21 determines whether the number of times the maximum opening angle was determined in step S23 has reached the number of times the input was accepted in step S20, and if it is determined that the number of times has been reached (step S24: YES), the process proceeds to step S25. On the other hand, if CPU 21 does not determine that the number of times has been reached (step S24: NO), the process returns to step S21.

[0112] In step S25, the CPU 21 opens the driver's door to the maximum opening angle determined in the immediately preceding step S23, and then ends the opening process.

[0113] As described above, in the second embodiment, the CPU 21 determines the maximum opening angle once, and then controls the driver's door to open to a predetermined angle within the range of the maximum opening angle at which no image is captured by the camera 41. The CPU 21 also re-specifies the three-dimensional position of the obstacle relative to the vehicle 20 using corresponding points of the obstacle determined based on multiple images captured by the camera 41 from a viewpoint at a predetermined angle and viewpoints at other door opening angles, and the door opening angles associated with each of the multiple images. The CPU 21 then re-determines the maximum opening angle using the re-specified three-dimensional position of the obstacle and the door information. Thus, according to the second embodiment, by re-determining the maximum opening angle, the accuracy of the determined maximum opening angle can be improved compared to a configuration in which the maximum opening angle is determined only once.

[0114] Furthermore, in the second embodiment, the CPU 21 determines the predetermined angle according to the determined maximum opening angle. Here, when the distance to the obstacle is far (a predetermined distance or more), the three-dimensional position of the obstacle can be identified with higher accuracy by using an image with a larger door opening angle rather than an image with a smaller door opening angle. Therefore, according to the second embodiment, as an example, when the determined maximum opening angle is larger than a specific angle, the three-dimensional position of the obstacle can be identified with high accuracy by determining the predetermined angle to be larger than usual.

[0115] In the second embodiment, the CPU 21 receives an input of the number of determinations. Therefore, according to the second embodiment, for example, when the occupant has ample time, the determination of the maximum opening angle can be repeated many times to determine the maximum opening angle at which the door can be opened close to an obstacle. Also, according to the second embodiment, when the occupant does not have ample time, the determination of the maximum opening angle can be completed after a small number of repetitions, allowing the driver's door to be opened quickly.

[0116] (Third embodiment) Next, a third embodiment will be described while omitting or simplifying parts that overlap with other embodiments.

[0117] FIG. 8 is a second block diagram showing the hardware configuration of vehicle 20. As shown in FIG. As shown in FIG. 8, the vehicle 20 in the third embodiment includes an on-board device 15, a door ECU 30, an actuator 31, an angle sensor 32, a microphone 40, a camera 41, an input switch 42, a monitor 43, a speaker 44, a GPS device 45, and a sonar sensor 46.

[0118] The sonar sensor 46 is provided at least on the driver's door and is a device that uses ultrasonic waves to detect the distance to an obstacle approaching the side of the vehicle. The sonar sensor 46 is an example of a "distance sensor."

[0119] FIG. 9 is a third block diagram showing an example of the functional configuration of the vehicle-mounted device 15. As shown in FIG. 9, the CPU 21 of the vehicle-mounted device 15 has, as functional components, an acquisition unit 21A, an identification unit 21B, a determination unit 21C, a control unit 21D, a reception unit 21E, and a correction unit 21F. Each functional component is realized by the CPU 21 reading and executing an information processing program stored in the storage unit 24.

[0120] The control unit 21D in the third embodiment performs control to prohibit the opening of the driver's door based on the detection result of the sonar sensor 46 provided on the driver's door. Specifically, when the sonar sensor 46 detects an obstacle that is close to or approaching the driver's door, the control unit 21D performs control to prohibit the opening of the driver's door.

[0121] Correction unit 21F corrects the image captured by camera 41 using the internal parameters of camera 41. As an example, correction unit 21F corrects the image by performing distortion correction. At this time, correction unit 21F uses parameters for correcting optical distortion for each camera model, focal length, etc. as the internal parameters of camera 41. These internal parameters are stored in storage unit 24 in advance.

[0122] As an example, distortion correction by the correction unit 21F is performed using the following method: Scaramuzza, D., A. Martinelli, and R. Siegwart, "A Toolbox for Easy Calibrating Omnidirectional Cameras," Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), October 7-15, 2006.

[0123] FIG. 10 is a third flowchart showing the flow of the release process. 10, the CPU 21 accepts input of the number of determinations, and then proceeds to step S31.

[0124] In step S31, the CPU 21 acquires an image captured by the camera 41 and the door opening angle at the time the image was captured, and then proceeds to step S32.

[0125] In step S32, CPU 21 corrects the image acquired in step S31 using the internal parameters of camera 41. Then, the process proceeds to step S33.

[0126] In step S33, the CPU 21 uses the corresponding points of the obstacle determined based on the multiple images corrected in step S32 and the door opening angles associated with each of the multiple images to identify the three-dimensional position of the obstacle relative to the vehicle 20. Then, the process proceeds to step S34.

[0127] In step S34, the CPU 21 determines the maximum opening angle using the three-dimensional position of the obstacle identified in step S33 and the door information of the driver's door, and then proceeds to step S35.

[0128] In step S35, CPU 21 determines whether the number of times the maximum opening angle was determined in step S34 has reached the number of times the input was accepted in step S30, and if it is determined that the number of times has been reached (step S35: YES), the process proceeds to step S36. On the other hand, if CPU 21 does not determine that the number of times has been reached (step S35: NO), the process returns to step S31.

[0129] In step S36, the CPU 21 opens the driver's door to the maximum opening angle determined in the immediately preceding step S34, and then ends the opening process.

[0130] As described above, in the third embodiment, the CPU 21 performs control to prohibit the opening of the driver's door based on the detection result of the sonar sensor 46 provided on the driver's door. As a result, according to the third embodiment, for example, the opening of the driver's door can be prohibited when the sonar sensor 46 detects an obstacle that is close to or approaching the driver's door.

[0131] Furthermore, in the third embodiment, CPU 21 corrects the image captured by camera 41 using the internal parameters of camera 41. As a result, according to the third embodiment, the three-dimensional position of the obstacle is identified using the corrected image, and therefore the three-dimensional position of the obstacle can be identified with higher accuracy than in a configuration in which image correction is not performed.

[0132] (others) In the above embodiment, the driver's door of the vehicle 20 is an example of a "hinge door." However, instead of or in addition to this, at least one of the passenger door and the rear door may be an example of a "hinge door." When at least one of the passenger door and the rear door is an example of a "hinge door," the vehicle 20 is equipped with actuators that automatically drive these doors to open and close, angle sensors that detect the door opening angles of these doors, and cameras that are provided on these doors and that capture images of the sides of the vehicle. Furthermore, when at least one of the passenger door and the rear door is an example of a "hinge door," a sonar sensor may be provided on these doors.

[0133] In the above embodiment, an example has been described in which the opening process is initiated when the occupant is inside the vehicle 20, but this is not limiting, and the opening process may be initiated when the occupant is outside the vehicle 20. As an example, the opening process may be initiated when an electronic key corresponding to the vehicle 20 is detected when the occupant is outside the vehicle 20.

[0134] In the above embodiment, the camera 41 is provided on the door mirror 33 of the driver's door of the vehicle 20, but this is not limiting, and the camera 41 may be provided on the driver's door itself.

[0135] In the above embodiment, the obstacle present around the driver's door may be any object that appears in the image captured by camera 41, and may be present in a position that will come into contact with the driver's door when the driver's door is opened, or may be present in a position that will not come into contact with the driver's door.

[0136] In the above embodiment, the in-vehicle device 15 is an example of the "information processing device," but the present invention is not limited to this, and an external device such as a server connectable to the vehicle 20 may also be an example of the "information processing device." In this case, for example, the external device may be configured to have the functions of the acquisition unit 21A, the identification unit 21B, the determination unit 21C, and the correction unit 21F described in the above embodiment, and the vehicle 20 may be configured to have the functions of the control unit 21D and the reception unit 21E.

[0137] In the above embodiment, the release process executed by the CPU 21 after reading the software (program) may be executed by various processors other than a CPU. Examples of such processors include programmable logic devices (PLDs) whose circuit configuration can be changed after fabrication, such as field-programmable gate arrays (FPGAs), and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. The release process may be executed by one of these processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0138] In the above embodiment, the information processing program is pre-stored (installed) in the storage unit 24, but the present invention is not limited to this. The information processing program may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program may also be downloaded from an external device via a network. [Explanation of symbols]

[0139] 15. In-vehicle device (an example of an information processing device) 20 vehicles 21A Acquisition Department 21B Specific part 21C Decision section 41 Camera (an example of an imaging unit)

Claims

1. an acquisition unit that acquires an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle that indicates an angle at which the hinged door is opened from a closed state when the image is captured; an identification unit that identifies a three-dimensional position of the obstacle relative to the vehicle using a corresponding point of the obstacle present around the hinged door, which is determined based on the plurality of images captured by the imaging unit from a plurality of viewpoints with different door opening angles and acquired by the acquisition unit, and the door opening angle associated with each of the plurality of images; a determination unit that determines the maximum door opening angle at which the hinge door does not come into contact with the obstacle, using the three-dimensional position of the obstacle identified by the identification unit and door information related to the shape and dimensions of the hinge door; a control unit that controls the hinge door to open to the maximum door opening angle at which the hinge door does not come into contact with the obstacle, the maximum door opening angle being determined by the determination unit; Equipped with The control unit After the determination unit determines the maximum door opening angle at which the door will not come into contact with the obstacle, the determination unit controls the hinge door to open to a predetermined angle at which the image is not captured by the imaging unit within the range of the maximum door opening angle at which the door will not come into contact with the obstacle, The identification unit re-identifying a three-dimensional position of the obstacle relative to the vehicle using the corresponding point of the obstacle determined based on the plurality of images captured by the imaging unit from a viewpoint at the predetermined angle and a viewpoint at another door opening angle, and the door opening angle associated with each of the plurality of images; The determination unit using the three-dimensional position of the obstacle identified again by the identification unit and the door information, again determining the maximum door opening angle at which the hinge door does not come into contact with the obstacle; Information processing device.

2. The control unit determining the door opening angle when the imaging unit captures the image; performing control to open the hinge door to the determined door opening angle; The information processing device according to claim 1 .

3. The control unit determining the predetermined angle according to the maximum door opening angle at which the door does not come into contact with the obstacle, as determined by the determination unit; The information processing device according to claim 1 .

4. a receiving unit that receives an input of the number of times that the determination unit determines the maximum door opening angle at which the hinge door does not come into contact with the obstacle, The information processing device according to claim 1 .

5. The control unit a control for prohibiting the hinge door from opening based on a detection result of a distance measuring sensor provided on the hinge door; The information processing device according to claim 1 .

6. a correction unit that corrects the image captured by the imaging unit using internal parameters of the imaging unit, The information processing device according to claim 1 .

7. Acquire an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating an angle by which the hinged door was opened from a closed state when the image was captured; identifying a three-dimensional position of the obstacle relative to the vehicle using a corresponding point of the obstacle present around the hinged door, which is determined based on a plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the door opening angle associated with each of the plurality of images; determining a maximum door opening angle at which the hinged door does not come into contact with the obstacle by using the identified three-dimensional position of the obstacle and door information relating to the shape and dimensions of the hinged door; After determining the maximum door opening angle at which the door will not come into contact with the obstacle, control is performed to open the hinged door to a predetermined angle at which the image is not captured by the imaging unit within the range of the maximum door opening angle at which the door will not come into contact with the obstacle, re-identifying a three-dimensional position of the obstacle relative to the vehicle using the corresponding point of the obstacle determined based on the plurality of images captured by the imaging unit from a viewpoint at the predetermined angle and a viewpoint at another door opening angle, and the door opening angle associated with each of the plurality of images; Using the re-identified three-dimensional position of the obstacle and the door information, the maximum door opening angle at which the hinged door does not come into contact with the obstacle is determined again; performing control to open the hinge door to the determined maximum door opening angle at which the hinge door does not come into contact with the obstacle; An information processing method in which processing is performed by a computer.

8. On the computer, Acquire an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating an angle by which the hinged door was opened from a closed state when the image was captured; identifying a three-dimensional position of the obstacle relative to the vehicle using a corresponding point of the obstacle present around the hinged door, which is determined based on a plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles, and the door opening angle associated with each of the plurality of images; determining a maximum door opening angle at which the hinged door does not come into contact with the obstacle by using the identified three-dimensional position of the obstacle and door information relating to the shape and dimensions of the hinged door; After determining the maximum door opening angle at which the door will not come into contact with the obstacle, control is performed to open the hinged door to a predetermined angle at which the image is not captured by the imaging unit within the range of the maximum door opening angle at which the door will not come into contact with the obstacle, re-identifying a three-dimensional position of the obstacle relative to the vehicle using the corresponding point of the obstacle determined based on the plurality of images captured by the imaging unit from a viewpoint at the predetermined angle and a viewpoint at another door opening angle, and the door opening angle associated with each of the plurality of images; Using the re-identified three-dimensional position of the obstacle and the door information, the maximum door opening angle at which the hinged door does not come into contact with the obstacle is determined again; performing control to open the hinge door to the determined maximum door opening angle at which the hinge door does not come into contact with the obstacle; An information processing program for executing processing.

Images