Information processing device, information processing method, and information processing program

Abstract

To accurately identify three-dimensional positions of obstacles around a vehicle.SOLUTION: An information processing device includes: an acquisition unit that acquires an image captured by an imaging unit provided on a hinge door and a door opening angle indicating an angle at which the hinge door is opened from a closed state when the image is captured; an identification unit that identifies a three-dimensional position of an obstacle relative to a vehicle and an error in the door opening angle by using corresponding points of an obstacle present around the hinge door determined based on a plurality of images captured by the imaging unit from a plurality of viewpoints with different acquired door opening angles and the door opening angles associated with each of the plurality of images; and a determination unit that removes a part of the corresponding points based on the error in the door opening angle, and determines the maximum door opening angle at which the hinge door does not come into contact with the obstacle by using the three-dimensional position of the obstacle identified based on the corresponding points after the removal and door information related to a shape and dimensions of the hinge door.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Patent Document 1 discloses a technology in which a power hinge door is opened by a power door control unit when a vehicle user approaches the power hinge door. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-147856 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] As mentioned above, there is a conventional technology that automatically opens vehicle hinged doors without human intervention. In situations where obstacles such as walls and other vehicles are present around the vehicle, the hinged doors must be opened within a range that avoids contact with the obstacles. To open the hinged doors within a range that avoids contact with obstacles, it is necessary to accurately determine the three-dimensional position of the obstacles relative to the vehicle, but there is still room for improvement in the method of determining the three-dimensional position of obstacles.

[0005] Therefore, the present invention aims to provide an information processing device, an information processing method, and an information processing program that can accurately identify the three-dimensional position of obstacles present around a vehicle. [Means for solving the problem]

[0006] The information processing apparatus according to the first aspect includes an acquisition unit that acquires an image captured by an imaging unit provided on a hinged door of a vehicle and a door opening angle indicating an angle at which the hinged door has opened from a closed state when the image was captured, a specifying unit that specifies a three-dimensional position of an obstacle existing around the hinged door and an error in the door opening angle with respect to the vehicle, based on a plurality of the images captured by the imaging unit at a plurality of viewpoints with different door opening angles acquired by the acquisition unit, and a correspondence point of the obstacle, and a determination unit that determines a maximum door opening angle at which the hinged door does not contact the obstacle, using a three-dimensional position of the obstacle specified based on the correspondence point after a part of the correspondence point has been removed based on the error in the door opening angle, and door information regarding the shape and dimensions of the hinged door. A control unit that controls the opening of the hinged door to the maximum door opening angle determined by the determination unit so as not to come into contact with the obstacle, comprises The control unit, after determining the maximum door opening angle at which the determination unit does not come into contact with the obstacle, controls the hinged door to open to a predetermined angle within the range of the maximum door opening angle at which the door does not come into contact with the obstacle, and the identification unit then uses the viewpoint at the predetermined angle, the corresponding point of the obstacle determined based on the multiple images captured by the imaging unit at other door opening angles, and the door opening angle associated with each of the multiple images to re-determine the three-dimensional position of the obstacle relative to the vehicle.

[0007] The information processing apparatus according to the second aspect is the information processing apparatus according to the first aspect, wherein the specifying unit re-specifies the three-dimensional position of the obstacle based on the correspondence point after removal and the error in the door opening angle, and the determination unit determines a maximum door opening angle at which the hinged door does not contact the obstacle, using the three-dimensional position of the obstacle re-specified by the specifying unit and the door information.

[0009] The 3 information processing apparatus according to the aspect is the information processing apparatus according to the 1 aspect, wherein the control unit determines the door opening angle when the imaging unit captures the image, and performs control to open the hinged door up to the determined door opening angle.

[0010] The 4 information processing apparatus according to the aspect is the information processing apparatus according to the 1 aspect, and before the determination unit re-determines a maximum door opening angle at which the hinged door does not contact the obstacle, using the three-dimensional position of the obstacle re-specified by the specifying unit and the door information.

[0011] The5 The information processing device of the form is the 4 An information processing apparatus in the manner of the above, wherein the control unit determines the predetermined angle according to the maximum door opening angle that does not come into contact with the obstacle, as determined by the determination unit.

[0012] The 6 The information processing device of the form is the 4 An information processing device of the present invention, comprising a receiving unit that receives input of the number of times the determination unit determines the maximum door opening angle at which the hinged door does not come into contact with the obstacle.

[0013] The 7 The information processing device of the form is the 1 An information processing device in the manner of the above, wherein the control unit performs control to prohibit the opening of the hinged door based on the detection result of a distance measuring sensor provided on the hinged door.

[0014] The 8 The information processing method of this embodiment acquires an image captured by an imaging unit provided on the hinge door of a vehicle, and a door opening angle indicating the angle at which the hinge door was opened from a closed state when the image was captured. Using the acquired door opening angles and corresponding points of obstacles present around the hinge door, determined based on a plurality of images captured by the imaging unit from multiple viewpoints with different door opening angles, and the door opening angles associated with each of the plurality of images, the method identifies the error in the three-dimensional position of the obstacle relative to the vehicle and the door opening angle. A portion of the corresponding points is removed based on the error in the door opening angle, and the three-dimensional position of the obstacle identified based on the removed corresponding points, along with door information relating to the shape and dimensions of the hinge door, determines the maximum door opening angle at which the hinge door does not come into contact with the obstacle. The system controls the hinged door to open it to the maximum door opening angle that does not come into contact with the determined obstacle. After determining the maximum door opening angle that does not come into contact with the obstacle, the system controls the hinged door to open it to a predetermined angle within the range of the maximum door opening angle that does not come into contact with the obstacle, where no image has been captured by the imaging unit. The system then re-determines the three-dimensional position of the obstacle relative to the vehicle using the viewpoint at the predetermined angle, the corresponding point of the obstacle determined based on the multiple images captured by the imaging unit at other door opening angles, and the door opening angles associated with each of the multiple images. The computer then performs the processing.

[0015] The 9The information processing program in this embodiment acquires an image captured by an imaging unit provided on the vehicle's hinged door and a door opening angle indicating the angle at which the hinged door was opened from a closed state when the image was captured. Using the acquired door opening angles and corresponding points of obstacles located around the hinged door, determined based on a plurality of images captured by the imaging unit from multiple viewpoints with different door opening angles, and the door opening angles associated with each of the plurality of images, the program identifies the error in the three-dimensional position of the obstacle relative to the vehicle and the door opening angle. Some of the corresponding points are removed based on the error in the door opening angle, and the three-dimensional position of the obstacle identified based on the removed corresponding points, along with door information relating to the shape and dimensions of the hinged door, determines the maximum door opening angle at which the hinged door does not come into contact with the obstacle. The system controls the hinged door to open it to the maximum door opening angle that does not come into contact with the determined obstacle. After determining the maximum door opening angle that does not come into contact with the obstacle, the system controls the hinged door to open it to a predetermined angle within the range of the maximum door opening angle that does not come into contact with the obstacle, where no image has been captured by the imaging unit. The system then re-determines the three-dimensional position of the obstacle relative to the vehicle using the viewpoint at the predetermined angle, the corresponding point of the obstacle determined based on the multiple images captured by the imaging unit at other door opening angles, and the door opening angles associated with each of the multiple images. , execute the process. [Effects of the Invention]

[0016] The information processing apparatus, information processing method, and information processing program according to the present invention can accurately identify the three-dimensional position of obstacles present around a vehicle. [Brief explanation of the drawing]

[0017] [Figure 1] This is the first block diagram showing the vehicle's hardware configuration. [Figure 2] This is the first block diagram showing an example of the functional configuration of an in-vehicle device. [Figure 3] This is the first flowchart illustrating the release process. [Figure 4] This is the first explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle. [Figure 5] This is the second explanatory diagram illustrating a method for identifying the three-dimensional position of an obstacle and a method for determining the maximum opening angle. [Figure 6] This is a second block diagram showing an example of the functional configuration of an in-vehicle device. [Figure 7] This is a third block diagram showing an example of the functional configuration of an in-vehicle device. [Figure 8] This is the second flowchart illustrating the release process. [Figure 9] This is a second block diagram showing the vehicle's hardware configuration. [Figure 10] This is the third flowchart illustrating the release process. [Modes for carrying out the invention]

[0018] The vehicle 20 according to this embodiment will be described below. (First embodiment) Figure 1 is a first block diagram showing the hardware configuration of vehicle 20. Vehicle 20 may be a gasoline vehicle, a hybrid vehicle, or an electric vehicle, but in the first embodiment, as an example, vehicle 20 is a gasoline vehicle. Vehicle 20 also has a driver's side door, a passenger side door, and a rear door at the rear of the vehicle 20. In the first embodiment, the driver's side door of vehicle 20 is a hinged door with a known axis of rotation in vehicle body coordinates. The driver's side door is an example of a "hinged door".

[0019] As shown in Figure 1, the vehicle 20 is composed of an in-vehicle unit 15, a door ECU (Electronic Control Unit) 30, an actuator 31, an angle sensor 32, a microphone 40, a camera 41, an input switch 42, a monitor 43, a speaker 44, and a GPS device 45. The in-vehicle unit 15 is an example of an "information processing device".

[0020] The in-vehicle unit 15 consists of a CPU 21 (Central Processing Unit), ROM 22 (Read Only Memory), RAM 23 (Random Access Memory), storage unit 24, in-vehicle communication interface 25, input / output interface 26, and wireless communication interface 27. The CPU 21, ROM 22, RAM 23, storage unit 24, in-vehicle communication interface 25, input / output interface 26, and wireless communication interface 27 are interconnected via an internal bus 28 so that they can communicate with each other.

[0021] The CPU 21 is a central processing unit that executes various programs and controls various components. Specifically, the CPU 21 reads programs from the ROM 22 or memory unit 24 and executes them using the RAM 23 as a working area. The CPU 21 controls each of the above components and performs various calculations according to the programs recorded in the ROM 22 or memory unit 24.

[0022] ROM22 stores various programs and data. RAM23 temporarily stores programs or data as a working area.

[0023] The storage unit 24 is composed of a storage device such as eMMC (embedded Multi Media Card) or UFS (Universal Flash Storage) and stores various programs and data. The storage unit 24 stores at least an information processing program 24A for performing the release process described later.

[0024] The in-vehicle communication interface 25 is an interface for connecting to the door ECU 30. This interface uses the CAN protocol for communication. The in-vehicle communication interface 25 is connected to the external bus 29.

[0025] In the first embodiment, a door ECU 30 is provided as the ECU. Although not shown in the figures, multiple ECUs are provided for each function of the vehicle 20, and the configuration includes other ECUs besides the door ECU 30. The door ECU 30 is connected to the actuator 31 and the angle sensor 32.

[0026] The actuator 31 automatically drives at least the driver's side door of the vehicle 20 to open and close. In the first embodiment, the door ECU 30 drives the actuator 31 based on the control of the in-vehicle device 15, so that the driver's side door can be opened and closed automatically without the occupant having to perform the opening and closing operation of the driver's side door.

[0027] The angle sensor 32 is installed on at least the driver's side door of the vehicle 20 and is a sensor for detecting the door opening angle, which indicates the angle at which the driver's side door is open from the closed state. The door opening angle detected by the angle sensor 32 is stored in the storage unit 24.

[0028] The input / output interface 26 is an interface for communicating with the microphone 40, camera 41, input switch 42, monitor 43, speaker 44, and GPS device 45 mounted on the vehicle 20.

[0029] The microphone 40 is installed on the front pillar or dashboard of the vehicle 20 and is a device that collects voices emitted by the user of the vehicle 20.

[0030] Camera 41 is configured to include, for example, solid-state image sensors such as a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Camera 41 is, for example, installed at least on the door mirror 33 of the driver's side door of the vehicle 20 (see Figures 4 and 5) and captures images of the side of the vehicle. The images captured by camera 41 are linked to the door opening angle at the time each image was captured and stored in the storage unit 24. Camera 41 may also be connected to the in-vehicle unit 15 via an ECU (for example, a camera ECU). Camera 41 is an example of an "imaging unit".

[0031] Furthermore, the orientation of camera 41 in the vehicle body coordinates when the driver's side door is closed is known, and information regarding this orientation is stored in the memory unit 24.

[0032] The input switch 42 is provided on the instrument panel, center console, steering wheel, etc., and is a switch that receives input from the driver's fingers. For example, the input switch 42 can be a push-button type numeric keypad or a touchpad. In the first embodiment, at least an open switch for opening the driver's side door is provided as the input switch 42. In the first embodiment, when the vehicle 20 is stopped or parked, the open switch is operated and the driver's side door can be opened automatically.

[0033] The monitor 43 is a liquid crystal monitor provided on the instrument panel or meter panel, etc., for displaying images related to the operation of the functions of the vehicle 20 and explanations of those functions. The monitor 43 may also be provided as a touch panel that also functions as an input switch 42.

[0034] The speaker 44 is installed on the instrument panel, center console, front pillar, or dashboard, and is a device for outputting audio related to the operation of the functions of the vehicle 20 and explanations of those functions. The speaker 44 may also be installed on the monitor 43.

[0035] The GPS device 45 is a device that measures the current position of the vehicle 20. The GPS device 45 includes an antenna (not shown) that receives signals from GPS satellites. The GPS device 45 may be connected to the in-vehicle unit 15 via a car navigation system connected to an ECU (e.g., a multimedia ECU).

[0036] The wireless communication interface (I / F27) is a wireless communication module for communicating with other devices. This wireless communication module utilizes communication standards such as 5G, LTE, and Wi-Fi (registered trademark).

[0037] Next, we will explain the functional configuration of the in-vehicle unit 15. Figure 2 is a first block diagram showing an example of the functional configuration of the in-vehicle unit 15.

[0038] As shown in Figure 2, the CPU 21 of the in-vehicle device 15 has a functional configuration consisting of an acquisition unit 21A, a identification unit 21B, a determination unit 21C, and a control unit 21D. Each functional configuration is realized by the CPU 21 reading and executing an information processing program 24A stored in the storage unit 24.

[0039] The acquisition unit 21A acquires an image captured by the camera 41 and the door opening angle at the time the image was captured. In the first embodiment, the acquisition unit 21A acquires multiple images captured by the camera 41 from multiple viewpoints with different door opening angles, and the door opening angle associated with each of the multiple images.

[0040] The identification unit 21B identifies the error in the three-dimensional position of the obstacle and the door opening angle relative to the vehicle 20 (hereinafter sometimes referred to as "door angle error") by using the corresponding points of obstacles present around the driver's side door, determined based on multiple images captured by the camera 41 from multiple viewpoints with different door opening angles acquired by the acquisition unit 21A, and the door opening angles associated with each of the multiple images. The corresponding points of the obstacle are determined by performing a process to extract feature points of known images on multiple images captured by the camera 41 from multiple viewpoints with different door opening angles. Furthermore, the identification unit 21B identifies the three-dimensional position of the obstacle using the Multi-View Stereo (MVS) method, which is a technique for reconstructing the three-dimensional shape of an object using multiple images captured from different viewpoints. Here, the door opening angle detected by the angle sensor 32 may have errors due to measurement errors dependent on the angle sensor 32 (e.g., sensor mounting errors, sampling errors) and measurement errors dependent on the driver's side door (e.g., errors due to door deflection). Therefore, the identification unit 21B assumes that there is an error in the door opening angle detected by the angle sensor 32 and identifies the door angle error using the Multi-View Stereo method.

[0041] The determination unit 21C removes some of the corresponding points based on door angle errors, and uses the 3D position of the obstacle identified based on the removed corresponding points, along with door information regarding the shape and dimensions of the driver's door, to determine the maximum door opening angle at which the driver's door does not come into contact with the obstacle (hereinafter referred to as the "maximum opening angle"). The door information is pre-stored in the storage unit 24. As will be described in detail later, the determination unit 21C removes corresponding points (e.g., 20 points) from among the corresponding points (e.g., 100 points) extracted as feature points from multiple images with different door opening angles, for example, two pairs of images showing the same obstacle, where the correspondence between the image pairs is different, i.e., where it is assumed that a different feature point was extracted in the image pair. As a result, the determination unit 21C can determine the maximum opening angle using the 3D position of the obstacle identified based on the corresponding points (e.g., 80 points) where the correspondence between the image pairs is assumed to be correct. In the following, corresponding points where it is assumed that a different feature point was extracted in multiple images with different door opening angles will be referred to as "incorrect corresponding points".

[0042] The control unit 21D determines the door opening angle when the camera 41 captures an image and controls the driver's side door to open to the determined door opening angle. In the first embodiment, as an example, the control unit 21D determines the first door opening angle when the camera 41 captures an image to be "0 degrees" and the second door opening angle to be "7 degrees".

[0043] Furthermore, the control unit 21D controls the driver's side door to open to the maximum opening angle determined by the determination unit 21C.

[0044] Figure 3 is a first flowchart showing the flow of the opening process, which involves determining the maximum opening angle and opening the driver's side door to the determined maximum opening angle. The CPU 21 reads the information processing program 24A from the memory unit 24, loads it into the RAM 23, and executes it to perform the opening process. For example, the opening process starts when the opening switch is operated while the vehicle 20 is stopped or parked.

[0045] In step S10 shown in Figure 3, the CPU 21 acquires the image captured by the camera 41 and the door opening angle at which the image was captured. As an example, the CPU 21 acquires the image captured by the camera 41 at a door opening angle of "0 degrees" and the image captured by the camera 41 at a door opening angle of "7 degrees". Then, the process proceeds to step S11.

[0046] In step S11, the CPU 21 uses the corresponding points of the obstacle determined in step S10 based on multiple images captured by the camera 41 from multiple viewpoints with different door opening angles, and the door opening angles associated with each of those multiple images, to identify the three-dimensional position of the obstacle and the door angle error relative to the vehicle 20. Then, the process proceeds to step S12. The method for identifying the three-dimensional position of the obstacle and the door angle error will be described later.

[0047] In step S12, the CPU 21 removes the incorrectly matched points among the corresponding points of the obstacle determined in step S11. In other words, the CPU 21 excludes the information (data) indicated by the incorrectly matched points. Then, the process proceeds to step S13. The method for removing these incorrectly matched points will be described later.

[0048] In step S13, the CPU 21 determines the maximum opening angle using the 3D position of the obstacle identified based on the corresponding points after removing the mismatched points in step S12, along with the door information. Then, the process proceeds to step S14. The method for determining the maximum opening angle will be described later.

[0049] In step S14, the CPU 21 opens the driver's side door to the maximum opening angle determined in step S13. Then, the opening process ends.

[0050] Next, using Figures 4 and 5, we will explain how to identify the three-dimensional position of an obstacle and how to determine the maximum opening angle.

[0051] FIG. 4 is a first explanatory diagram for explaining a method of specifying the three-dimensional position of an obstacle and a method of determining the maximum opening angle. FIG. 4 shows a reference coordinate system and also shows a state where the door mirror 33 of the driver's seat door is viewed from above. In the first embodiment, the reference coordinate system is set such that the hinge 34, which is the rotation axis of the driver's seat door, coincides with the Y-axis, where X indicates the direction toward the rear of the vehicle, Y indicates the direction toward the bottom of the vehicle, and Z indicates the direction toward the right side of the vehicle. Also, the origin is set as the point where the Y-axis intersects the ground.

[0052] As shown in FIG. 4, the initial position of the camera 41 in the reference coordinate system when the door is closed is T c0 _w=(X c0 _w,Y c0 _w,Z c0 _w) T , the radius of rotation of the camera 41 with respect to the hinge 34 is L c , the door opening angle is α, and the position of the camera 41 in the reference coordinate system when the door opening angle is α is T cα _w=(X cα _w,Y cα _w,Z cα _w) T . T represents transpose. Also, as shown in FIG. 4, the camera 41 in the initial position can be regarded as having rotated by α0 around the hinge 34. At this time, the CPU 21 uses the following mathematical formulas (1) and (2) to calculate L c0 _w and Z c0 _w from X c and α0.

[0053]

Equation

[0054]

Equation

[0055] Also, the CPU 21 uses the following mathematical formula (3) to calculate (X cα _w,Y cα _w,Z cα _w).

[0056]

number

[0057] As shown in Figure 4, the orientation of camera 41 when the door is closed and when the door is open at angle α is represented by a rotation matrix in the reference coordinate system, and these are R c0 _w and R cα Let _w be the rotation matrix representing a rotation of angle θ around the Y axis. Y Let _w(θ), then R c0 _w and R cα The relationship between _w is shown in the following equation (4).

[0058]

number

[0059] Based on the above, the position and orientation of camera 41 at door opening angle α can be determined using the position and orientation of camera 41 when the door is closed and the door opening angle α.

[0060] Next, Figure 5 shows the relationship between the camera coordinate system and the reference coordinate system, as well as the coordinates of the obstacle being measured. Figure 5 is a second explanatory diagram illustrating the method for identifying the three-dimensional position of an obstacle and determining the maximum opening angle.

[0061] As shown in Figure 5, the coordinates of the obstacle in the reference coordinate system corresponding to the corresponding point of the obstacle are given by P_w=(X_w,Y_w,Z_w) T When the door is closed, the coordinates of the obstacle in the camera coordinate system are given by P_c0=(X_c0,Y_c0,Z_c0). T The coordinates of the obstacle in the camera coordinate system at door opening angle α are given by P_cα = (X_cα, Y_cα, Z_cα). T Let's assume that the image coordinates of the obstacle when the door is closed and when the door is open at angle α are given by I_i0=(x_i0,y_i0). T , and I_iα=(x_iα,y_iα) T Let's assume that.

[0062] Here, P_w, P_c0, and P_cα represent the same obstacle in different coordinate systems and have the relationship shown in equations (5) and (6) below.

[0063]

number

[0064]

number

[0065] Furthermore, the internal parameters of the camera 41, namely the focal length in pixels and the image center, are set to f and I, respectively. c _i=(x c _i,y c _i) T Let's assume that the projection formula for the image is given by the following equations (7) and (8).

[0066]

number

[0067]

number

[0068] From the above equations (7) and (8), the following equations (9), (10), (11), and (12) are obtained.

[0069]

number

[0070]

number

[0071]

number

[0072]

number

[0073] Substituting equations (9) and (10) into equation (5) above, we obtain the following equation (13).

[0074]

number

[0075] Substituting equations (11) and (12) into equation (6) above, we obtain the following equation (14).

[0076]

number

[0077] To simplify the following formulas, we replace the constant terms in formulas (13) and (14) above with those shown in formulas (15) and (16) below.

[0078]

number

[0079]

number

[0080] Here, assuming that the door angle error is ε and that ε is small, the rotation matrix is ​​given by the following equation (17).

[0081]

number

[0082] Rotation matrix R for door angle error εBy adding the following, we obtain the equations shown in formulas (18) and (19).

[0083]

number

[0084]

number

[0085] By transforming the above equation (19), we obtain the following equation (20).

[0086]

number

[0087] By setting Z = εZ_cα and solving equation (20) above, we obtain equation (21) below.

[0088]

number

[0089] From Z = εZ_cα, the following equation (22) is obtained.

[0090]

number

[0091] Here, from the above formulas (21) and (22), we obtain Z_c0_i, Z_cα_i, and ε_i for each corresponding point i (i=1 to N). Removing mismatched points (excluding the information indicated by mismatched points) involves removing Z_c0_i and Z_cα_i that correspond to outlier ε_i from the set of ε_i (i=1 to N) that constitute the corresponding point i, and consists of the following two steps.

[0092] In step 1, the CPU 21 removes mismatched points based on a fixed threshold. Specifically, the CPU 21 removes Z_c0_i and Z_cα_i that correspond to ε_i such that |ε_i|>εt, with respect to the angle threshold εt which is predetermined as the threshold. The angle threshold εt can be determined experimentally and can be any angle, such as 5 degrees or 10 degrees.

[0093] In step 2, CPU21 removes mismatched points based on a statistical determination method. In step 2, the points remaining after removing mismatched points in step 1 are denoted as i' (i' = 1 to N'), where N' ≤ N. CPU21 calculates the mean and standard deviation for N' ε_i' points, denoting them as m and σ, respectively. For example, CPU21 uses the Smirnov-Grubbs test to identify and remove outliers. CPU21 uses a coefficient μ determined according to N' and a predetermined significance level (e.g., 5%) to remove Z_c0_i' and Z_cα_i' corresponding to ε_i' where |ε_i'-m| / σ > μ. The Smirnov-Grubbs test is disclosed, for example, in the references (https: / / ja.wikipedia.org / wiki / %E5%A4%96%E3%82%8C%E5%80%A4).

[0094] Here, CPU21 calculates X_c0 and Y_c0 by substituting the values ​​of Z_c0_i' (i'=1~N'), which are obtained from Z_c0_i (i=1~N) from the above formula (21) after removing the mismatched points in steps 1 and 2 above, into the above formulas (9) and (10), and then sets P_c0=(X_c0,Y_c0,Z_c0_i') into the above formula (5). T P_w is calculated by substituting the values. Similarly, CPU21 calculates P_w by substituting the values ​​of Z_cα_i' obtained from the above formula (21), after removing the mismatched points in steps 1 and 2, into the above formulas (11) and (12), and then substituting them into the above formula (6).

[0095] In this case, since P_w calculated from equation (5) above and P_w calculated from equation (6) above do not usually coincide, in the first embodiment, the CPU 21 determines the final solution, i.e., the three-dimensional position of the obstacle relative to the vehicle 20, by taking the average of the two P_w. However, the CPU 21 is not limited to this, and may adopt one of the two P_w values ​​predetermined, or a weighted average of the two P_w, as the final solution.

[0096] The CPU 21 then identifies the three-dimensional position of the obstacle as described above at multiple locations within the obstacle and calculates the three-dimensional positions of N locations within the obstacle.

[0097] Here, the shape of the driver's side door is known, and the CPU 21 determines the height Y in the reference coordinate system. h Radius L of the driver's side door h It is assumed that the 3D points P_w of all the obstacles calculated can be calculated. CPU21 calculates all the 3D points P_w of all obstacles. n =(X_w n ,Y_w n ,Z_w n ) T For n=0 to N-1, perform the following calculations.

[0098] First, CPU21 determines the height of the obstacle Y_w n Height Y is equal to h Radius L of the driver's side door h The CPU 21 then calculates the radius L of the driver's side door. h If the relationship shown in equation (23) below exists, there is a possibility of contact with an obstacle, and therefore the angle θ shown in equation (24) below n Calculate.

[0099]

number

[0100]

number

[0101] Meanwhile, CPU21 measures the radius L of the driver's side door. h If the relationship shown in the following equation (25) holds, then the angle θ n It does not calculate the angle θ. Then, CPU21 calculates all the angles θ. n The smallest of these is determined to be the maximum opening angle.

[0102]

number

[0103] As described above, in the first embodiment, the CPU 21 acquires an image captured by the camera 41 and the door opening angle at the time the image was captured. The CPU 21 also uses the corresponding points of the obstacle determined based on multiple images captured by the camera 41 from multiple viewpoints with different door opening angles, and the door opening angles associated with each of the multiple images, to identify the three-dimensional position of the obstacle relative to the vehicle 20 and the door angle error. The CPU 21 then removes erroneous points based on the door angle error and uses the three-dimensional position of the obstacle identified based on the corresponding points after removing the erroneous points, and the door information of the driver's side door, to determine the maximum opening angle. Here, among the corresponding points extracted from multiple images with different door opening angles, for example, from a pair of images showing the same obstacle, there may be some that correspond to feature points indicating different positions of the obstacle between the image pair. Identifying the three-dimensional position of the obstacle based on corresponding points that include such erroneous points is undesirable because it reduces the accuracy of identifying the three-dimensional position.

[0104] Therefore, in the first embodiment, the CPU 21 identifies the three-dimensional position of the obstacle based on the corresponding points after removing the erroneous points, and uses the three-dimensional position of the obstacle to determine the maximum opening angle. As a result, according to the first embodiment, the three-dimensional position of obstacles present around the vehicle 20 can be identified with high accuracy. Furthermore, according to the first embodiment, the accuracy of identifying the three-dimensional position of obstacles is improved, allowing for accurate determination of the maximum opening angle. Moreover, according to the first embodiment, since the CPU 21 removes erroneous points based on door angle errors, the computational cost required to determine erroneous points can be reduced, which is advantageous for real-time processing.

[0105] When using the Multi-View Stereo method to determine the 3D position of an obstacle, it is necessary to calculate the camera's position and orientation in each image captured from multiple viewpoints. Conventionally, this calculation has been difficult, resulting in insufficient accuracy in estimating the camera's position and orientation. Furthermore, conventional methods have had the problem of requiring a large number of images due to insufficient accuracy in estimating the camera's position and orientation. For example, a relevant document on the Multi-View Stereo method is "A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, CVPR2006".

[0106] In contrast, in the first embodiment, the movement of the camera 41 is constrained by having a hinge 34 that is a common axis of rotation. Therefore, by using information indicating the coordinates of the hinge 34, the position and orientation of the camera in each image captured from multiple viewpoints can be estimated with high accuracy. Accordingly, according to the first embodiment, the 3D position of an obstacle can be determined with fewer images compared to when the 3D position of an obstacle is determined using the conventional Multi-View Stereo method.

[0107] Furthermore, in the first embodiment, the three-dimensional position of an obstacle is determined using a door camera 41, which is a camera installed on the door mirror that is mounted on many vehicles, and an angle sensor 32, so there is no need to add any special parts for this determination.

[0108] Furthermore, in the first embodiment, the CPU 21 controls the driver's side door to open to the determined maximum opening angle. As a result, according to the first embodiment, even when there are obstacles around the driver's side door, the driver's side door can be automatically opened to the maximum extent without contacting the obstacle, without the occupant having to open the door themselves.

[0109] Furthermore, in the first embodiment, the CPU 21 determines the door opening angle when the camera 41 captures an image, and controls the driver's side door to open to the determined door opening angle. As a result, according to the first embodiment, even when there are obstacles around the driver's side door, the driver's side door can be automatically opened within a range that does not come into contact with the obstacle, without the occupant having to open the driver's side door.

[0110] In the first embodiment, even while the CPU 21 is automatically opening the driver's door, the occupant can still manually open and close the driver's door.

[0111] (Second embodiment) Next, the second embodiment will be described, omitting or simplifying any parts that overlap with the other embodiments.

[0112] Figure 6 is a second block diagram showing an example of the functional configuration of the in-vehicle unit 15. As shown in Figure 6, the CPU 21 of the in-vehicle unit 15 has the following functional configuration: acquisition unit 21A, identification unit 21B, determination unit 21C, control unit 21D, and correction unit 21E. Each functional configuration is realized by the CPU 21 reading and executing the information processing program 24A stored in the storage unit 24.

[0113] The correction unit 21E corrects the door angle error acquired by the acquisition unit 21A. The door opening angle corrected by the correction unit 21E is stored in the storage unit 24. As described above, the door opening angle detected by the angle sensor 32 may have errors. Therefore, in the second embodiment, it is assumed that there is an error in the door opening angle detected by the angle sensor 32, and the correction unit 21E corrects this error.

[0114] In the second embodiment, the identification unit 21B re-identifies the three-dimensional position of the obstacle based on the corresponding points and door angle errors after removing the incorrectly identified points.

[0115] In the second embodiment, the determination unit 21C determines the maximum opening angle using the three-dimensional position of the obstacle identified again by the identification unit 21B and the door information of the driver's side door.

[0116] Next, the method for identifying the three-dimensional position of an obstacle and the method for determining the maximum opening angle in the second embodiment will be described, omitting or simplifying any parts that overlap with the first embodiment.

[0117] In the first embodiment, the three-dimensional position of the obstacle was determined based on the corresponding points after removing the erroneous points in steps 1 and 2 above. In the second embodiment, the three-dimensional position of the obstacle is determined again based on the corresponding points after removing the erroneous points and the door angle error.

[0118] The following shows the process after removing the incorrect points in the two steps described in Step 1 and Step 2 above. The CPU 21 calculates the average value of the set of ε' that constitute the correct points after removing the incorrect points. - Let ε be the function of CPU21. - Using ε, R -ε Constitute the following equation (26) and A α Correct B α Let's assume that.

[0119]

number

[0120] CPU21 calculates Z_c0 and Z_cα using the following formula (27).

[0121]

number

[0122] Here, CPU21 calculates X_c0 and Y_c0 by substituting the value of Z_c0 obtained from the above formula (27), after removing the mismatched points in steps 1 and 2, into the above formulas (9) and (10), and then sets P_c0=(X_c0,Y_c0,Z_c0) into formula (5). T P_w is calculated by substituting the values. Similarly, CPU21 calculates P_w by substituting the value of Z_cα obtained from the above formula (27), after removing the mismatched points in steps 1 and 2, into the above formulas (11) and (12), and then substituting it into the above formula (6).

[0123] At this time, since P_w calculated from equation (5) above and P_w calculated from equation (6) above do not usually coincide, in the second embodiment, the CPU 21 takes the average of the two P_w to re-determine the final solution, i.e., the 3D position of the obstacle relative to the vehicle 20. The process after re-determining the 3D position of the obstacle is the same as in the first embodiment, so the explanation is omitted.

[0124] As described above, in the second embodiment, the CPU 21 re-identifies the 3D position of the obstacle based on the corresponding points and door angle errors after removing the erroneous points. Then, the CPU 21 determines the maximum opening angle using the re-identified 3D position of the obstacle and the door information of the driver's side door. As a result, according to the second embodiment, the 3D position of obstacles present around the vehicle 20 can be identified with greater accuracy compared to when the 3D position of the obstacle is not re-identified. Furthermore, in the second embodiment, by assuming that there is an error in the door opening angle detected by the angle sensor 32 and correcting for this error, the 3D position of obstacles present around the vehicle 20, specifically around the driver's side door, can be identified with greater accuracy.

[0125] In the second embodiment, the CPU 21 identifies the three-dimensional position of the obstacle based on the process shown in the first embodiment, and then identifies the three-dimensional position of the obstacle again. However, the CPU 21 is not limited to this, and may identify the three-dimensional position of the obstacle based on the process shown in the second embodiment without first identifying the three-dimensional position of the obstacle based on the process shown in the first embodiment.

[0126] (Third embodiment) Next, the third embodiment will be described, omitting or simplifying any parts that overlap with the other embodiments.

[0127] Figure 7 is a third block diagram showing an example of the functional configuration of the in-vehicle unit 15. As shown in Figure 7, the CPU 21 of the in-vehicle unit 15 has the following functional configuration: acquisition unit 21A, identification unit 21B, determination unit 21C, control unit 21D, and reception unit 21F. Each functional configuration is realized by the CPU 21 reading and executing the information processing program 24A stored in the storage unit 24.

[0128] In the third embodiment, after the determination unit 21C has determined the maximum opening angle, the control unit 21D controls the opening of the driver's side door to a predetermined angle within the range of the maximum opening angle at which no image is captured by the camera 41. At this time, the control unit 21D determines a predetermined angle according to the maximum opening angle determined by the determination unit 21C. For example, the control unit 21D basically updates the predetermined angle in 10-degree increments, but if the maximum opening angle determined by the determination unit 21C is greater than a certain angle (e.g., 70 degrees), the predetermined angle is updated in 20-degree increments.

[0129] In the third embodiment, the identification unit 21B re-identifies the three-dimensional position of the obstacle relative to the vehicle 20 using the corresponding point of the obstacle and the door opening angle associated with each of the multiple images, which are determined based on a viewpoint at a predetermined angle and a plurality of images captured by the camera 41 at viewpoints at other door opening angles. The above-mentioned "other door opening angles" may be a door opening angle of "0 degrees" or a door opening angle other than "0 degrees", i.e., a door opening angle of "1 degree" or more.

[0130] In the third embodiment, the determination unit 21C uses the three-dimensional position of the obstacle identified again by the identification unit 21B and the door information to determine the maximum opening angle again.

[0131] The reception unit 21F receives input for the number of times the maximum opening angle is determined by the determination unit 21C (hereinafter referred to as the "number of determinations"). For example, the reception unit 21F receives a value specified by the crew member operating the monitor 43 as the number of determinations.

[0132] Figure 8 is a second flowchart showing the flow of the release process. In step S20 shown in Figure 8, the CPU 21 receives input for the number of decisions to be made. Then, the process proceeds to step S21. As an example, let's assume that the CPU 21 received input for the number of decisions to be made was 2.

[0133] In step S21, the CPU 21 acquires an image captured by the camera 41 and the door opening angle at which the image was captured. For example, in the first step S21, the CPU 21 acquires an image captured by the camera 41 at a door opening angle of "0 degrees" and an image captured by the camera 41 at a door opening angle of "7 degrees". Then, in the second step S21, the CPU 21 acquires an image captured by the camera 41 at a predetermined angle, namely a door opening angle of "17 degrees". The process then proceeds to step S22.

[0134] In step S22, the CPU 21 determines the corresponding point of the obstacle relative to the vehicle 20 and the door angle error, using the corresponding point of the obstacle determined based on multiple images captured by the camera 41 from multiple viewpoints with different door opening angles acquired in step S21, and the door opening angle associated with each of those multiple images. For example, in the first step S22, the CPU 21 determines the corresponding point of the obstacle based on images captured by the camera 41 from a viewpoint with a door opening angle of "0 degrees" and a viewpoint with a door opening angle of "7 degrees". Then, in the second step S22, the CPU 21 determines the corresponding point of the obstacle based on images captured by the camera 41 from a viewpoint with a door opening angle of "17 degrees" and a viewpoint with a door opening angle of "0 degrees". Then, the process proceeds to step S23.

[0135] In step S23, the CPU 21 removes the incorrectly matched points among the corresponding points of the obstacle determined in step S22. Then, the process proceeds to step S24.

[0136] In step S24, the CPU 21 determines the maximum opening angle using the 3D position of the obstacle identified based on the corresponding points after removing the incorrect points in step S23, and the door information. Then, the process proceeds to step S25.

[0137] In step S25, the CPU 21 determines whether the number of times the maximum opening angle was determined in step S24 has reached the number of determinations for which input was received in step S20. If it determines that the number of determinations has been reached (step S25: YES), the process proceeds to step S26. On the other hand, if the CPU 21 does not determine that the number of determinations has been reached (step S25: NO), the process returns to step S21.

[0138] In step S26, the CPU 21 opens the driver's side door to the maximum opening angle determined in the preceding step S24. Then, the opening process ends.

[0139] As described above, in the third embodiment, the CPU 21 determines the maximum opening angle once, and then controls the driver's side door to open to a predetermined angle within the range of the maximum opening angle where no image has been captured by the camera 41. The CPU 21 also re-identifies the three-dimensional position of the obstacle relative to the vehicle 20 using the corresponding point of the obstacle determined based on the viewpoint at the predetermined angle and multiple images captured by the camera 41 at viewpoints at other door opening angles, and the door opening angle associated with each of the multiple images. Then, the CPU 21 re-determines the maximum opening angle using the re-identified three-dimensional position of the obstacle and the door information. Thus, according to the third embodiment, by re-determining the maximum opening angle, the accuracy of the determined maximum opening angle can be improved compared to a configuration in which the maximum opening angle is determined only once.

[0140] In the third embodiment, the CPU 21 determines a predetermined angle according to the determined maximum opening angle. Here, when the distance to the obstacle is far (more than a predetermined distance), using an image with a larger door opening angle allows for more accurate identification of the obstacle's three-dimensional position than using an image with a smaller door opening angle. Therefore, according to the third embodiment, for example, when the determined maximum opening angle is greater than a specific angle, the predetermined angle can be set to be larger than usual, thereby enabling more accurate identification of the obstacle's three-dimensional position.

[0141] In the third embodiment, the CPU 21 accepts input for the number of determinations. According to the third embodiment, for example, if the occupant has sufficient time, the determination of the maximum opening angle can be repeated many times to determine the maximum opening angle that allows the door to open close to an obstacle. Also, according to the third embodiment, if the occupant does not have sufficient time, the determination of the maximum opening angle can be completed in fewer repetitions, allowing the driver's side door to be opened earlier.

[0142] (Fourth embodiment) Next, the fourth embodiment will be described, omitting or simplifying any parts that overlap with the other embodiments.

[0143] Figure 9 is a second block diagram showing the hardware configuration of vehicle 20. As shown in Figure 9, the vehicle 20 in the fourth embodiment is configured to include an in-vehicle unit 15, a door ECU 30, an actuator 31, an angle sensor 32, a microphone 40, a camera 41, an input switch 42, a monitor 43, a speaker 44, a GPS device 45, and a sonar sensor 46.

[0144] The sonar sensor 46 is installed at least on the driver's side door and is a device that uses ultrasound to detect the distance to an obstacle approaching from the side of the vehicle. The sonar sensor 46 is an example of a "distance measuring sensor".

[0145] An example of the functional configuration of the in-vehicle device 15 in the fourth embodiment is the same as the example of the functional configuration of the in-vehicle device 15 in the second embodiment shown in Figure 6.

[0146] In the fourth embodiment, the correction unit 21E corrects the image captured by the camera 41 using the internal parameters of the camera 41. As an example, the correction unit 21E performs distortion correction as a correction of the image. In this case, the correction unit 21E uses parameters for correcting optical distortion specific to each camera model, as well as the focal length, etc., as the internal parameters of the camera 41. These internal parameters are stored in advance in the storage unit 24.

[0147] As an example, distortion correction by the correction unit 21E is performed using the following method: Scaramuzza, D., A. Martinelli, and R. Siegwart. "A Toolbox for Easy Calibrating Omnidirectional Cameras." Proceedings to IEEE International Conference on Intelligent Robots and Systems (IROS). October 7-15, 2006.

[0148] In the fourth embodiment, the control unit 21D performs control to prohibit the opening of the driver's side door based on the detection result of the sonar sensor 46 installed on the driver's side door. Specifically, if the sonar sensor 46 detects an obstacle that is close to or approaching the driver's side door, the control unit 21D performs control to prohibit the opening of the driver's side door.

[0149] Figure 10 is a third flowchart showing the flow of the release process. In step S30 shown in Figure 10, the CPU 21 acquires the image captured by the camera 41 and the door opening angle at the time the image was captured. Then, the process proceeds to step S31.

[0150] In step S31, the CPU 21 corrects the image acquired in step S30 using the internal parameters of the camera 41. Then, the process proceeds to step S32.

[0151] In step S32, the CPU 21 uses the corresponding points of the obstacle determined based on the multiple images corrected in step S31, and the door opening angles associated with each of those multiple images, to identify the three-dimensional position of the obstacle and the door angle error relative to the vehicle 20. Then, the process proceeds to step S33.

[0152] In step S33, the CPU 21 removes the incorrectly matched points among the corresponding points of the obstacle determined in step S32. Then, the process proceeds to step S34.

[0153] In step S34, the CPU 21 determines the maximum opening angle using the 3D position of the obstacle identified based on the corresponding points after removing the incorrect points in step S33, and the door information. Then, the process proceeds to step S35.

[0154] In step S35, the CPU 21 opens the driver's side door to the maximum opening angle determined in step S34. Then, the opening process ends.

[0155] As described above, in the fourth embodiment, the CPU 21 controls the opening of the driver's side door based on the detection result of the sonar sensor 46 installed on the driver's side door. Thus, according to the fourth embodiment, for example, the opening of the driver's side door can be prohibited when the sonar sensor 46 detects an obstacle that is close to or approaching the driver's side door.

[0156] Furthermore, in the fourth embodiment, the CPU 21 corrects the image captured by the camera 41 using the internal parameters of the camera 41. As a result, according to the fourth embodiment, the three-dimensional position of the obstacle is determined using the corrected image, so the three-dimensional position of the obstacle can be determined with higher accuracy compared to a configuration in which no image correction is performed.

[0157] (others) In the above embodiment, the driver's side door of the vehicle 20 was used as an example of a "hinged door," but instead or in addition, at least one of the passenger side door and the rear door may be used as an example of a "hinged door." If at least one of the passenger side door and the rear door is used as an example of a "hinged door," the vehicle 20 is equipped with an actuator for automatically opening and closing these doors, an angle sensor for detecting the door opening angle of these doors, and cameras provided on these doors and for imaging the side of the vehicle. Furthermore, if at least one of the passenger side door and the rear door is used as an example of a "hinged door," sonar sensors may be provided on these doors.

[0158] In the above embodiment, an example was described in which the unlocking process is initiated when the occupant is inside the vehicle 20. However, the process is not limited to this, and the unlocking process may be initiated when the occupant is outside the vehicle 20. For example, the unlocking process may be initiated when the electronic key corresponding to the vehicle 20 is detected while the occupant is outside the vehicle 20.

[0159] In the above embodiment, the camera 41 was provided on the door mirror 33 of the driver's side door of the vehicle 20, but the invention is not limited to this, and the camera 41 may also be provided on the driver's side door itself.

[0160] In the above embodiment, any obstacles present around the driver's side door may be objects that are captured in the image taken by the camera 41, and may be located in a position that comes into contact with the driver's side door when it is opened, or may be located in a position that does not come into contact with the driver's side door.

[0161] In the above embodiment, the in-vehicle unit 15 was used as an example of an "information processing device," but the invention is not limited to this, and an external device such as a server that can be connected to the vehicle 20 may also be used as an example of an "information processing device." In this case, for example, the external device may be configured to have the functions of the acquisition unit 21A, identification unit 21B, determination unit 21C, and correction unit 21E described in the above embodiment, while the vehicle 20 may be configured to have the functions of the control unit 21D and reception unit 21F.

[0162] In addition, the open process that the CPU 21 reads and executes in the above embodiment may be executed by various processors other than the CPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processes, such as ASICs (Application Specific Integrated Circuits). Furthermore, the open process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0163] Furthermore, although the above embodiment describes an embodiment in which the information processing program 24A is pre-stored (installed) in the storage unit 24, the invention is not limited to this. The information processing program 24A may be provided in the form of a recording medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the information processing program 24A may be provided in the form of a download from an external device via a network.

[0164] This disclosure may adopt the following embodiments: (1) An acquisition unit that acquires an image captured by an imaging unit provided on the hinged door of a vehicle, and the door opening angle, which indicates the angle at which the hinged door was opened from a closed state when the image was captured. A determination unit identifies the error in the three-dimensional position of the obstacle and the door opening angle relative to the vehicle, using the corresponding points of the obstacles present around the hinged door, determined based on the plurality of images captured by the imaging unit from multiple viewpoints with different door opening angles acquired by the acquisition unit, and the door opening angles associated with each of the plurality of images. A determination unit removes some of the aforementioned corresponding points based on the error in the door opening angle, and uses the three-dimensional position of the obstacle identified based on the removed corresponding points and door information relating to the shape and dimensions of the hinged door to determine the maximum door opening angle at which the hinged door does not come into contact with the obstacle. Equipped with, Information processing device.

[0165] (2) The specified unit then re-identifies the three-dimensional position of the obstacle based on the error between the corresponding point and the door opening angle after removal. The determination unit uses the three-dimensional position of the obstacle, which has been identified again by the identification unit, and the door information to determine the maximum door opening angle at which the hinged door does not come into contact with the obstacle. (1) The information processing device described above.

[0166] (3) The system includes a control unit that controls the opening of the hinged door to the maximum door opening angle determined by the determination unit, which does not involve contact with the obstacle. The information processing device described in (1) or (2).

[0167] (4) The control unit, The imaging unit determines the door opening angle when capturing the image, Control is performed to open the hinged door to the determined door opening angle. (3) The information processing device described above.

[0168] (5) The control unit, After the determination unit determines the maximum door opening angle at which the door does not come into contact with the obstacle, it controls the opening of the hinged door to a predetermined angle within the range of the maximum door opening angle at which the door does not come into contact with the obstacle, and at which the imaging unit does not capture an image. The specified part is, Using the corresponding points of the obstacle determined based on the multiple images captured by the imaging unit at the predetermined angle viewpoint and other door opening angles viewpoints, and the door opening angles associated with each of the multiple images, the three-dimensional position of the obstacle relative to the vehicle is re-determined. The aforementioned determination unit, Using the three-dimensional position of the obstacle identified again by the specified unit and the door information, the maximum door opening angle at which the hinged door does not come into contact with the obstacle is determined again. (3) or (4) The information processing device described above.

[0169] (6) The control unit, The determination unit determines the predetermined angle according to the maximum door opening angle that does not come into contact with the obstacle, as determined by the determination unit. (5) The information processing device described above.

[0170] (7) The system includes a receiving unit that receives input for the number of times the determination unit determines the maximum door opening angle at which the hinged door does not come into contact with the obstacle. The information processing device described in (5) or (6).

[0171] (8) The control unit, Based on the detection result of a distance measuring sensor installed on the hinged door, control is performed to prevent the opening of the hinged door. An information processing device described in any one of (3) through (7).

[0172] (9) The system acquires an image captured by an imaging unit installed on the vehicle's hinged door, and the door opening angle, which indicates the angle at which the hinged door was opened from a closed state when the image was captured. Based on the multiple images captured by the imaging unit from multiple viewpoints with different door opening angles, the corresponding points of obstacles present around the hinged door, and the door opening angles associated with each of the multiple images, are used to identify the error in the three-dimensional position of the obstacle and the door opening angle relative to the vehicle. A portion of the aforementioned corresponding points are removed based on the error in the door opening angle, and the maximum door opening angle at which the hinged door does not come into contact with the obstacle is determined using the three-dimensional position of the obstacle identified based on the removed corresponding points and door information relating to the shape and dimensions of the hinged door. An information processing method in which a computer performs the processing.

[0173] (10) On the computer, The system acquires an image captured by an imaging unit installed on the vehicle's hinged door, and the door opening angle, which indicates the angle at which the hinged door was opened from a closed state when the image was captured. Based on the multiple images captured by the imaging unit from multiple viewpoints with different door opening angles, the corresponding points of obstacles present around the hinged door, and the door opening angles associated with each of the multiple images, are used to identify the error in the three-dimensional position of the obstacle and the door opening angle relative to the vehicle. A portion of the aforementioned corresponding points are removed based on the error in the door opening angle, and the maximum door opening angle at which the hinged door does not come into contact with the obstacle is determined using the three-dimensional position of the obstacle identified based on the removed corresponding points and door information relating to the shape and dimensions of the hinged door. An information processing program used to execute a process. [Explanation of symbols]

[0174] 15. In-vehicle devices (an example of information processing equipment) 20 vehicles 21A Acquisition Department 21B Specific part 21C Decision section 21D Control Unit 21E Correction Unit 21F Reception 41. Camera (an example of an imaging unit)

Claims

1. An acquisition unit that acquires an image captured by an imaging unit provided on the hinged door of a vehicle, and the door opening angle, which indicates the angle at which the hinged door was opened from a closed state when the image was captured. A determination unit identifies the error in the three-dimensional position of the obstacle and the door opening angle relative to the vehicle, using the corresponding points of the obstacles present around the hinged door, determined based on the plurality of images captured by the imaging unit from multiple viewpoints with different door opening angles acquired by the acquisition unit, and the door opening angles associated with each of the plurality of images. A determination unit removes some of the aforementioned corresponding points based on the error in the door opening angle, and uses the three-dimensional position of the obstacle identified based on the removed corresponding points and door information relating to the shape and dimensions of the hinged door to determine the maximum door opening angle at which the hinged door does not come into contact with the obstacle. A control unit that controls the opening of the hinged door to the maximum door opening angle determined by the determination unit so as not to come into contact with the obstacle, Equipped with, The control unit, After the determination unit determines the maximum door opening angle at which the door does not come into contact with the obstacle, it controls the opening of the hinged door to a predetermined angle within the range of the maximum door opening angle at which the door does not come into contact with the obstacle, and at which the imaging unit does not capture an image. The specified part is, The three-dimensional position of the obstacle relative to the vehicle is re-determined using the corresponding points of the obstacle determined based on the plurality of images captured by the imaging unit at the predetermined angle viewpoint and other door opening angles viewpoints, and the door opening angles associated with each of the plurality of images. Information processing device.

2. The specified unit then re-identifies the three-dimensional position of the obstacle based on the error between the corresponding point and the door opening angle after removal. The determination unit uses the three-dimensional position of the obstacle, which has been identified again by the identification unit, and the door information to determine the maximum door opening angle at which the hinged door does not come into contact with the obstacle. The information processing apparatus according to claim 1.

3. The control unit, The imaging unit determines the door opening angle when capturing the image, Control is performed to open the hinged door to the determined door opening angle. The information processing apparatus according to claim 1.

4. The determination unit is Using the three-dimensional position of the obstacle identified again by the specified unit and the door information, the maximum door opening angle at which the hinged door does not come into contact with the obstacle is determined again. The information processing apparatus according to claim 1.

5. The control unit, The determination unit determines the predetermined angle according to the maximum door opening angle that does not come into contact with the obstacle, as determined by the determination unit. The information processing apparatus according to claim 4.

6. The system includes a receiving unit that receives input for the number of times the determination unit determines the maximum door opening angle at which the hinged door does not come into contact with the obstacle. The information processing apparatus according to claim 4.

7. The control unit, Based on the detection result of a distance measuring sensor installed on the hinged door, control is performed to prevent the opening of the hinged door. The information processing apparatus according to claim 1.

8. The system acquires an image captured by an imaging unit installed on the vehicle's hinged door, and the door opening angle, which indicates the angle at which the hinged door was opened from a closed state when the image was captured. Based on the multiple images captured by the imaging unit from multiple viewpoints with different door opening angles, the corresponding points of obstacles present around the hinged door, and the door opening angles associated with each of the multiple images, are used to identify the error in the three-dimensional position of the obstacle and the door opening angle relative to the vehicle. A portion of the aforementioned corresponding points is removed based on the error in the door opening angle, and the maximum door opening angle at which the hinged door does not come into contact with the obstacle is determined using the three-dimensional position of the obstacle identified based on the removed corresponding points and door information relating to the shape and dimensions of the hinged door. Control is performed to open the hinged door to the maximum door opening angle that does not come into contact with the determined obstacle. After determining the maximum door opening angle that does not come into contact with the obstacle, control is performed to open the hinged door to a predetermined angle within the range of the maximum door opening angle that does not come into contact with the obstacle, where the image is not captured by the imaging unit. The three-dimensional position of the obstacle relative to the vehicle is re-determined using the corresponding points of the obstacle determined based on the plurality of images captured by the imaging unit at the predetermined angle viewpoint and other door opening angles viewpoints, and the door opening angles associated with each of the plurality of images. An information processing method in which a computer performs the processing.

9. On the computer, The system acquires an image captured by an imaging unit installed on the vehicle's hinged door, and the door opening angle, which indicates the angle at which the hinged door was opened from a closed state when the image was captured. Based on the multiple images captured by the imaging unit from multiple viewpoints with different door opening angles, the corresponding points of obstacles present around the hinged door, and the door opening angles associated with each of the multiple images, are used to identify the error in the three-dimensional position of the obstacle and the door opening angle relative to the vehicle. A portion of the aforementioned corresponding points is removed based on the error in the door opening angle, and the maximum door opening angle at which the hinged door does not come into contact with the obstacle is determined using the three-dimensional position of the obstacle identified based on the removed corresponding points and door information relating to the shape and dimensions of the hinged door. Control is performed to open the hinged door to the maximum door opening angle that does not come into contact with the determined obstacle. After determining the maximum door opening angle that does not come into contact with the obstacle, control is performed to open the hinged door to a predetermined angle within the range of the maximum door opening angle that does not come into contact with the obstacle, where the image is not captured by the imaging unit. The three-dimensional position of the obstacle relative to the vehicle is re-determined using the corresponding points of the obstacle determined based on the plurality of images captured by the imaging unit at the predetermined angle viewpoint and other door opening angles viewpoints, and the door opening angles associated with each of the plurality of images. An information processing program used to execute a process.

Images