Object position estimation method, and vehicle object position estimation device

Abstract

Accurately estimating the three-dimensional position of an object reflected on a mirror surface, such as a convex mirror. [Solution] The object position estimation method according to the embodiment is an object position estimation method for estimating the three-dimensional position of an object reflected in a convex mirror, comprising the steps of: obtaining the three-dimensional positions of a plurality of points from a sensor that measures a plurality of points on the outer edge of a convex mirror; determining the radius of curvature of the convex mirror; determining the three-dimensional position of the center of curvature of the convex mirror based on the three-dimensional positions of the plurality of points and the radius of curvature; obtaining distance information between the sensor and the object from the sensor based on the reflection of light rays or electromagnetic waves emitted by the sensor from the convex mirror and from the object; obtaining angle information of the reflection point on the convex mirror with respect to the sensor from the sensor; and estimating the three-dimensional position of the object based on the radius of curvature, the three-dimensional position of the center of curvature, the distance information and the angle information.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to an object position estimation method and an object position estimation device for a vehicle. [Background technology]

[0002] Conventionally, there are technologies that use sensors such as LiDAR (Light Detection and Ranging) to estimate the position of oneself and the position information of surrounding objects. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Yuto Utsumi, Shingo Kitagawa, Iori Yanokura, Kei Okada, Masayuki Inaba, "Object Perception in Blind Spots Using Mirrors Based on Depth Prediction by CNN," 2019 Annual Conference of the Japanese Society for Artificial Intelligence (33rd), 1L2-J-11-02, The Japanese Society for Artificial Intelligence, June 4-7, 2019, DOI: https: / / doi.org / 10.11517 / pjsai.JSAI2019.0_1L2J1102 [Non-Patent Document 2] Tomoo Taniguchi, Fumitaka Nakaizumi, "Three-Dimensional Position Estimation of Out-of-Field Objects Using Mirror Images," 24th Annual Meeting of the Virtual Reality Society of Japan, 5C-10, September 2019, https: / / conference.vrsj.org / ac2019 / program / common / doc / pdf / 5C-10.pdf [Overview of the project] [Problems that the invention aims to solve]

[0004] For example, even if a vehicle or other object is reflected in a convex mirror installed at an intersection with poor visibility, the sensor may not be able to recognize the object's position, or it may mistakenly identify the position of a virtual image as the object's position.

[0005] In one aspect, the present invention aims to provide an object position estimation method and a vehicle object position estimation device capable of accurately estimating the three-dimensional position of an object reflected on a mirror surface such as a convex mirror. [Means for solving the problem]

[0006] In one embodiment, the object position estimation method disclosed in this application is an object position estimation method for estimating the three-dimensional position of an object reflected in a convex mirror, comprising: a step of obtaining the three-dimensional positions of a plurality of points from a sensor that measures a plurality of points on the outer edge of the convex mirror; a step of determining the radius of curvature of the convex mirror; a step of determining the three-dimensional position of the center of curvature of the convex mirror based on the three-dimensional positions of the plurality of points and the radius of curvature; a step of obtaining distance information between the sensor and the object from the sensor based on the reflection of light rays or electromagnetic waves emitted by the sensor from the convex mirror and the reflection from the object; a step of obtaining angle information from the sensor of the reflection point on the convex mirror with respect to the sensor; and a step of estimating the three-dimensional position of the object based on the radius of curvature, the three-dimensional position of the center of curvature, the distance information and the angle information. [Effects of the Invention]

[0007] According to one embodiment of the object position estimation method disclosed in this application, the three-dimensional position of an object reflected on a mirror surface such as a convex mirror can be estimated with high accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of an information processing system including a vehicle object position estimation device according to an embodiment. [Figure 2] Figure 2 shows an example of the hardware configuration of a vehicle object position estimation device according to an embodiment. [Figure 3] Figure 3 is a diagram illustrating an example of the functional configuration of a distance measuring device and a processor mounted on a vehicle object position estimation device, according to an embodiment. [Figure 4]Figure 4 shows an example of a LiDAR field of view relating to an embodiment, illustrating multiple points on the outer edges of a convex mirror and a plane mirror. [Figure 5] Figure 5 shows an example of an overview of the data acquired by the acquisition unit according to the embodiment. [Figure 6] Figure 6 shows an example of a radius of curvature correspondence table relating to an embodiment. [Figure 7] Figure 7 is an explanatory diagram illustrating the determination of the center of curvature in an embodiment. [Figure 8] Figure 8 is an explanatory diagram illustrating an example of an embodiment for calculating the angle POQ and the length of the distance from the distance measuring device to the center of curvature. [Figure 9] Figure 9 shows an example of the positional relationship between the feature points of object OB, the virtual image, and the distance measuring device in an embodiment in which the position of the distance measuring device is taken as the origin O, the line segment OQ is defined as a new X' axis in a plane including the reflection point and the center of curvature, and the line perpendicular to the X' axis and passing through the origin O is defined as the Y' axis. [Figure 10] Figure 10 is a diagram illustrating the parameters required when converting the coordinates (x, y) of a feature point S in the X'Y' plane to coordinates in the XYZ space, according to an embodiment. [Figure 11] Figure 11 is an explanatory diagram illustrating an embodiment that shows an overview of how to convert the coordinates (x, y) of a feature point S in the X'Y' plane to coordinates in the XYZ space. [Figure 12] Figure 12 is an explanatory diagram showing an example of a planar mirror processing according to an embodiment. [Figure 13] Figure 13 is a flowchart showing an example of the estimation process procedure according to the embodiment. [Figure 14] Figure 14 shows an example of a rectangular mirror and a double-sided mirror, relating to a third modification of the embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of the object position estimation method and vehicle object position estimation device disclosed in this application will be described in detail with reference to the drawings. Note that the following embodiments do not limit the disclosed technology. Furthermore, each embodiment can be appropriately combined as long as the processing content is not inconsistent.

[0010] (Embodiment) Figure 1 shows an example of an information processing system 1 including the vehicle object position estimation device 100 of this embodiment. The information processing system 1 shown in Figure 1 is mounted on a mobile body 200, such as an automobile. The mobile body 200 on which the information processing system 1 is mounted is a movable object. The mobile body 200 is, for example, a vehicle, an airworthy object (manned aircraft, unmanned aircraft (e.g., UAV (Unmanned Aerial Vehicle), drone)), a transport robot, a ship, etc.). The mobile body 200 is also, for example, a mobile body that moves through human driving operation, or a mobile body that can move automatically (autonomously) without human driving operation.

[0011] Furthermore, the various processes and functions realized by the vehicle object position estimation device 100 may be performed by a server on the cloud or the like using the position information of the object obtained by distance measurement using the distance measuring device 19. The object to be distance measured is an object different from the moving body 200, and may be a moving body such as another vehicle or a person, or a stationary object such as a roadside structure or fallen object on the road. In addition, the moving body 200 may be equipped with an imaging device such as an optical camera (for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor). In this case, the vehicle object position estimation device 100 may further use the image information (image data) obtained from the imaging device to perform the various processes and functions realized by the vehicle object position estimation device 100. In this embodiment, the case where the moving body 200 is a vehicle will be described as an example. The vehicle may be, for example, a two-wheeled vehicle, a three-wheeled vehicle, or a four-wheeled vehicle. In this embodiment, the case where the vehicle is a four-wheeled vehicle will be described as an example. Various objects, including the moving body on which the vehicle object position estimation device 100 is mounted, may be equipped with various sensors for detecting the position and speed of the object.

[0012] As shown in Figure 1, the information processing system 1 includes a vehicle object position estimation device 100 and a distance measuring device 19. The distance measuring device 19 has multiple sensors 19A, 19B, 19C, 19D, and 19E. In Figure 1, for the sake of clarity, the information processing system 1 is superimposed on an image of the moving body 200 viewed from above. However, in reality, the vehicle object position estimation device 100 is mounted on a control board or the like attached to the moving body 200.

[0013] Multiple sensors 19A, 19B, 19C, 19D, and 19E are provided, for example, in front of the moving body 200 in the direction of movement D, on the left side of the moving body 200 in the direction of movement D, on the right side of the moving body 200 in the direction of movement D, at the rear of the moving body 200, and inside the vehicle of the moving body 200. Note that the number of multiple sensors 19A, 19B, 19C, 19D, and 19E is not limited to five, and they may be placed at various ends of the moving body 200. Each of the multiple sensors 19A, 19B, 19C, 19D, and 19E may function as an independent distance measuring device 19, or they may be integrated and referred to as a distance measuring device 19.

[0014] As shown in Figure 1, the distance measuring device 20 is electrically connected to the vehicle object position estimation device 100. However, the distance measuring device 19 and the vehicle object position estimation device 100 are not limited to being electrically connected directly as shown in Figure 1; they may also be connected wirelessly.

[0015] The distance measuring device 19 is implemented, for example, by a LiDAR (Light Detection And Ranging) sensor. The distance measuring device (LiDAR sensor) 19 acquires positional information of objects surrounding the moving body 200. Surrounding objects correspond to other moving bodies or stationary objects (hereinafter collectively referred to simply as "objects") that are different from the moving body 200. Note that the distance measuring device 19 is not limited to LiDAR and may be implemented by radar such as an ultrasonic sensor. Also, the installation location of the distance measuring device 19A in front of the moving direction D of the moving body 200 is not limited to inside the vehicle of the moving body 200 as shown in Figure 1, but may be installed outside the vehicle of the moving body 200. The detection range of the distance measured by the distance measuring device 19 is set by, for example, the output of the laser beam, the detection accuracy of the laser beam, etc. The detection range of the distance measured by the distance measuring device 19 may also be called the measurement range, measurement target range, etc.

[0016] Figure 2 shows an example of the hardware configuration of the vehicle object position estimation device 100. The vehicle object position estimation device 100 includes, for example, a processor 101, memory 103, storage device 105, I / F (Interface) 107, input device 109, output device 111, communication device 113, and bus 115.

[0017] The processor 101 is a computing device such as a CPU (Central Processing Unit) that performs predetermined processing by executing a program stored in a storage medium such as a storage device 105. The memory 103 includes, for example, volatile memory such as RAM (Random Access Memory) used as a work area for the processor 101, and non-volatile memory such as ROM (Read Only Memory) that stores a program for starting the processor 101. The storage device 105 is a large-capacity, non-volatile storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive). The I / F 107 includes various interfaces for connecting external devices to the vehicle object position estimation device 100.

[0018] The input device 109 includes various devices that accept external input (e.g., keyboard, touch panel, pointing device, microphone, switch, button, or sensor). The output device 111 includes various devices that perform external output (e.g., display, speaker, indicator).

[0019] The communication device 113 includes various communication devices for communicating with other devices via a wired or wireless network. The I / F 107 or output device 111 is connected to a control circuit that controls, for example, the mobile unit 200. The bus 115 is connected to each of the above components and transmits, for example, address signals, data signals, and various control signals.

[0020] Figure 3 shows an example of the functional configuration of the processor 101 mounted on the vehicle object position estimation device 100 and the functional configuration of the distance measuring device 19. As shown in Figures 2 and 3, the processor 101 is connected to the distance measuring device 19 via the I / F 107, input device 109, or communication device 113. The distance measuring device 19 has a laser light emitting unit 21, a laser light receiving unit 23, and a measurement unit 25. The distance measuring device 19 further has a control circuit that controls the functions realized by the laser light emitting unit 21 and the measurement unit 25.

[0021] The laser emission unit 21 adjusts the frequency of laser generation and the LiDAR field of view under the control of the control circuit. After this adjustment, the laser emission unit 21 emits a laser into the LiDAR field of view under the control of the control circuit. At this time, the laser emission unit 21 outputs information such as the angle and time of laser emission to the measurement unit 25. The laser light receiving unit 23 receives the laser reflected by an object in the LiDAR field of view. At this time, the laser light receiving unit 23 outputs information such as the angle and time of laser reception to the measurement unit 25. Since the laser emission unit 21 and the laser light receiving unit 23 are based on those mounted in known LiDARs, a detailed explanation is omitted.

[0022] The measurement unit 25 measures the three-dimensional position of an object in the LiDAR field of view (hereinafter referred to as the three-dimensional position) based on the laser emission angle and time and the laser reception angle and time. The three-dimensional position of an object may also be referred to as the object's position information. Since the measurement of the three-dimensional position is based on the known TOF (Time-of-Flight) format, an explanation will be omitted. In the above explanation, the distance measuring device 19 was described using LiDAR, but it is not limited to this. For example, the distance measuring device 19 may be realized by various ToF (Time of Flight) radars or by a distance image sensor using a camera.

[0023] If the mobile body 200 is a four-wheeled vehicle, the processor 101 is implemented, for example, by an electronic control unit (ECU). As shown in Figure 3, the processor (ECU) 101 includes an acquisition unit 11, a determination unit 12, a curvature radius determination unit 13, a curvature center determination unit 14, an estimation unit 15, and a mirror surface position determination unit 16. The functions performed by the acquisition unit 11, the determination unit 12, the curvature radius determination unit 13, the curvature center determination unit 14, the estimation unit 15, and the mirror surface position determination unit 16 are executed by the processing circuit in the processor 101.

[0024] The acquisition unit 11 acquires the three-dimensional positions of multiple objects in the LiDAR field of view from the distance measuring device 19. For example, the acquisition unit 11 acquires the three-dimensional positions of multiple points on the outer edge of a road mirror (e.g., a convex mirror and / or a plane mirror) in the LiDAR field of view. Specifically, the acquisition unit 11 acquires the three-dimensional positions of multiple points on the outer edge of the convex mirror in the LiDAR field of view from the measurement unit 25. The multiple points on the outer edge of the convex mirror are, for example, three points. Note that the number of multiple points is not limited to three, but may be four or more. If the convex mirror is rectangular, the multiple points on the outer edge of the convex mirror may correspond to the vertices of the rectangle.

[0025] Figure 4 shows an example of multiple points OEP on the outer edge of a convex mirror CM and a plane mirror VCN in the LiDAR field of view LFV. As shown in Figure 4, the acquisition unit 11 acquires the three-dimensional positions of multiple points (10 points in Figure 4) on the outer edge of the convex mirror CM, which is a road reflector RF, from the measurement unit 25. Also, as shown in Figure 4, the acquisition unit 11 acquires the three-dimensional positions of multiple points (4 points in Figure 4) on the outer edge of the convex mirror CM, which is a vehicle confirmation mirror (also called a safety mirror) VCM installed at the back of the entrance of a mechanical multi-story parking garage, from the measurement unit 25. Both convex mirrors CM reflect virtual images VI of objects. The multiple points on the outer edge of the road reflector RF are shown as 10 points in Figure 4, and the multiple points on the outer edge of the vehicle confirmation mirror VCM are shown as 4 points in Figure 4, but for the sake of simplicity, we will explain them as 3 points below. In the above explanation, the reflector is described as a convex mirror, but it is not limited to this and may be a plane mirror. While many vehicle inspection mirrors installed at the back of the entrance to mechanical multi-story parking garages are flat mirrors, the following explanation will focus on mirrors that are convex mirrors (CM) in order to make the explanation more concrete.

[0026] Furthermore, the acquisition unit 11 acquires distance information between the distance measuring device 19 and the object from the sensor (distance measuring device 19) based on the reflection of light rays (laser) or electromagnetic waves emitted by the sensor corresponding to the distance measuring device 19 on the convex mirror CM and the reflection of light rays or electromagnetic waves from the object reflected on the convex mirror CM. Specifically, with respect to the object reflected on the convex mirror CM, the acquisition unit 11 acquires the sum of the distance from the distance measuring device 19 to the position (reflection point) indicating the object on the convex mirror and the distance from the reflection point to the object as distance information (optical path distance) from the measurement unit 25.

[0027] Furthermore, the acquisition unit 11 acquires angle information from the distance measuring device 19 regarding the reflection point of the convex mirror CM for the above reflection. Specifically, the acquisition unit 11 acquires angle information from the measurement unit 25 that includes the angle (elevation angle) between the optical path connecting the distance measuring device 19 and the reflection point (hereinafter referred to as the pre-reflection optical path) and the plane containing the direction of movement of the moving body 200 (the horizontal plane if the direction of movement is in the horizontal plane), and the angle (azimuth angle) between the pre-reflection optical path and the direction of travel of the moving body 200. In other words, the acquisition unit 11 acquires the elevation angle and azimuth angle of the pre-reflection optical path as angle information from the measurement unit 25. The acquisition unit 11 stores the three-dimensional positions of multiple points on the outer edge acquired from the measurement unit 25, along with the distance information and angle information, in the memory 103 or storage device 105.

[0028] Figure 5 shows an example of an overview of the data acquired by the acquisition unit 11. In Figure 5, a plane is shown that includes the distance measuring device 19 and the object OB reflected in the mirror surface MS. The mirror surface MS shown in Figure 5 corresponds to a circular arc obtained by cutting a sphere formed by the curvature center CC and the radius of curvature RC with this plane. At this time, the object OB reflected in the mirror surface MS forms an optical virtual image VI. The optical path distance OPD in Figure 5 corresponds to the sum of the distance between the reflection point P and the distance measuring device 19 on the tangent line (tangent plane in three dimensions) TL tangent to the mirror surface MS, and the distance between the reflection point P and the object OB. Also, as shown in Figure 5, the angle information has an angle EA that indicates the angle in the direction from the distance measuring device 19 toward the reflection point P, with the distance measuring device 19 as the reference. In addition, in Figure 5, the pre-reflection optical path corresponds to the optical path from the distance measuring device 19 toward the reflection point P.

[0029] The determination unit 12 determines whether the depth of the point cloud acquired by the distance measuring device 19 within the region enclosed by the three-dimensional positions of multiple points OEP on the outer edge of the convex mirror CM is greater than any of the three-dimensional positions of those multiple points. If the three-dimensional position of the point cloud located inside the region is greater than any of the three-dimensional positions of the multiple points OEP on the outer edge of the convex mirror, that is, if the optical path distance for the point cloud located inside the region is greater than the distance for the multiple points OEP on the outer edge of the convex mirror, the determination unit 12 determines that a mirror surface exists in that region.

[0030] The determination of the presence or absence of a mirror surface is not limited to the above process. For example, the presence or absence of a mirror surface may be determined within the acquisition range of the image data by inputting image data acquired by an optical camera mounted on the mobile body 200 or the distance measuring device 19 into a trained model capable of determining the presence or absence of a mirror surface. This trained model can be implemented, for example, by a neural network that has been pre-trained using a known method.

[0031] Furthermore, the determination unit 12 uses the angle information acquired by the acquisition unit 11 to determine whether the determined mirror surface is a convex mirror or a planar mirror. For example, if the elevation angle in the angle information is positive, the determination unit 12 determines that the mirror surface is a convex mirror. Also, if the elevation angle in the angle information is approximately zero, that is, if the mirror surface is located at a height approximately the same as the height from the horizontal plane of the distance measuring device 19, the determination unit 12 determines that the mirror surface is a planar mirror. This is based on the fact that planar mirrors, which are used as vehicle inspection mirrors, are often installed at a height approximately the same as the height of the vehicle from the road surface.

[0032] The radius of curvature determination unit 13 determines the radius of curvature RC of the convex mirror CM. For example, the radius of curvature determination unit 13 estimates the shape and size of the convex mirror CM based on the three-dimensional positions of multiple points OEP on the outer edge of the convex mirror CM. Next, the radius of curvature determination unit 13 reads a correspondence table (hereinafter referred to as the radius of curvature correspondence table) for multiple convex mirrors from the memory 103 or storage device 105, corresponding to the shape and size of the convex mirrors. The radius of curvature correspondence table may also include the material of the back plate and the material of the hood of the convex mirror. The radius of curvature correspondence table corresponds to the product standards for road reflectors using convex mirrors.

[0033] Figure 6 shows an example of a radius of curvature correspondence table LUT. The difference in letters in the part number indicates the difference in whether the outer edge of the convex mirror is circular or rectangular, i.e., the difference in shape. The difference in numbers in the part number indicates the difference in the size of the convex mirror. Figure 6 is an example and may further include elements of radius of curvature for other shapes and sizes.

[0034] The radius of curvature determination unit 13 collates the estimated shape and size of the convex mirror CM with a radius of curvature correspondence table LUT having the product specifications of the convex mirror. Thereby, the radius of curvature determination unit 13 identifies the product number corresponding to the estimated shape and size of the convex mirror CM, and determines the radius of curvature in the identified product number as the radius of curvature of the convex mirror CM. For example, when the convex mirror CM is estimated to be circular with a diameter near 600 millimeters, the road reflector is identified as the product with the product number A-1 from the radius of curvature correspondence table LUT, and the radius of curvature of the convex mirror is determined to be 2200 millimeters.

[0035] The center of curvature determination unit 14 determines the three-dimensional position of the center of curvature CC of the convex mirror CM based on the three-dimensional positions of a plurality of points OEP on the outer edge and the determined radius of curvature. For example, the center of curvature determination unit 14 sets up an equation for the distance between the center of curvature CC and each of the three-dimensional positions of a plurality (three or more) of points OEP on the outer edge, and solves the equation to determine the three-dimensional position of the center of curvature CC. Hereinafter, for the sake of specific explanation, it is described that the plurality of points on the outer edge are three points.

[0036] FIG. 7 is an explanatory diagram for explaining the determination of the center of curvature CC. In FIG. 7, with the position of the distance measuring device 19 as the origin O, a three-dimensional space (XYZ space) is defined by three orthogonal axes (X-axis, Y-axis, Z-axis). The three orthogonal axes are, for example, as shown in FIG. 7, with the traveling direction of the moving body 200 as the X-axis, in the plane including the traveling direction X-axis and the origin O, perpendicular to the traveling direction, the direction toward the left hand side of the traveling direction as the Y-axis, and the upward (zenith direction) as the Z-axis. When the XY plane is the horizontal plane, the Z direction corresponds to the vertically upward direction.

[0037] Also, the coordinates of the three-dimensional positions of the three points on the outer edge of the convex mirror CM are U1(X U1 , Y U1 , Z U1 ), U2(X U2 , Y U2 , Z U2 ), U3(X U3 , Y U3 , Z U3Let's assume that ). To make the explanation easier to understand, known values ​​will be written in uppercase letters, and unknown values ​​in lowercase letters.

[0038] In Figure 7, R represents the radius of curvature determined by the radius of curvature determination unit 13. Furthermore, the coordinates of the center of curvature CC, which indicates the center of the sphere in Figure 7, are given by Q(x Q ,y Q ,z Q Let ). In this case, the distance (radius of curvature R) between the center of curvature Q and each of the three points U1, U2, and U3 on the outer edge is expressed by the following equation (1).

[0039] {(x Q -X U1 ) 2 +(y Q -Y U1 ) 2 +(z Q -Z U1 ) 2} 1 / 2 =R {(x Q -X U2 ) 2 +(y Q -Y U2 ) 2 +(z Q -Z U2 ) 2} 1 / 2 =R {(x Q -X U3 ) 2 +(y Q -Y U3 ) 2 +(z Q -Z U3 ) 2} 1 / 2 =R ···(1)

[0040] The curvature center determination unit 14 uses the above equation (1) to determine the coordinates (x) of the curvature center Q. Q ,y Q ,z Q The three-dimensional position of the center of curvature Q is determined by solving for (x). Specifically, the three equations above are the coordinates of the center of curvature Q. Q ,y Q,z Q Since it is a quadratic equation for (x), two sets of coordinates that are solutions can be found, but Q 2 +y Q 2 +z Q 2 ) 1 / 2 The larger of the two calculated distances for OQ is the coordinate (x) of the center of curvature Q. Q ,y Q ,z Q ) is determined as follows. This is because, even if the mirror surface is a concave mirror, it can be expressed by equation (1), but a larger distance OQ indicates a convex mirror, and a smaller distance indicates a concave mirror. If it is possible to measure four or more points on the outer edge of the convex mirror CM, the curvature center determination unit 14 selects three points from the four or more points and solves the three equations obtained from the selected three points to determine the coordinates (x) of the curvature center Q. Q ,y Q ,z Q Alternatively, you may determine the coordinates Q(x) by setting up four or more equations similar to equation (1) above, and the curvature center determination unit 14 selects multiple sets of three different equations from the four or more equations, and then determines multiple coordinates Q(x) Q ,y Q ,z Q ) calculates multiple coordinates Q(x Q ,y Q ,z Q The 3D position of the center of curvature Q may be determined by performing statistical processing such as averaging on the coordinates Q(x Q ,y Q ,z Q This can improve the accuracy of the result.

[0041] The estimation unit 15 determines the radius of curvature R and the three-dimensional position Q(x) of the center of curvature CC. Q ,y Q ,z Q Based on the distance information acquired by the acquisition unit 11 and the angle information acquired by the acquisition unit 11, the three-dimensional position of object OB reflected in the convex mirror CM is estimated. Below, an example of the calculation for estimating the three-dimensional position of object OB will be explained using Figures 8 to 11. The process of estimating the three-dimensional position of object OB in the convex mirror CM will be referred to as the convex mirror process below.

[0042] FIG. 8 is an explanatory diagram showing an example of an explanation for calculating an angle ∠POQ formed by a reflection point P corresponding to one of the feature points of a virtual image VI of an object OB on a mirror surface MS, a position (origin O) of a distance measuring device 19, and a center of curvature Q, and a length W of a distance OQ from the distance measuring device 19 to the center of curvature Q. In FIG. 8, an angle (elevation angle) between the XY plane and an optical path OP before reflection and an azimuth angle of the optical path OP before reflection with respect to the moving direction X of the moving body 200 are acquired by an acquisition unit 11 as angle information. The elevation angle of the optical path OP before reflection may be referred to as a vertical angle A V and the azimuth angle of the optical path OP before reflection may be referred to as a horizontal angle A H . The three-dimensional position of the reflection point P in FIG. 8 cannot be measured by total reflection on the mirror surface MS.

[0043] An estimation unit 15 calculates angles (horizontal angle B Q , vertical angle B Q ) of Q shown in FIG. 8 by the following formula (2) using coordinates (x Q ) of the center of curvature Q determined by a center of curvature determination unit 14. Here, the horizontal angle B H of Q corresponds to an angle between a projection line obtained by projecting a line connecting the distance measuring device 19 and the center of curvature Q (hereinafter referred to as a center of curvature direction line) onto the XY plane (a plane including the traveling direction X) and the traveling direction X, as shown in FIG. 8. Further, the vertical angle B V of the center of curvature Q corresponds to an angle between the center of curvature direction line and the projection line, as shown in FIG. 8. H V

[0044] B H = tan -1 (y Q / x Q ) B V = tan -1 {z Q / (x Q 2 +y Q 2 ) 1 / 2} ···(2)

[0045] ​​Next, the estimation unit 15 calculates the ∠POQ = T and the length W of OQ shown in FIG. 8 using the following formula (3).

[0046] T = cos -1 {cos(A H -B H )·cos(A V -B V )} W=(x Q 2 +y Q 2 +z Q 2 ) 1 / 2 ···(3)

[0047] FIG. 9 shows an example of the positional relationship among the feature point S (a point indicating the feature of another moving object (such as a light or a bumper)), the virtual image VI, and the distance measuring device 19 in a plane including the reflection point P and the center of curvature Q with the position of the distance measuring device 19 as the origin O. As shown in FIG. 9, the feature point S of the object OB corresponding to the reflection point P is located on the X'Y' plane. As shown in FIG. 9, the optical path distance L from the distance measuring device 19 (origin O) to the feature point S of the object OB is measured by the distance measuring device 19 (ToF sensor) and acquired by the acquisition unit 11. As shown in FIG. 9, let the length (distance) of OP be m and ∠OQP be a.

[0048] The dashed line TP shown in FIG. 9 indicates the tangent plane at the reflection point P of the mirror surface MS. Also, the dashed line NL shown in FIG. 9 indicates the normal line of the tangent plane at the reflection point P. As shown in FIG. 9, the coordinates of the reflection point P are (m·cos(T), m·sin(T)). Also, as shown in FIG. 9, the coordinates (x, y) of the feature point S of the object OB are represented by the following formula (4).

[0049] x = m·cos(T)-(L - m)cos(2a + T) y = m·sin(T)+(L - m)sin(2a + T) ···(4)

[0050] If the length (distance) m of OP and ∠OQP=a can be determined from equation (4) and the coordinates of the reflection point P, the estimation unit 15 can estimate the coordinates (x, y) of the characteristic point S of object OB. First, the estimation unit 15 applies the law of cosines to the triangle OPQ shown in Figure 9 and obtains equation R 2 =m 2 +W 2 The length of OP, m, is calculated by solving -2m·W·cos(T) for m. Next, the estimation unit 15 calculates the angle a of ∠OQP by substituting the value of m into the equation m / sin(a)=R / sin(T), which is obtained by applying the Law of Sines to the triangle OPQ shown in Figure 9. From the above, the estimation unit 15 calculates the coordinates (x, y) of the feature point S of object OB by substituting the calculated length m of OP and ∠OQP=a into equation (4). Since the calculated coordinates (x, y) of the feature point S are on the X'Y' plane, the estimation unit 15 converts the coordinates (x, y) of the feature point S into coordinates in XYZ space.

[0051] Figure 10 is a diagram illustrating the parameters required when converting the coordinates (x, y) of a feature point S in the X'Y' plane to coordinates in XYZ space. As shown in Figure 10, the length of OS is K and ∠SOQ = D. In this case, the length K of OS and the angle D are expressed by the following equation (5) using the coordinates (x, y) of the feature point S in the X'Y' plane. The estimation unit 15 calculates the length K of OS and ∠SOQ = D by substituting the coordinates (x, y) of the feature point S into equation (5).

[0052] K=(x 2 +y 2 ) 1 / 2 D=tan -1 (y / x) ···(5)

[0053] Figure 11 is an explanatory diagram illustrating the general process of converting the coordinates (x, y) of a feature point S in the X'Y' plane to coordinates in XYZ space. The estimation unit 15 calculates ∠SOQ=D and the horizontal angle B of Q. H And the perpendicular angle B of Q V And the vertical angle A of the pre-reflection optical path OP. V And the horizontal angle A of the pre-reflection optical path OP HUsing the following equation (6), the angle of S shown in Figure 11 (horizontal angle c) H and vertical angle c V ) is calculated. Here, the horizontal angle c of the feature point S in XYZ space is calculated. H As shown in Figure 11, this corresponds to the angle between the projection line obtained by projecting the line connecting the distance measuring device 19 and object OB (hereinafter referred to as the object direction line) onto the XY plane (the plane containing the direction of travel X) and the direction of travel X. Also, the vertical angle c of the feature point S in XYZ space. V This corresponds to the angle between the object direction line and the projection line, as shown in Figure 11.

[0054] cos(D) = cos(c) H -B H )·cos(c V -B V ) cos(DT) = cos(c H -A H )·cos(c V -A V ) ···(6)

[0055] The estimation unit 15 calculates the angle (horizontal angle c) of the feature point S in the XYZ space. H and vertical angle c V Using the following equation (7) with the length of OS, the 3D position S(x) of the feature point S in XYZ space is given by the following equation: s , y s , z s ) calculate.

[0056] x S =K·cos(c v )·cos(c H ) y S =K·cos(c v )·sin(c H ) z S =K·sin(c v ) ···(7)

[0057] Based on the above calculations, the estimation unit 15 determines the three-dimensional position S(x) of object OB in the XYZ space corresponding to real space. s , ys , z s The 3D position S(x) of object OB is estimated. s , y s , z s The estimation of ( ) is not limited to the calculation method described above and may be achieved by other methods, other calculation means, etc. For example, the estimation unit 15 uses the three-dimensional positions of multiple points on the outer edge of the convex mirror CM, distance information between the distance measuring device 19 and the object OB, and angular information regarding the reflection point for reflection on the mirror surface MS of the convex mirror CM (elevation angle of the pre-reflection optical path OP (vertical angle A) V ) and the azimuth angle (horizontal angle A) of the reflected light path OP H The 3D position S(x) of object OB is input into a pre-trained neural network, and the output from the neural network is used to determine the 3D position S(x) of object OB. s , y s , z s ) may be estimated.

[0058] Using a pre-trained neural network, the 3D position S(x) of object OB is calculated. s , y s , z s When estimating the curvature radius determination unit 13 and the curvature center determination unit 14 are unnecessary. Furthermore, the neural network used in the estimation unit 15 is trained by inputting the distance information and angle information of each of several known objects, using the 3D positions of each of these known objects as training data. As for the training method, known methods such as backpropagation can be used, so an explanation is omitted. The neural network is implemented by machine learning models such as CNN (Convolutional Neural Network) and DNN (Deep Neural Network).

[0059] Next, we will explain the case of a plane mirror. The mirror position determination unit 16 determines the mirror position of the plane mirror based on the three-dimensional positions of multiple points on the outer edge of the plane mirror VCM. Determining the position of the mirror surface involves, for example, determining the equation of the plane relating to the mirror surface in XYZ space. For example, the mirror position determination unit 16 determines the three-dimensional positions of three points on the outer edge of the plane mirror VCM, U1(X U1 ,Y U1 ,Z U1 ), U2(X U2 ,Y U2 ,Z U2 ), U3(X U3 ,Y U3 ,Z U3 By substituting these into the plane equation (α·x+β·y+γ·z+δ=0), we obtain, for example, the following three plane equations (8). Here, the vectors (α, β, γ) correspond to the normal vector of the mirror surface MS.

[0060] α·X U1 +β·Y U1 +γ·Z U1 +δ=0 α·X U2 +β·Y U2 +γ·Z U2 +δ=0 α·X U3 +β·Y U3 +γ·Z U3 +δ=0 ···(8)

[0061] The four coefficients (α, β, γ, δ) in equation (8) are parameters that define the plane equation. The mirror position determination unit 16 determines the plane equation that indicates the position of the mirror by solving the three equations in equation (8) simultaneously. The plane equation of the mirror is not limited to the above; for example, vectors U1U2 and U1U3 are calculated, the normal vector of the mirror is calculated by the cross product of vectors U1U2 and U1U3, and the normal vector and the coordinates (X) of U1 are calculated. U1 ,Y U1 ,Z U1 ) may be determined using the following:

[0062] The process of estimating the three-dimensional position of object OB in a plane mirror will be referred to as the plane mirror process below. In the plane mirror process, the estimation unit 15 estimates the three-dimensional position of object OB based on the three-dimensional position of the virtual image VI and the mirror surface position. Specifically, the estimation unit 15 calculates the three-dimensional position of object OB using the three-dimensional position of the virtual image VI measured by the distance measuring device 19 and the plane equation of the mirror surface. To make the explanation more concrete, the plane mirror process for determining the three-dimensional position of object OB in a plane mirror will be described below using Figure 12.

[0063] Figure 12 is an explanatory diagram showing an example of the plane mirror processing. As shown in Figure 12, in reflection with a plane mirror, the distance from the distance measuring device 19 to the virtual image V is equal to the optical path distance from the distance measuring device 19 to the object OB via the reflection point P. The estimation unit 15 calculates the feature points V(X) of the virtual image of object OB by geometric calculation using the distance information and angle information acquired by the acquisition unit 11. V ,Y V ,Z V ) is calculated. At this time, the coordinates S(x,y,z) of object OB are expressed by the following equation (9) using the parameter t and the parameters (α,β,γ) relating to the definition of the plane equation.

[0064] (x,y,z)=(X V ,Y V ,Z V ) + 2·t·(α,β,γ) =( X V +2·t·α,Y V +2·t·β,Z V +2·t·γ) ...(9)

[0065] At this time, as shown in Figure 12, the coordinate (X) is given by equation (9). V +t·α,Y V +t·β,Z V +t·γ) is located on the mirror surface MS. Therefore, the coordinate (X V +t·α,Y V +t·β,Z V Since +t·γ) must satisfy the plane equation, the following equation (10) holds true.

[0066] α·(X V +t·α)+β·(Y V +t·β)+γ·(Z V +t·γ)+δ=0 ...(10)

[0067] Solving equation (10) for t yields equation (11).

[0068] t = -(α·X) V +β·Y V +r·Z V +δ) / (α 2 +β 2 +γ 2 ) ···(11)

[0069] Since the right-hand side of equation (11) is known, the estimation unit 15 calculates the parameter t and substitutes the calculated parameter t into equation (9) to calculate the coordinates S(x,y,z) of object OB. Note that the estimation of the coordinates S(x,y,z) of object OB is not limited to the above calculation method and may be achieved by other methods, other calculation means, etc. For example, the estimation unit 15 may input the three-dimensional positions of multiple points on the outer edge of the plane mirror VCM and the three-dimensional position of the virtual image VI of object OB into a pre-trained neural network and estimate the coordinates S(x,y,z) of object OB from the output of the neural network.

[0070] When estimating the coordinates S(x,y,z) of object OB using a pre-trained neural network, the mirror position determination unit 16 becomes unnecessary. Furthermore, the neural network used in the estimation unit 15 is trained by using the 3D positions of multiple objects whose 3D positions are known as training data, and inputting the 3D position of the virtual image VI of object OB into the neural network. As known methods such as backpropagation can be used for training, a detailed explanation is omitted. The neural network can be implemented using machine learning models such as CNN (Convolutional Neural Network) or DNN (Deep Neural Network).

[0071] The overall configuration and functions of the information processing system 1, including the vehicle object position estimation device 100 according to this embodiment, have been described above. Under this configuration, the vehicle object position estimation device 100 performs a process (hereinafter referred to as the estimation process) to estimate the position of an object corresponding to the virtual image VI in the mirror surface MS within the detection range of the distance measuring device 19. The procedure for the estimation process will be described below.

[0072] Figure 13 is a flowchart illustrating an example of the estimation process procedure. Prior to the execution of the estimation process, it is assumed that, for example, the engine of the mobile unit 200 has been started. That is, the estimation process is performed in response to the starting of the engine of the mobile unit 200.

[0073] (Estimation process) (Step S1) The distance measuring device 19 acquires positional information of an object within the detection range (measurement range) for distance measurement around the moving object 200. For example, the distance measuring device 19 measures the three-dimensional positions of multiple points in the LiDAR field of view. As a result, the acquisition unit 11 acquires, for example, the three-dimensional positions of multiple points on the outer edge of a convex or plane mirror in the LiDAR field of view from the measurement unit 25 of the distance measuring device 19.

[0074] Specifically, the acquisition unit 11 acquires, for objects reflected in the convex mirror CM, the sum of the distance from the distance measuring device 19 to the reflection point of the convex mirror and the distance from the reflection point to the object, as distance information (optical path distance) from the measurement unit 25. For objects reflected in the plane mirror VCM, the acquisition unit 11 acquires, for objects reflected in the plane mirror VCM, the distance from the distance measuring device 19 to the convex virtual image V, as distance information from the measurement unit 25. The acquisition unit 11 also acquires, for objects reflected in the plane mirror VCM, the angle between the pre-reflection optical path and the plane containing the direction of movement of the moving body 200, and the angle between the pre-reflection optical path and the direction of travel of the moving body 200, as angle information from the measurement unit 25. The acquisition unit 11 stores the distance information and angle information, including the point cloud positions in the LiDAR field of view, in the memory 103 or storage device 105.

[0075] (Step S2) The determination unit 12 determines whether the distance to the 3D position of the point cloud inside the region enclosed by the 3D positions of the multiple points is greater than any of the distances to the 3D positions of the multiple points. For example, by comparing the 3D positions (distance information) of the multiple points with the distance information of the point cloud inside the region enclosed by the multiple points, if the distance of the point cloud inside the region is greater than the distance to the outer edge of the region, that is, if the depth of the point cloud inside the region is greater than any of the 3D positions of the multiple points, the determination unit 12 determines that a mirror surface exists within the detection range (inside the region enclosed by the multiple points). Specifically, if the difference in distance information between the inside of a particular region and the outer edge of that region exceeds a predetermined value for each of the distance information of the multiple points in the LiDAR field of view, the determination unit 12 determines that the region is a mirror surface. If a mirror surface is found in the detection range (Yes in step S2), the process in step S3 is executed. If there is no mirror surface in the detection range (No in step S2), the process in step S11 is executed.

[0076] (Step S3) The determination unit 12 uses the angle information acquired by the acquisition unit 11 to determine whether the mirror surface is a convex mirror or not (or a plane mirror). For example, if the elevation angle in the angle information is positive, the determination unit 12 determines that the mirror surface is a convex mirror. Also, if the elevation angle in the angle information is approximately zero, that is, if the mirror surface is at approximately the same height as the distance measuring device, the determination unit 12 determines that the mirror surface is a plane mirror. If the mirror surface is a convex mirror (Yes in step S3), the processes in steps S4 and S5 are executed. If the mirror surface is not a convex mirror, that is, if the mirror surface is a plane mirror (No in step S3), the processes in steps S8 and S9 are executed.

[0077] (Step S4) The acquisition unit 11 acquires the optical path distance L related to the object and angular information related to the reflection point (the vertical angle A of the optical path OP before reflection). V and horizontal angle A H The acquisition unit 11 obtains the optical path distance L and the vertical angle A from the measurement unit 25. V and horizontal angle A HThe data is stored in memory 103 or storage device 105.

[0078] (Step S5) The radius of curvature determination unit 13 determines the radius of curvature of the convex mirror CM based on distance information. For example, the radius of curvature determination unit 13 estimates the shape and size of the convex mirror CM based on the three-dimensional positions of multiple points on the outer edge of the convex mirror CM, and compares the estimated shape and size of the convex mirror CM with the radius of curvature correspondence table LUT. This allows the radius of curvature determination unit 13 to determine the radius of curvature R of the convex mirror CM. The radius of curvature determination unit 13 stores the determined radius of curvature R in the memory 103 or storage device 105. The order of the processing in step S4 and step S5 is arbitrary; they may be simultaneous or different.

[0079] (Step S6) The curvature center determination unit 14 determines the curvature center Q based on distance information and the radius of curvature R. For example, the curvature center determination unit 14 determines the coordinates (x Q ,y Q ,z Q Three equations (1) are set up for the distance between the center of curvature CC and each of the three-dimensional positions of multiple (three or more) points on the outer edge, and the three-dimensional position Q of the center of curvature CC is determined by solving these three equations. The center of curvature determination unit 14 stores the determined coordinates Q of the center of curvature CC in the memory 103 or storage device 105.

[0080] (Step S7) The estimation unit 15 determines the three-dimensional position S(x) of the object based on the optical path distance L, radius of curvature R, center of curvature Q, and angular information. s , y s , z s The estimation unit 15 estimates the horizontal angle B of the curvature center Q by applying the coordinates of the curvature center Q to equation (2). H and vertical angle B V The estimation unit 15 calculates the angle information (vertical angle A of the reflection point P) obtained by the acquisition unit 11. V and horizontal angle A H ) and the horizontal angle B of the center of curvature QH and vertical angle B V By applying this to equation (3), we can calculate ∠POQ=T and use the coordinates of the center of curvature Q to calculate the length W of OQ.

[0081] Next, the estimation unit 15 calculates the length m of OP by applying the law of cosines to triangle OPQ shown in Figure 9. The estimation unit 15 also calculates ∠OQP by applying the calculated length m of OP to the law of sines for triangle OPQ. Subsequently, the estimation unit 15 calculates the coordinates (x, y) of the feature point S of object OB on the X'Y' plane by applying ∠POQ=T, the optical path distance L, ∠OQP=a, and the length m of OP to equation (4).

[0082] Furthermore, the estimation unit 15 calculates ∠SOQ=D using the coordinates (x, y) of the feature point S. The estimation unit 15 then calculates ∠SOQ=D and the horizontal angle B of the curvature center Q. H and vertical angle B V And the vertical angle A of the pre-reflection optical path OP. V and horizontal angle A H Applying ∠POQ=T to equation (6), we obtain the angle of S shown in Figure 11 (horizontal angle c H and vertical angle c V The estimation unit 15 calculates the length K of OS using the coordinates (x, y) of the feature point S. Finally, the estimation unit 15 calculates the angle of S (horizontal angle c). H and vertical angle c V Using the length K of the OS, the 3D position S(x) of the feature point S in XYZ space is determined. s , y s , z s ) is calculated. The processes in steps S4 to S7 correspond to the convex mirror process.

[0083] The processing in steps S5 to S7 may be implemented by a pre-trained neural network (which may also be called a trained model). In this case, steps S5 and S6 become unnecessary. In step S7, the estimation unit 15 obtains the three-dimensional positions of multiple points on the outer edge of the convex mirror CM, distance information (optical path distance L) between the distance measuring device 19 and the object OB, and angular information (vertical angle A of the pre-reflection optical path OP) relating to the reflection point.V and horizontal angle A H The 3D position S(x) of object OB is input to a pre-trained neural network, and the output from the neural network is used to determine the 3D position S(x) of object OB. s , y s , z s We estimate ).

[0084] The neural network is generated by training it using the 3D positions of multiple objects whose 3D positions are known as training data, and inputting the distance and angle information of each of these known objects into the neural network. The trained neural network may be configured in hardware or software within the estimation unit 15, and the parameters of the neural network may be stored in the memory 103 or storage device 105.

[0085] (Step S8) The acquisition unit 11 obtains the 3D position V(X) of the virtual image from the measurement unit 25. V ,Y V ,Z V ) is obtained. 3D position V(X V ,Y V ,Z V The measurement unit 25 calculates the 3D position V(X) of the virtual image using distance information (TOF information). The acquisition unit 11 calculates the 3D position V(X) of the virtual image. V ,Y V ,Z V ) is stored in memory 103 or storage device 105.

[0086] (Step S9) The mirror surface position determination unit 16 determines the mirror surface position of the plane mirror based on the three-dimensional positions of multiple points (for example, three points U1, U2, and U3) on the outer edge of the plane mirror VCM. For example, the mirror surface position determination unit 16 determines the plane equation of the mirror surface by substituting the three-dimensional positions of the three points U1, U2, and U3 into the equation of the plane. The mirror surface position determination unit 16 also calculates the normal vector of the mirror surface based on vectors U1U2 and U1U3, and then calculates the normal vector and the coordinates (X) of U1. U1 ,YU1 ,Z U1 The plane equation of the mirror surface is determined using ). Note that the order of the process in step S8 and the process in step S9 is arbitrary and may be simultaneous or different.

[0087] (Step S10) The estimation unit 15 estimates the three-dimensional position of object OB based on the three-dimensional position of the virtual image and the mirror surface position (the plane equation of the mirror surface). For example, as shown in equation (9), the estimation unit 15 estimates the coordinates S(x,y,z) of object OB and the three-dimensional position V(X) of the virtual image. V ,Y V ,Z V The 3D position V(X) of the virtual image is expressed using the normal vector (α, β, γ) of the mirror surface and the parameter t. The estimation unit 15 also calculates the 3D position V(X) of the virtual image. V ,Y V ,Z V The parameter is determined by substituting the position on the mirror surface, using the normal vector of the mirror surface (α, β, γ) and the parameter t into the plane equation of the mirror surface and solving equation (10). Finally, the estimation unit 15 calculates the coordinates S(x,y,z) of object OB by substituting the determined parameter t into equation (9).

[0088] Note that the processing in steps S9 to S10 may be implemented by a pre-trained neural network (which may also be called a trained model). In this case, step S9 is unnecessary. In step S9, the estimation unit 15 determines the three-dimensional positions of multiple points on the outer edge of the plane mirror VCM and the three-dimensional position V(X) of the virtual image. V ,Y V ,Z V The values ​​of ) and are input into a pre-trained neural network, and the coordinates S(x,y,z) of object OB are estimated from the output of the neural network.

[0089] The neural network is generated by training it using the 3D positions of multiple objects whose 3D positions are known as training data, and inputting the 3D position information of the virtual images of each of the multiple objects into the neural network. The trained neural network may be configured in hardware or software within the estimation unit 15, and the parameters of the neural network may be stored in the memory 103 or storage device 105.

[0090] (Step S11) If the estimation process does not complete (No in step S11), the processes from step S1 onwards are executed. If the estimation process completes (Yes in step S11), the flow of this estimation process ends. The completion of the estimation process is determined by, for example, the determination unit 12, based on events such as the stopping of the engine of the mobile unit 200, the disembarking of the user from the mobile unit 200, or the locking of the passenger doors of the mobile unit 200.

[0091] After the processing in step S7 or step S10, if the distance between the mobile body 200 and object OB is approaching to a predetermined value or less, the processor 101 outputs a warning from the output device 111. The warning output may be, for example, an alarm, a buzzer, a voice message such as "An object (such as a car) is approaching from ~", or a display showing the positional relationship between the vehicle (mobile body 200) and object OB. This display is not limited to a display and may be implemented with a projector or the like. Through these means, the output device 111 informs the user riding in the mobile body 200 that the distance between the mobile body 200 and object OB is approaching to a predetermined value or less, using sound, voice, display, etc.

[0092] Furthermore, the processor 101 may control the movement of the moving body 200 if the distance between the moving body 200 and object OB is below a predetermined value. Such movements may include, for example, releasing the accelerator, applying the brakes, or operating the steering wheel. The processor 101 may also perform both the notification of a warning and the control of the moving body 200.

[0093] As described above, the vehicle object position estimation device 100 according to the embodiment obtains the three-dimensional positions of a plurality of points from a sensor (distance measuring device 19) that measures a plurality of points on the outer edge of a convex mirror CM, determines the radius of curvature R of the convex mirror CM, determines the three-dimensional position of the center of curvature Q of the convex mirror CM based on the three-dimensional positions of the plurality of points and the radius of curvature R, obtains distance information between the sensor and the object OB based on the reflection of light rays or electromagnetic waves emitted by the sensor from the convex mirror CM and the reflection from the object OB, obtains angle information of the reflection point P on the convex mirror CM relative to the sensor from the sensor, and estimates the three-dimensional position of the object OB based on the radius of curvature R, the three-dimensional position of the center of curvature Q, the distance information and the angle information.

[0094] Furthermore, the vehicle object position estimation device 100 according to the embodiment determines the radius of curvature R by referring to the product specifications (radius of curvature correspondence table LUT) of multiple road reflectors based on the shape and size of the road reflectors (convex mirrors CM) estimated based on the three-dimensional positions of multiple points. In addition, the vehicle object position estimation device 100 according to the embodiment acquires a point cloud located inside the region enclosed by the three-dimensional positions of multiple points, and determines that a convex road reflector exists in the region if the point cloud is located at a position where the depth from the laser emission unit 21 is greater than any of the three-dimensional positions of the multiple points, and the region is located above the horizontal direction relative to the laser emission unit 21.

[0095] As a result, according to the vehicle object position estimation device 100 according to the embodiment, when the presence of a mirror surface and the determination that the mirror surface is a convex mirror are made, the radius of curvature R and the center of curvature of the convex mirror CM are determined using the three-dimensional positions of a plurality of points on the outer edge of the convex mirror CM. Then, using the determined radius of curvature R and center of curvature, the distance information between the distance measuring device 19 and the object OB, and the angle information of the reflection point P, the three-dimensional position of the object can be uniquely and accurately determined. Thus, according to the vehicle object position estimation device 100 according to the embodiment, for example, when an object such as a vehicle is reflected in a curve mirror installed at an intersection with poor visibility, the three-dimensional position of the vehicle can be accurately estimated.

[0096] Furthermore, the vehicle object position estimation device 100 according to the embodiment estimates the 3D position of object OB by inputting the 3D positions, distance information, and angle information of multiple points into a pre-trained neural network, and using the output from the neural network. In this case, the neural network of the vehicle object position estimation device 100 according to the embodiment is trained by inputting the 3D positions of multiple known objects as training data, and the distance information and angle information of each known object into the neural network. As a result, the vehicle object position estimation device 100 according to the embodiment can more easily estimate the 3D position of object OB by inputting the 3D positions, distance information, and angle information of multiple points into a trained model.

[0097] Furthermore, the vehicle object position estimation device 100 according to the embodiment measures the three-dimensional positions of a plurality of points on the outer edge of the mirror with a sensor (distance measuring device 19), determines that the mirror is a convex mirror if the region enclosed by the three-dimensional positions of the plurality of points is above the horizontal direction relative to the sensor, determines that the mirror is a plane mirror if the region overlaps with the horizontal direction relative to the sensor, performs convex mirror processing if the mirror is determined to be a convex mirror, performs plane mirror processing if the mirror is determined to be a plane mirror, estimates the radius of curvature R of the convex mirror CM, and calculates the curvature of the convex mirror CM based on the three-dimensional positions of the plurality of points and the radius of curvature R The 3D position Q of the center CC is estimated, distance information between the sensor and object OB is acquired by the sensor based on the reflection of light rays or electromagnetic waves emitted by the sensor from the convex mirror CM and object OB, angle information of the reflection point P on the convex mirror CM relative to the sensor is acquired by the sensor, the 3D position of object OB is estimated based on the radius of curvature R, the 3D position Q of the center of curvature CC, the distance information and the angle information, in the plane mirror processing, the 3D position of the virtual image VI of object OB is measured by the sensor, the mirror surface position (plane equation of the mirror surface) of the plane mirror is determined based on the 3D positions of multiple points, and the 3D position of object OB is estimated based on the 3D position of the virtual image VI and the mirror surface position.

[0098] As a result, according to the vehicle object position estimation device 100 of this embodiment, if the mirror is a convex mirror, convex mirror processing can be performed, and if the mirror is a planar mirror, planar mirror processing can be performed, and the three-dimensional position of object OB can be accurately estimated regardless of whether the mirror is a convex or planar mirror.

[0099] Based on the above, the vehicle object position estimation device 100 according to the embodiment can accurately estimate the three-dimensional position of an object OB reflected in a mirror, thereby notifying the user of an object OB approaching the vehicle (mobile body 200) and / or controlling the operation of the mobile body 200. For this reason, the vehicle object position estimation device 100 according to the embodiment can accurately estimate the three-dimensional position of an object reflected on a mirror surface such as a convex mirror, thereby always obtaining accurate position information of the object OB, and improving the safety of the user and the mobile body 200 as the mobile body 200 moves.

[0100] (First variation) This modified version acquires the three-dimensional positions of at least four points on the outer edge of the convex mirror CM from the distance measuring device 19 (sensor), and determines the radius of curvature R based on the three-dimensional positions of at least four points. For example, the acquisition unit 11 acquires the three-dimensional positions of at least four points on the outer edge of the convex mirror CM from the measurement unit 25. The curvature center determination unit 14 determines the radius of curvature R based on the three-dimensional positions of at least four points. Specifically, the curvature center determination unit 14 determines the curvature center Q and three points U1(X) on the outer edge. U1 ,Y U1 ,Z U1 ), U2(X U2 ,Y U2 ,Z U2 ), U3(X U3 ,Y U3 ,Z U3 ), U4(X U4 ,Y U4 ,Z U4 The distance (radius of curvature R) between each of these is expressed by the following equation (12).

[0101] {(x Q -X U1 ) 2 +(yQ -Y U1 ) 2 +(z Q -Z U1 ) 2} 1 / 2 =R {(x Q -X U2 ) 2 +(y Q -Y U2 ) 2 +(z Q -Z U2 ) 2} 1 / 2 =R {(x Q -X U3 ) 2 +(y Q -Y U3 ) 2 +(z Q -Z U3 ) 2} 1 / 2 =R {(x Q -X U4 ) 2 +(y Q -Y U4 ) 2 +(z Q -Z U4 ) 2} 1 / 2 =R ···(12)

[0102] In equation (12), there are four unknowns (x Q , y Q , z Q Since there are four equations, the curvature center determination unit 14 solves the four equations in equation (12) simultaneously to determine the coordinates Q(x) of the curvature center CC. Q ,y Q ,z QThe radius of curvature R is uniquely determined. Therefore, in the estimation process in this modified example, the radius of curvature determination unit 13 and step S5 are unnecessary. For this reason, the estimation process can be further simplified in this modified example. Furthermore, according to the vehicle object position estimation device 100 of this modified example, the radius of curvature R can be determined even for convex mirrors not listed in the radius of curvature correspondence table. Other effects of this modified example are the same as in the embodiment, so their explanation is omitted.

[0103] (Second variation) This modified version is applicable when a second object, different from object OB, is located in a position where it can be directly measured by the distance measuring device 19, and the second object is reflected in the convex mirror. In this case, object OB, which is directly measured by the distance measuring device 19, will be referred to as the real image of the second object. In this modified version, the radius of curvature R is calculated using the coordinates of the real image of the second object and the optical path distance from the distance measuring device 19 to the second object via the reflection point.

[0104] The modified example will be explained below with reference to Figure 9. In Figure 9, the coordinates S(x,y) of the feature points of object OB are assumed to be those of a second object. The second object is, for example, a white line on a road, a guardrail, or a wall reflected in a mirror. The determination unit 12 determines that the virtual image and the real image reflected in the mirror are from the same object. The determination unit 12 determines that the mirror image and the real image of the second object are from the same object by, for example, analyzing the image acquired by the optical camera mounted on the mobile body 200 using a known method. A known method is, for example, semantic segmentation (region classification) processing using a pre-trained model.

[0105] The acquisition unit 11 acquires the three-dimensional position S(x,y) of the second object from the measurement unit 25 of the distance measuring device 19 (sensor). The acquisition unit 11 also acquires second distance information between the distance measuring device 19 and the second object from the distance measuring device 19, based on the reflection of the light ray or electromagnetic wave emitted by the laser light-emitting unit 21 of the distance measuring device 19 (sensor) from the convex mirror and the reflection from the second object. The second distance information corresponds to the optical path distance L, which is the sum of the distance between OP and the distance between PS in Figure 9. The acquisition unit 11 stores the three-dimensional position S(x,y) of the second object and the second distance information (optical path distance L) in the memory 130 or storage device 105.

[0106] The radius of curvature determination unit 13 determines the radius of curvature R based on the three-dimensional position S(x,y) of the second object and the second distance information. For example, the radius of curvature determination unit 13 uses equation (4), the sine rule (m / sin(a)=R / sin(T)=W / sin(180°-(a+T))) in triangle OPQ shown in Figure 9, and the cosine rule (R 2 =m 2 +W 2 The radius of curvature R is determined based on -2m·W·cos(T). Specifically, the radius of curvature R is calculated by solving a system of five equations consisting of two equations obtained by substituting the x and y coordinates (optical path distance L) of the 3D position S of the second object into equation (4), two equations using the Law of Sines, and one equation using the Law of Cosines, along with ∠POQ=T, ∠OQP=a, the length of OP m, and the length of OQ W.

[0107] As a result, the vehicle object position estimation device 100 according to this modified example can determine the radius of curvature R even for convex mirrors not listed in the radius of curvature correspondence table. Other effects of this modified example are the same as those of the embodiment, so their explanation is omitted.

[0108] (Third variation) This modified version describes the case where the mirror is a rectangular convex mirror (hereinafter referred to as a rectangular mirror), which is different from the round mirror described in the embodiment, and the case where the road reflector consists of two convex mirrors (two-sided mirrors).

[0109] Figure 14 shows an example of a rectangular mirror RM and a two-sided mirror TSM. In the rectangular mirror RM shown in Figure 14, the coordinates of at least three points (U1, U2, U3) on the outer edge of the mirror are measured by the distance measuring device 19, similar to the embodiment. The mirror surface of the rectangular mirror RM is part of a sphere, similar to the mirror surface of a round mirror. Therefore, even if the road reflector is a rectangular mirror RM, the estimation process is performed in the same way as in the embodiment.

[0110] In the two-mirror TSM shown in Figure 14, estimation processing is performed for each of the two convex mirrors (M1, M2) in the same manner as in the embodiment. For example, if vehicles in Japan or other countries drive on the left side of the road, estimation processing is performed for the right-hand mirror M2 relative to the moving vehicle 200, and then for the left-hand mirror M1 relative to the moving vehicle 200. If multiple convex mirrors are included in the detection range for distance measurement, that is, if multiple mirrors at distant positions are detected by the distance measuring device 19, estimation processing is performed starting with the convex mirror closest to the moving vehicle 200.

[0111] As a result, with the vehicle object position estimation device 100 according to this modified example, when multiple convex mirrors are included in the distance measurement detection range, estimation processing can be performed from the convex mirror closest to the moving object 200 and / or the convex mirror located in a direction where collision with the moving object 200 is highly likely, thereby improving the safety of the user and the moving object 200 as the moving object 200 moves. Other effects of this modified example are the same as those of the embodiment, so their explanation will be omitted.

[0112] When the technical concept of the embodiment is realized by an object position estimation method, the object position estimation method is an object position estimation method for estimating the three-dimensional position of an object OB reflected in a convex mirror CM, and includes the steps of: obtaining the three-dimensional positions of a plurality of points from a sensor that measures a plurality of points on the outer edge of the convex mirror CM; determining the radius of curvature R of the convex mirror CM; determining the three-dimensional position Q of the center of curvature CC of the convex mirror CM based on the three-dimensional positions of the plurality of points and the radius of curvature R; obtaining distance information between the sensor and the object OB from the sensor based on the reflection of light rays or electromagnetic waves emitted by the sensor from the convex mirror CM and from the object OB; obtaining angle information from the sensor of the reflection point P on the convex mirror CM with respect to the sensor for the said reflection; and estimating the three-dimensional position of the object OB based on the radius of curvature R, the three-dimensional position Q of the center of curvature CC, the distance information and the angle information. The procedure and effects of the estimation process performed by the object position estimation method are the same as in the embodiment, so a description is omitted.

[0113] According to the vehicle object position estimation device 100 and object position estimation method having the above configuration, the three-dimensional position of an object reflected on a mirror surface such as a convex mirror can be estimated with high accuracy. As a result, according to one embodiment of the vehicle object position estimation device 100 and object position estimation method disclosed in this application, for example, by accurately estimating the three-dimensional position of an object reflected on a mirror surface such as a convex mirror, accurate position information of an object OB can always be obtained, thereby improving the safety of the user and the moving body 200 as the moving body 200 moves.

[0114] Although embodiments and various modifications have been described above, the vehicle object position estimation device 100 and object position estimation method disclosed in this application are not limited to the above embodiments, etc., and the components can be modified and implemented in each implementation stage, etc., without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments and various modifications, etc. For example, some components may be deleted from all the components shown in the embodiments. [Explanation of Symbols]

[0115] 1. Information Processing System 11 Acquisition Department 12 Judgment section 13 Curvature radius determination section 14 Center of curvature determination part 15 Estimation part 16 Mirror position determination section 19 Ranging device 19A, 19B, 19C, 19D, 19E sensors 21 Laser light-emitting section 23 Laser light receiving section 25 Measurement section 100 Vehicle object position estimation device 101 Processors 103 memory 105 Storage Devices 107 I / F (Interface) 109 Input device 111 Output device 113 Communication equipment 115 Bus 200 Mobile Units

Claims

1. An object position estimation method for estimating the three-dimensional position of an object reflected in a convex mirror, The steps include obtaining the three-dimensional positions of a plurality of points from a sensor that measures a plurality of points on the outer edge of the convex mirror, The steps include determining the radius of curvature of the convex mirror, A step of determining the three-dimensional position of the center of curvature of the convex mirror based on the three-dimensional positions of the plurality of points and the radius of curvature, A step of obtaining distance information between the sensor and the object based on the reflection of light rays or electromagnetic waves emitted by the sensor from the convex mirror and from the object, The steps include obtaining angular information from the sensor regarding the reflection point of the convex mirror with respect to the sensor, A step of estimating the three-dimensional position of the object based on the radius of curvature, the three-dimensional position of the center of curvature, the distance information, and the angle information, An object position estimation method comprising the following features.

2. The radius of curvature is determined by referring to the product specifications of the convex mirror based on the shape and size of the convex mirror estimated based on the three-dimensional positions of the plurality of points. The method for estimating the position of an object according to claim 1.

3. The three-dimensional positions of at least four points on the outer edge of the convex mirror are obtained from the sensor. The radius of curvature is determined based on the three-dimensional positions of the at least four points. The method for estimating the position of an object according to claim 1.

4. If a second object, different from the aforementioned object, is located in a position where it can be directly measured by the sensor, and the second object is reflected in the convex mirror, The steps include obtaining the three-dimensional position of the second object described above from the sensor, The step of obtaining second distance information between the sensor and the second object from the sensor, based on the reflection of light rays or electromagnetic waves emitted by the sensor from the convex mirror and from the second object, The radius of curvature is determined based on the three-dimensional position of the second object and the second distance information. The method for estimating the position of an object according to claim 1.

5. The system includes a mirror surface determination step in which it is determined that a mirror surface exists in a region if the depth of the point cloud acquired by the sensor is greater than any of the three-dimensional positions of the plurality of points within the region enclosed by the three-dimensional positions of the plurality of points. The method for estimating the position of an object according to any one of claims 1 to 4.

6. The mirror surface determination step determines that a convex mirror exists in the region because the region is located above the horizontal direction with respect to the sensor. The method for estimating the position of an object according to claim 5.

7. The three-dimensional positions of the aforementioned points, the distance information, and the angle information are input into a pre-trained neural network, and the three-dimensional position of the object is estimated from the output of the neural network. The method for estimating the position of an object according to claim 1.

8. The aforementioned neural network was trained by inputting the distance and angle information of each of several objects whose three-dimensional positions are known, using the three-dimensional positions of each of the known objects as training data. The method for estimating the position of an object according to claim 7.

9. An acquisition unit acquires the three-dimensional position of a measurement target, including distance information and angle information to the measurement target, calculated based on the elapsed time from when the laser emission unit emits a laser until the laser receiving unit receives the laser, and the receiving angle of the received laser, from a sensor having the laser emission unit and the laser receiving unit. When an object is reflected in a road mirror within the laser's irradiation range, A radius of curvature determination unit determines the radius of curvature of the road reflector based on the three-dimensional positions of multiple points on the outer edge of the road reflector obtained from the sensor, A curvature center determination unit that determines the three-dimensional position of the curvature center of the road reflector based on the three-dimensional positions of the plurality of points and the radius of curvature, An estimation unit estimates the three-dimensional position of an object based on the reflection of the laser beam emitted by the laser light emitter from the road reflector and the object, and based on distance information and angle information about the object obtained from the sensor, the radius of curvature, and the three-dimensional position of the center of curvature. A vehicle object position estimation device having the following features.

10. The radius of curvature determination unit is, The radius of curvature is determined by referring to the product specifications of multiple road mirrors based on the shape and size of the road mirrors estimated based on the three-dimensional positions of the multiple points. The vehicle object position estimation device according to claim 9.

11. The acquisition unit acquires the three-dimensional positions of at least four points on the outer edge of the road reflector from the sensor, The radius of curvature determination unit determines the radius of curvature based on the three-dimensional positions of the at least four points. The vehicle object position estimation device according to claim 9.

12. If a second object, different from the aforementioned object, is located in a position where it can be directly measured by the sensor, and the second object is reflected in the road reflector, The acquisition unit is, The three-dimensional position of the second object described above is obtained from the sensor, Based on the reflection of the laser beam emitted by the laser light emitter from the road reflector and the second object, second distance information relating to the second object is obtained from the sensor. The radius of curvature determination unit determines the radius of curvature based on the three-dimensional position of the second object and the second distance information. The vehicle object position estimation device according to claim 9.

13. The acquisition unit acquires a point cloud located within the region enclosed by the three-dimensional positions of the plurality of points. The system further includes a determination unit that determines that a convex mirror road reflector exists in the region if the point cloud is located at a position where the depth from the laser emission unit is greater than any of the three-dimensional positions of the plurality of points, and the region is located above the horizontal relative to the laser emission unit. The vehicle object position estimation device according to any one of claims 9 to 12.

14. The estimation unit has a pre-trained neural network and estimates the three-dimensional position of the object by inputting the three-dimensional positions of the plurality of points, the distance information, and the angle information into the neural network, and using the output from the neural network. The vehicle object position estimation device according to claim 9.

15. The aforementioned neural network was trained by inputting the distance and angle information of each of several known objects, using the three-dimensional positions of each known object as training data. The vehicle object position estimation device according to claim 14.

16. An object position estimation method for estimating the three-dimensional position of an object reflected in a mirror, The steps include measuring the three-dimensional positions of multiple points on the outer edge of the mirror using a sensor, The steps include determining that the mirror is a convex mirror if the region enclosed by the three-dimensional positions of the plurality of points is above the horizontal direction with respect to the sensor, and determining that the mirror is a plane mirror if the region coincides with the horizontal direction with respect to the sensor, The process includes the steps of: performing a convex mirror process if the mirror is determined to be a convex mirror, and performing a plane mirror process if the mirror is determined to be a plane mirror. The aforementioned convex mirror processing is The steps include: estimating the radius of curvature of the convex mirror; A step of estimating the three-dimensional position of the center of curvature of the convex mirror based on the three-dimensional positions of the plurality of points and the radius of curvature, The steps include: acquiring distance information between the sensor and the object based on the reflection of light rays or electromagnetic waves emitted by the sensor by the convex mirror and the object; The steps include: acquiring angular information of the reflection point of the convex mirror with respect to the sensor for the aforementioned reflection using the sensor; The step includes estimating the three-dimensional position of the object based on the radius of curvature, the three-dimensional position of the center of curvature, the distance information, and the angle information. The aforementioned plane mirror processing is performed by The steps include measuring the three-dimensional position of the virtual image of the object using the sensor, The steps include determining the mirror surface position of the plane mirror based on the three-dimensional positions of the plurality of points, The step includes estimating the three-dimensional position of the object based on the three-dimensional position of the virtual image and the mirrored position. Object position estimation method.

Images