Simulation system

Abstract

To provide a simulation system making it possible to reduce man-days required to create a virtual space model.SOLUTION: A simulation system 20 includes: a point group acquisition unit 25 that acquires point group data, which includes position coordinates of a plurality of reflection points P constituting a point group of an object 7 existing in a simulation target actual environment; and a reflection point extraction unit 26 that, when virtual laser light L0 having a three-dimensional specific region Rs is radiated from a virtual laser sensor 3A, which corresponds to a laser sensor 3, to a targeted point group PG while a virtual moving body 2A corresponding to a moving body 2 is virtually being traveled, extracts position coordinates of the plurality of reflection points P existing within the three-dimensional specific region Rs.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a simulation system.

Background Art

[0002] For example, Patent Document 1 describes a simulation system that performs machine learning and testing of an image recognition function module in an automatic driving system of a vehicle using a virtual image of a LiDAR sensor.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, for example, in the simulation of a moving body that performs autonomous driving, a virtual space model for virtually driving the moving body is required. Recently, although the disclosure of virtual space models for general roads and urban areas for passenger cars has started, there is no available virtual space model for the inside of a facility (for example, inside a factory or an airport) where industrial vehicles such as forklifts and towing tractors travel. At present, in order to reproduce the user's environment, after acquiring point cloud data by a distance measuring sensor such as LiDAR in the user's facility, a virtual space model is created by manually arranging 3D CG data. Therefore, the man-hours required for creating the virtual space model increase.

[0005] An object of the present invention is to provide a simulation system capable of reducing the man-hours required for creating a virtual space model.

Means for Solving the Problems

[0006] (1) One aspect of the present invention is a simulation system for simulating the movement of a moving body, comprising: a point cloud data acquisition unit that acquires point cloud data including the position coordinates of a plurality of reflection points that constitute a point cloud of an object existing in the real environment to be simulated; and a reflection point extraction unit that extracts the position coordinates of a plurality of reflection points existing within a predetermined three-dimensional region when a virtual signal having a predetermined three-dimensional region is emitted from a virtual sensor corresponding to the sensor toward the target point cloud while a virtual moving body corresponding to the moving body is virtually driven.

[0007] In such a simulation system, point cloud data is first acquired, which includes the position coordinates of multiple reflection points that constitute the point cloud of an object existing in the real environment being simulated. Then, while a virtual mobile body corresponding to the moving object is virtually driven, a virtual signal with a defined 3D region is emitted from a virtual sensor corresponding to the sensor used for the mobile body's movement toward the target point cloud. At this point, the position coordinates of multiple reflection points existing within the defined 3D region are extracted. In this way, the sensor simulation is performed using the point cloud data of the real environment being simulated directly, eliminating the need to manually place 3D CG data. This reduces the man-hours required to create the virtual space model.

[0008] Furthermore, for example, if a one-dimensional virtual signal is emitted from a virtual sensor towards the target point cloud, the one-dimensional virtual signal may pass between reflection points. Therefore, the virtual sensor emits a virtual signal with a defined three-dimensional region. When the virtual sensor emits a virtual signal, the position coordinates of multiple reflection points that exist within the defined three-dimensional region are extracted. As a result, even if the virtual signal passes between reflection points without hitting any of them, other reflection points that exist within the defined region are extracted. Consequently, even when performing a simulation using point cloud data as is, it is possible to extract the position coordinates of multiple reflection points in the same way as when performing a simulation using a virtual space model with 3D CG data.

[0009] (2) In (1) above, the specified region is the region centered on the center line of the virtual signal, and may be set so that the area of ​​the plane cut perpendicular to the center line increases as the distance from the virtual sensor increases. In such a configuration, even when the point cloud is far away from the virtual sensor, multiple reflection points will exist within the specified region, so that there will be enough reflection points to be used for subsequent processing.

[0010] (3) In (1) or (2) above, the simulation system may further include a distance calculation unit that calculates the distance from the virtual sensor to the position of the target point cloud based on the position coordinates of a plurality of reflection points extracted by the reflection point extraction unit. In such a configuration, by calculating the distance from the virtual sensor to the position of the target point cloud, it is possible to easily determine the accuracy of self-position estimation of the virtual moving object and the accuracy of detection of obstacles present around the virtual moving object.

[0011] (4) In (3) above, the distance calculation unit may calculate the average value of the position coordinates of multiple reflection points as the estimated coordinate of the target, and calculate the distance from the virtual sensor to the position of the target point cloud based on the estimated coordinate. In such a configuration, by calculating the average value of the position coordinates of multiple reflection points, the distance from the virtual sensor to the position of the target point cloud can be determined by a simple calculation formula.

[0012] (5) In (3) above, the specified region is the region centered on the center line of the virtual signal, and the distance calculation unit may calculate the shortest distance between the center line of the virtual signal and the position coordinates of multiple reflection points, and calculate the distance from the virtual sensor to the position of the target point cloud by applying weights according to the shortest distance between the center line and the position coordinates of the reflection points. In such a configuration, since weights according to the shortest distance between the center line of the virtual signal and the position coordinates of multiple reflection points located within the specified region are taken into consideration, the distance from the virtual sensor to the position of the target point cloud can be determined with high accuracy.

[0013] (6) In any of (3) to (5) above, the point cloud data may further include the reflection intensity of multiple reflection points, and the distance calculation unit may calculate the distance from the virtual sensor to the position of the target point cloud based on the position coordinates and reflection intensity of the multiple reflection points. In such a configuration, not only the position coordinates of multiple reflection points that exist within a defined area but also the reflection intensity of each reflection point are taken into consideration, so the distance from the virtual sensor to the position of the target point cloud can be determined with high accuracy.

[0014] (7) In (1) or (2) above, a matching unit may be further provided to compare the position coordinates of multiple reflection points acquired by the point cloud data acquisition unit with the position coordinates of multiple reflection points extracted by the reflection point extraction unit. In such a configuration, by comparing the position coordinates of the reflection points acquired by the point cloud data acquisition unit with the position coordinates of the reflection points extracted by the reflection point extraction unit, it is possible to easily determine the accuracy of self-position estimation of the virtual moving object and the accuracy of detection of obstacles present around the virtual moving object. [Effects of the Invention]

[0015] According to the present invention, the man-hours required to create a virtual space model can be reduced. [Brief explanation of the drawing]

[0016] [Figure 1] This is a block diagram schematically showing the configuration of a driving control device related to a simulation system according to one embodiment of the present invention. [Figure 2] Figure 1 is a plan view showing how a laser is irradiated around a moving object by the laser sensor shown. [Figure 3] This is a block diagram schematically showing the configuration of a point cloud acquisition device related to a simulation system according to one embodiment of the present invention. [Figure 4] This is a conceptual diagram illustrating the principle of detecting the distance to objects surrounding a moving object using a laser sensor. [Figure 5] This is a plan view showing how point cloud data is acquired using the point cloud acquisition device shown in Figure 3. [Figure 6] It is a block diagram schematically showing the configuration of a simulation system according to an embodiment of the present invention. [Figure 7] It is a conceptual diagram showing a virtual laser irradiated from a virtual laser sensor together with a specified area centered on the center line of the virtual laser. [Figure 8] It is a flowchart showing the procedure of arithmetic processing executed by the arithmetic processing unit shown in FIG. 6. [Figure 9] It is a conceptual diagram showing a state of extracting the position coordinates of a plurality of reflection points existing within a specified area when a virtual laser is irradiated from a virtual laser sensor. [Figure 10] It is a flowchart showing a method of performing a simulation of autonomous driving in the past. [Figure 11] It is a flowchart showing a method of performing a simulation of autonomous driving using the simulation system shown in FIG. 6. [Figure 12] It is a conceptual diagram showing a state of acquiring the position coordinates of reflection points existing on the ray irradiation direction vector of LiDAR. [Figure 13] It is a block diagram schematically showing the configuration of a simulation system according to another embodiment of the present invention. [Figure 14] It is a flowchart showing the procedure of arithmetic processing executed by the arithmetic processing unit shown in FIG. 13. [Figure 15] It is a block diagram schematically showing the configuration of a simulation system according to still another embodiment of the present invention.

Mode for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant descriptions are omitted.

[0018] Figure 1 is a schematic block diagram showing the configuration of a driving control device related to a simulation system according to one embodiment of the present invention. In Figure 1, the driving control device 1 is a device that performs automatic driving, for example, by estimating the self-position of a moving body 2 such as a forklift and causing the moving body 2 to automatically travel along a travel path S (see Figure 2). The driving control device 1 is mounted on the moving body 2.

[0019] The driving control device 1 comprises a laser sensor 3, an environment map storage unit 4, an autonomous driving controller 5, and a drive unit 6.

[0020] As shown in Figure 2, the laser sensor 3 detects objects 7 present around the moving object 2 by irradiating a laser L around the moving object 2 within the detection area Ad of the laser sensor 3 and receiving the reflected light of the laser L. The laser sensor 3 is a distance measuring sensor that detects the distance to objects 7 present around the moving object 2 by emitting a laser L, which is a distance measuring signal. Objects 7 are stationary objects such as buildings 7A or columns 7B. The detection area Ad of the laser sensor 3 may be a predetermined angular range (e.g., 270 degrees) or a full 360 degrees. Note that in Figure 2, the detection area Ad of the laser sensor 3 is shown for convenience.

[0021] As the laser sensor 3, for example, a 3D (three-dimensional) laser scanner or a laser rangefinder can be used. Here, a LiDAR (Light Detection and Ranging) laser scanner is used as the laser sensor 3. The laser sensor 3 emits a cone-shaped laser L. The laser sensor 3 detects objects 7 present around the moving object 2 by scanning and emitting the laser L within a finite detection area Ad.

[0022] The environment map storage unit 4 stores map data of the environment in which the mobile body 2 travels. The map data includes the position coordinates of object 7, etc.

[0023] The autonomous driving controller 5 consists of a CPU, RAM, ROM, and input / output interfaces, etc. The autonomous driving controller 5 has a self-position estimation unit 8 and a driving control unit 9.

[0024] The self-position estimation unit 8 estimates the current self-position of the moving object 2 using the detection data from the laser sensor 3 and the map data stored in the environment map storage unit 4. The self-position estimation unit 8 estimates the self-position of the moving object 2 using the laser SLAM (simultaneous localization and mapping) method. SLAM is a self-position estimation technique that uses sensor data and map data to estimate the self-position.

[0025] Specifically, the self-position estimation unit 8 matches the detection data from the laser sensor 3 with map data to perform an estimation calculation of the self-position of the moving object 2. The self-position of the moving object 2 is expressed in three-dimensional coordinates (XYZ coordinates).

[0026] The driving control unit 9 controls the drive unit 6 to make the mobile body 2 autonomously travel toward the destination based on the self-position of the mobile body 2 estimated by the self-position estimation unit 8. The drive unit 6 includes, for example, a driving motor that moves the mobile body 2 and a steering motor that steers the mobile body 2 (though not shown in the figures).

[0027] Figure 3 is a schematic block diagram showing the configuration of a point cloud acquisition device related to a simulation system according to one embodiment of the present invention. In Figure 3, the point cloud acquisition device 10 is a device that acquires point cloud data used when performing an automated driving simulation using the simulation system 20 described later.

[0028] The point cloud acquisition device 10 is mounted on the mobile body 2. As shown in Figure 2, the point cloud acquisition device 10 acquires point cloud data of objects 7 present around the mobile body 2 while manually operating the mobile body 2 at a low speed along the travel path S.

[0029] The point cloud acquisition device 10 comprises the laser sensor 3 described above, a point cloud acquisition controller 11, and a point cloud data memory 12.

[0030] As described above, the laser sensor 3 detects the distance to an object 7 surrounding the moving object 2 by irradiating the moving object 2 with a laser L and receiving the reflected light of the laser L. Specifically, as shown in Figure 4, the laser sensor 3 measures the distance to object 7 from the time Δt from when it emits a laser pulse LP to when it receives the laser pulse LP (reflected pulse) reflected by object 7. Specifically, the distance to object 7 is calculated by multiplying the speed of light by the time Δt. The determination of whether or not a reflected pulse has been received is made by comparing it with a predetermined threshold A.

[0031] The point cloud acquisition controller 11 consists of a CPU, RAM, ROM, and an input / output interface, etc. The point cloud acquisition controller 11 also has a point cloud data processing unit 13 and a point cloud data storage unit 14.

[0032] As shown in Figure 5, the point cloud data processing unit 13 determines the position coordinates of multiple reflection points P that constitute the point cloud PG of the object 7 surrounding the moving object 2, based on the detection data from the laser sensor 3. The reflection points P are the points where the laser L emitted from the laser sensor 3 strikes the surface of the object 7 and is reflected. The point cloud PG is a collection of reflection points P and represents the surface of the object 7. The position coordinates of the reflection points P are expressed in three-dimensional coordinates.

[0033] The point cloud data storage unit 14 stores point cloud data, including the position coordinates of multiple reflection points P that constitute the point cloud PG obtained by the point cloud data processing unit 13, in the point cloud data memory 12. The point cloud data memory 12 is a portable memory such as a USB memory stick.

[0034] Figure 6 is a schematic block diagram showing the configuration of a simulation system according to one embodiment of the present invention. In Figure 6, the simulation system 20 of this embodiment is a system that performs a simulation of the autonomous driving of a mobile body 2. The simulation of the autonomous driving of the mobile body 2 includes a simulation of the laser sensor 3. The laser sensor 3 is a sensor used for the driving of the mobile body 2.

[0035] As shown in Figure 7, the simulation system 20 uses the point cloud data stored in the point cloud data memory 12 to determine the accuracy of the self-position estimation of the virtual mobile body 2A corresponding to the mobile body 2, and finally determines whether the virtual mobile body 2A is capable of moving.

[0036] The simulation system 20 virtually moves the virtual mobile object 2A along a virtual travel path S0 in a virtual space, and virtually shines a three-dimensional virtual laser L0 from a virtual laser sensor 3A corresponding to the laser sensor 3 towards the target point cloud PG. By calculating the distance to the position of the target point cloud PG, the system determines the accuracy of the self-position estimation of the virtual mobile object 2A.

[0037] The virtual mobile object 2A is equipped with a virtual laser sensor 3A. Therefore, the virtual laser sensor 3A moves in conjunction with the movement of the virtual mobile object 2A. The virtual space consists of a point cloud PG. The point cloud PG in question is a point cloud PG in which the distance from the virtual laser sensor 3A is calculated within the irradiation range of a finite virtual laser L0 virtually emitted from the virtual laser sensor 3A, and it moves in conjunction with the virtual movement of the virtual mobile object 2A.

[0038] The simulation system 20 includes a personal computer 21 and the point cloud data memory 12 described above.

[0039] The personal computer 21 includes an input device 22, a display device 23, and a processing unit 24 connected to the input device 22, the display device 23, and the point cloud data memory 12.

[0040] The input device 22 is a device for the user to input data and information related to the autonomous driving simulation, and to instruct the start of the autonomous driving simulation. The display device 23 is a device that displays the judgment results from the autonomous driving simulation.

[0041] The arithmetic processing unit 24 consists of a CPU, RAM, ROM, and an input / output interface, etc. The arithmetic processing unit 24 includes a point cloud acquisition unit 25, a reflection point extraction unit 26, a distance calculation unit 27, a determination unit 28, and an output unit 29.

[0042] When the input device 22 instructs the start of the automated driving simulation, the point cloud acquisition unit 25 acquires point cloud data stored in the point cloud data memory 12. The point cloud acquisition unit 25 works in cooperation with the point cloud data memory 12 to acquire point cloud data that includes the position coordinates of multiple reflection points P that constitute the point cloud PG of an object 7 present in the real environment being simulated.

[0043] As shown in Figure 7, the reflection point extraction unit 26 virtually moves the virtual mobile body 2A and, when a virtual laser L0 having a three-dimensional defined region Rs is emitted from the virtual laser sensor 3A toward the target point cloud PG, it extracts the position coordinates of multiple reflection points P that exist within the defined region Rs. The reflection point extraction unit 26 virtually moves the virtual mobile body 2A along a predetermined virtual travel path S0 determined, for example, based on point cloud data acquired by the point cloud acquisition unit 25.

[0044] The virtual laser sensor 3A is a virtual sensor that emits a cone-shaped virtual laser L0. The virtual laser L0 is a virtual signal emitted from the virtual laser sensor 3A. The virtual laser L0 is a signal corresponding to the laser L, which is a distance measuring signal emitted from the laser sensor 3. The defined region Rs is a three-dimensional region centered on the center line CL of the virtual laser L0. The reflection point extraction unit 26 is executed sequentially according to the virtual movement of the virtual moving body 2A.

[0045] The distance calculation unit 27 calculates the distance from the virtual laser sensor 3A to the position of the target point cloud PG (referred to as the target point cloud position F) based on the position coordinates of multiple reflection points P extracted by the reflection point extraction unit 26. The distance calculation unit 27 is executed sequentially according to the virtual movement of the virtual moving object 2A.

[0046] The determination unit 28 determines whether the virtual mobile object 2A is capable of moving by determining the accuracy of the self-position estimation of the virtual mobile object 2A based on the distance from the virtual laser sensor 3A to the target point cloud position F calculated by the distance calculation unit 27.

[0047] The determination unit 28 has a self-position estimation function similar to that of the self-position estimation unit 8 described above. The arithmetic processing unit 24 is also provided with a storage unit similar to that of the environment map storage unit 4 described above. The determination unit 28 estimates the self-position of the virtual mobile object 2A and compares the estimated value with the virtual travel path S0 to determine the accuracy of the self-position estimation of the virtual mobile object 2A. Note that the self-position estimation function may be provided separately as a self-position estimation unit in the arithmetic processing unit 24, in addition to the determination unit 28. Furthermore, the self-position estimation unit and the storage unit may be provided in a unit separate from the arithmetic processing unit 24.

[0048] The output unit 29 displays the self-position estimation accuracy and the determination result of whether or not it is possible to drive, as determined by the determination unit 28, on the display unit 23.

[0049] Figure 8 is a flowchart showing the steps of the calculation process performed by the calculation processing unit 24. This process is executed when the input device 22 instructs the start of the automated driving simulation.

[0050] In Figure 8, the arithmetic processing unit 24 first acquires the point cloud data stored in the point cloud data memory 12 (procedure S101). Next, the arithmetic processing unit 24 virtually moves the virtual moving object 2A along a virtual travel path S0 determined based on the position coordinates of multiple reflection points P that constitute the point cloud PG (procedure S102). The virtual travel path S0 is, for example, an intermediate portion between point cloud PGs representing different objects 7.

[0051] Next, the arithmetic processing unit 24 irradiates a virtual laser L0 having a defined region Rs from the virtual laser sensor 3A toward the target point cloud position F, as shown in Figure 7 (procedure S103). The defined region Rs is a three-dimensional region centered on the center line CL of the virtual laser L0, as described above. The defined region Rs may be the entire virtual laser L0 irradiated from the virtual laser sensor 3A, or it may be a part of the virtual laser L0 along the irradiation direction of the virtual laser sensor 3A. The virtual laser L0 is irradiated by virtually scanning from the virtual laser sensor 3A toward the vicinity of the virtual moving object 2A.

[0052] The specified region Rs is set such that the area of ​​the plane M cut perpendicular to the center line CL of the virtual laser L0 increases continuously as the distance from the virtual laser sensor 3A increases. Specifically, the specified region Rs has a conical shape with an arbitrary point on the virtual laser sensor 3A as its vertex. Therefore, the cross-sectional shape of the specified region Rs (the cross-sectional shape of the plane M cut perpendicular to the center line CL of the virtual laser L0) is circular. However, the cross-sectional shape of the specified region Rs is not limited to a circular shape and may be polygonal. In other words, the specified region Rs may have a pyramidal shape with an arbitrary point on the virtual laser sensor 3A as its vertex.

[0053] Next, the arithmetic processing unit 24 extracts the position coordinates of multiple reflection points P located within the defined region Rs, as shown in Figure 9 (procedure S104). Here, for example, the raycast tracing function of a game engine is used to extract the multiple reflection points P contained within the defined region Rs. In this case, the LiDAR ray irradiation direction vector corresponds to the center line CL of the virtual laser L0.

[0054] The arithmetic processing unit 24 extracts the position coordinates of multiple reflection points P that are continuously located within a specified region Rs along the center line CL over the entire irradiation distance (corresponding to the length of the center line CL) of the finite virtual laser L0.

[0055] Specifically, as shown in Figure 9(a), if the virtual laser L0 emitted from the virtual laser sensor 3A hits only the point cloud PG corresponding to object 7P, and does not hit the point cloud PGs corresponding to objects 7Q and 7R which are located between the virtual laser sensor 3A and the point cloud PG corresponding to object 7P, then only the multiple reflection points P within the specified region Rs of the point cloud PG corresponding to object 7P will be acquired, and the reflection points P of the point cloud PGs corresponding to objects 7Q and 7R will not be acquired. In Figure 9, black circles represent acquired reflection points P, and white circles represent reflection points P that are not acquired.

[0056] Furthermore, as shown in Figure 9(b), when the virtual laser L0 emitted from the virtual laser sensor 3A hits the point cloud PG corresponding to object 7P and the point cloud PG corresponding to object 7S, which is located between the virtual laser sensor 3A and the point cloud PG corresponding to object 7P, multiple reflection points P that exist within the specified region Rs among the point cloud PGs corresponding to objects 7P and 7S are acquired. In this case, reflection points P that are shielded by the point cloud PG corresponding to object 7S among the point cloud PG corresponding to object 7P are not acquired.

[0057] Next, the arithmetic processing unit 24 calculates the average of the position coordinates of multiple reflection points P located within the specified region Rs, and uses the average of the position coordinates of each reflection point P as the estimated coordinate of the target point cloud position F (procedure S105).

[0058] Specifically, as shown in Figure 9, the coordinates of an arbitrary point on the virtual laser sensor 3A are (x0, y0, z0), and the coordinates of the multiple reflection points P located within the specified region Rs are (x1, y1, z1) ~ (x n ,y n ,z n If we assume that (x^, y^, z^) is the estimated coordinate of the target point cloud position, which is the average of the coordinates of each reflection point P, then the X coordinate value of the estimated coordinate of the target point cloud position F is expressed by the following formula.

number

[0059] Although omitted here, the Y and Z coordinate values ​​of the estimated coordinates of the target point cloud position F are calculated in the same manner.

[0060] Next, the arithmetic processing unit 24 calculates the distance from the virtual laser sensor 3A to the target point cloud position F based on the estimated coordinates of the target point cloud position F (procedure S106). Specifically, if the coordinates of an arbitrary point on the virtual laser sensor 3A are (x0, y0, z0) and the estimated coordinates of the target point cloud position are (x^, y^, z^), then the distance D from the virtual laser sensor 3A to the target point cloud position F is expressed by the following formula.

number

[0061] Next, the arithmetic processing unit 24 determines whether the virtual travel of the virtual mobile object 2A to the destination has been completed (procedure S107). If the arithmetic processing unit 24 determines that the virtual travel of the virtual mobile object 2A to the destination has not been completed, it repeats the above procedures S102 to S106.

[0062] When the arithmetic processing unit 24 determines that the virtual movement of the virtual mobile body 2A to the destination has been completed, it determines whether the virtual mobile body 2A is able to move by determining the accuracy of the estimation of the virtual mobile body 2A's own position based on the distance from the virtual laser sensor 3A to the target point cloud position F on the virtual travel path S0 (procedure S108).

[0063] The virtual laser sensor 3A is designed to mimic a real laser sensor 3 with a finite measurement range. Therefore, for example, in environments where there are few point clouds PG corresponding to objects 7 surrounding the virtual mobile object 2A, and the number of target point cloud positions F acquired by the virtual laser sensor 3A is small, the accuracy of the virtual mobile object 2A's self-position estimation tends to become unstable, and the virtual mobile object 2A may become unable to move.

[0064] The arithmetic processing unit 24 then outputs the determination result to the display unit 23 (procedure S109). As a result, the determination result is displayed on the display unit 23. The arithmetic processing unit 24 displays on the display unit 23, for example, locations in the point cloud PG where the estimation accuracy of the virtual mobile object 2A's own position tends to be unstable, along with whether the virtual mobile object 2A is able to move.

[0065] In the above, the point cloud acquisition unit 25 executes procedure S101. The reflection point extraction unit 26 executes procedures S102 to S104. The distance calculation unit 27 executes procedures S105 and S106. The determination unit 28 executes procedures S107 and S108. The output unit 29 executes procedure S109.

[0066] Figure 10 is a flowchart showing a conventional method for conducting autonomous driving simulations. In Figure 10, first, an operator goes to the site and acquires point cloud data using a laser sensor 3 with a point cloud acquisition device 10 (step S111).

[0067] Afterward, the worker returns from the site and, with the point cloud data displayed on the display unit 23 of the PC 21, manually places 3D CG data while looking at the screen of the display unit 23 to create a virtual space model that reproduces the user environment (process S112).

[0068] Then, the operator performs an automated driving simulation using the laser sensor model and the virtual space model (step S113). The simulation results are displayed on the display unit 23.

[0069] Then, the worker looks at the judgment result on the display unit 23 and provides feedback (process S114). For example, if it is determined that there are locations where the accuracy of estimating the self-position of the virtual mobile body 2A tends to be unstable, countermeasures such as installing a reflective plate or other distinctive object at the relevant location can be taken.

[0070] Conventionally, after acquiring point cloud data using the laser sensor 3, a virtual space model is created by manually arranging 3D CG data. Consequently, the amount of work required to create the virtual space model becomes substantial.

[0071] To address such challenges, in this embodiment, while the virtual mobile body 2A is virtually driven along a predetermined virtual travel path S0, a virtual laser L0 is irradiated from the virtual laser sensor 3A towards the target point cloud PG, thereby calculating the distance from the virtual laser sensor 3A to the target point cloud position F. Based on this distance from the virtual laser sensor 3A to the target point cloud position F, the accuracy of the self-position estimation of the virtual mobile body 2A is determined.

[0072] By using the point cloud data directly to calculate the laser sensor model and perform an autonomous driving simulation, the manual process of arranging 3D CG data (S112 in Figure 10) becomes unnecessary, as shown in Figure 11.

[0073] However, as shown in Figure 12, when calculating the distance from the virtual laser sensor 3A to the target point cloud position F by obtaining the position coordinates of the reflection point P located on the LiDAR ray irradiation direction vector V, if the ray irradiation direction vector V passes between the reflection points P in the point cloud PG, the position coordinates of the reflection point P cannot be accurately obtained.

[0074] Therefore, in this embodiment, first, point cloud data including the position coordinates of multiple reflection points P that constitute the point cloud PG of an object 7 existing in the real environment to be simulated is acquired. Then, while a virtual mobile body 2A corresponding to the mobile body 2 is virtually driven, a virtual laser L0 having a defined three-dimensional region Rs is emitted from a virtual laser sensor 3A corresponding to the laser sensor 3 toward the target point cloud PG, and the position coordinates of multiple reflection points P existing within the defined three-dimensional region Rs are extracted. Then, based on the position coordinates of the multiple reflection points P existing within the defined region Rs, the distance from the virtual laser sensor 3A to the position of the target point cloud PG (target point cloud position F) is calculated. In this way, since the automatic driving simulation is performed using the point cloud data of the real environment to be simulated as is, there is no need to manually place 3D CG data. This reduces the man-hours required to create the virtual space model.

[0075] Furthermore, for example, when a one-dimensional (linear) virtual laser L0 is emitted from the virtual laser sensor 3A towards the target point cloud PG, the one-dimensional virtual laser L0 may pass between reflection points P. Therefore, in this embodiment, the virtual laser sensor 3A emits a virtual laser L0 having a three-dimensional defined region Rs. When the virtual laser L0 is emitted from the virtual laser sensor 3A, the position coordinates of multiple reflection points P existing within the three-dimensional defined region Rs are extracted, and the distance from the virtual laser sensor 3A to the target point cloud position F is calculated. As a result, even if the virtual laser L0 passes between reflection points P without hitting any of them, other reflection points P existing within the defined region Rs are extracted, ensuring that reflection points P are available for use in calculating the distance from the virtual laser sensor 3A to the target point cloud position F. Therefore, even when performing a simulation using point cloud data as is, it is possible to extract the position coordinates of multiple reflection points P and calculate the distance from the virtual laser sensor 3A to the target point cloud position F, just as when performing a simulation using a virtual space model with 3D CG data.

[0076] Furthermore, in this embodiment, since point cloud data is used directly, autonomous driving simulations can be performed quickly. As a result, the lead time required for considering the introduction of a driving control device that autonomously drives the mobile body 2 along the driving path S, and for considering changes to the driving path S, can be shortened, leading to the acceleration of the introduction of autonomous vehicles and the like into user environments.

[0077] Furthermore, advancements in technologies such as drones and photogrammetry have made it easier to acquire point cloud data, and methods for automatically generating surface information from point cloud data have been developed. However, there are quality issues such as missing surfaces and the generation of unnecessary surfaces. In this embodiment, since the simulation is performed using the point cloud data as is, there is no need to automatically generate surface information from the point cloud data.

[0078] Furthermore, in this embodiment, the defined region Rs is set such that the area of ​​the plane M cut perpendicular to the center line CL of the virtual laser L0 increases as the distance from the virtual laser sensor 3A increases. Therefore, even when the point cloud PG is far from the virtual laser sensor 3A, multiple reflection points P will exist within the defined region Rs, ensuring that there are enough reflection points P to be used in calculating the distance from the virtual laser sensor 3A to the target point cloud position F.

[0079] Furthermore, in this embodiment, the accuracy of the self-position estimation of the virtual moving object 2A can be easily determined by calculating the distance from the virtual laser sensor 3A to the target point cloud position F based on the position coordinates of multiple reflection points P located within a defined region Rs.

[0080] Furthermore, in this embodiment, the distance from the virtual laser sensor 3A to the target point cloud position F can be determined by a simple calculation formula by calculating the average value of the position coordinates of multiple reflection points P located within a specified region Rs.

[0081] As a variation, when calculating the estimated coordinates of the target point cloud position F, instead of using the average value of the position coordinates of multiple reflection points P located within the specified region Rs, the shortest distance between the center line CL of the virtual laser L0 emitted from the virtual laser sensor 3A and the position coordinates of multiple reflection points P located within the specified region Rs may be calculated. The distance from the virtual laser sensor 3A to the target point cloud position F may then be calculated by weighting the shortest distance between the center line CL of the virtual laser L0 and the position coordinates of each reflection point P. For example, the shorter the shortest distance between the center line CL of the virtual laser L0 and the position coordinates of the reflection points P, the larger the weighting coefficient should be set.

[0082] In this case, the coordinates of the virtual laser sensor 3A are (x0, y0, z0), and the coordinates of the multiple reflection points P located within the specified region Rs are (x1, y1, z1) ~ (x n ,y n ,z n ) and the weight coefficients of the multiple reflection points P located within the specified region Rs are w1~w n Assuming that the estimated coordinates of the target point cloud position F are (x^, y^, z^), the X-coordinate value of the estimated coordinates of the target point cloud position F is expressed by the following formula.

number

[0083] Although omitted here, the Y and Z coordinate values ​​of the estimated coordinates of the target point cloud position F are calculated in the same manner.

[0084] In this modified example, a weight is taken into account based on the shortest distance between the center line CL of the virtual laser L0 and the position coordinates of multiple reflection points P located within the specified region Rs. As a result, the distance from the virtual laser sensor 3A to the target point cloud position F can be determined with high accuracy.

[0085] Figure 13 is a schematic block diagram showing the configuration of a simulation system according to another embodiment of the present invention, and corresponds to Figure 6.

[0086] In Figure 13, in the simulation system 20 of this embodiment, the point cloud data stored in the point cloud data memory 12 includes not only the position coordinates of the multiple reflection points P that constitute the point cloud PG, but also the reflection intensities of the multiple reflection points P. The reflection intensity of the reflection points P is the reflection intensity of the laser L when the laser L irradiated from the laser sensor 3 is reflected off the surface of the object 7, and is detectable by the laser sensor 3.

[0087] Furthermore, the simulation system 20 includes a calculation processing unit 24A instead of the calculation processing unit 24 in the embodiment described above. The calculation processing unit 24A includes the point cloud acquisition unit 25, the reflection point extraction unit 26, the distance calculation unit 27A, the determination unit 28, and the output unit 29.

[0088] The distance calculation unit 27A calculates the distance from the virtual laser sensor 3A to the target point cloud position F based on the position coordinates and reflection intensities of multiple reflection points P located within a specified region Rs. The reflection intensities of the multiple reflection points P are selected by the reflection point extraction unit 26 from among the reflection intensities of all reflection points P acquired by the point cloud acquisition unit 25.

[0089] Figure 14 is a flowchart showing the procedure of the arithmetic processing performed by the arithmetic processing unit 24A, and corresponds to Figure 8.

[0090] In Figure 14, the arithmetic processing unit 24A, similar to the embodiment described above, executes the above steps S101 to S104, and then selects the reflection intensities of multiple reflection points P located within a specified region Rs from among the reflection intensities of all reflection points P constituting the point cloud PG acquired by the point cloud acquisition unit 25 (step 105A).

[0091] Next, the arithmetic processing unit 24A calculates the distance from the virtual laser sensor 3A to the target point cloud position F based on the position coordinates and reflection intensity of multiple reflection points P located within the specified region Rs (procedure S106A).

[0092] For example, the arithmetic processing unit 24A calculates the distance from the virtual laser sensor 3A to the target point cloud position F by weighting the shortest distance between the center line CL of the virtual laser L0 and the position coordinates of multiple reflection points P located within the specified region Rs, as described above, and by weighting the reflection intensity of the multiple reflection points P located within the specified region Rs.

[0093] In this case, as described above, the arithmetic processing unit 24A sets the weighting coefficient to be larger as the shortest distance between the center line CL of the virtual laser L0 and the position coordinates of the reflection point P decreases. Furthermore, the arithmetic processing unit 24A sets the weighting coefficient to be larger as the reflection intensity increases.

[0094] Next, the arithmetic processing unit 24A executes the above steps S107 to S109 in the same manner as in the embodiment described above.

[0095] In the above configuration, the distance calculation unit 27A executes procedures S105A and S106A. Other functions are the same as in the embodiment described above.

[0096] In this embodiment, since not only the position coordinates of multiple reflection points P located within the specified region Rs but also the reflection intensity of each reflection point P are considered, the distance from the virtual laser sensor 3A to the target point cloud position F can be determined with even greater accuracy.

[0097] In this embodiment, the distance from the virtual laser sensor 3A to the target point cloud position F is calculated by setting a weighting coefficient corresponding to the shortest distance between the center line CL of the virtual laser L0 and the position coordinates of multiple reflection points P located within a defined region Rs, and a weighting coefficient corresponding to the reflection intensity of the multiple reflection points P located within the defined region Rs. However, the embodiment is not limited to this configuration. For example, similar to the embodiment described above, the average value of the position coordinates of multiple reflection points P located within a defined region Rs may be calculated as the estimated coordinates of the target point cloud position F, and the distance from the virtual laser sensor 3A to the target point cloud position F may be calculated based on the estimated coordinates of the target point cloud position F and a weighting coefficient corresponding to the reflection intensity of the multiple reflection points P located within the defined region Rs.

[0098] Furthermore, if the laser sensor 3 detects the color (RGB, etc.) of the laser L, the point cloud data stored in the point cloud data memory 12 may include not only the reflection intensity of the laser L reflected by multiple reflection points P, but also the color information of the laser L reflected by multiple reflection points P. In this case, the distance from the virtual laser sensor 3A to the target point cloud position F may be calculated based on the position coordinates, reflection intensity, and color information of multiple reflection points P located within a defined region Rs.

[0099] Figure 15 is a schematic block diagram showing the configuration of a simulation system according to yet another embodiment of the present invention, and corresponds to Figure 6.

[0100] In Figure 15, the simulation system 20 of this embodiment includes a calculation processing unit 24B instead of the calculation processing unit 24 in the embodiment described above. The calculation processing unit 24B includes the point cloud acquisition unit 25, the reflection point extraction unit 26, the matching unit 30, the determination unit 28B, and the output unit 29.

[0101] The matching unit 30 compares the point cloud data acquired by the point cloud acquisition unit 25 with the position coordinates of multiple reflection points P extracted by the reflection point extraction unit 26. Specifically, the matching unit 30 compares the position coordinates of all reflection points P constituting the point cloud PG acquired by the point cloud acquisition unit 25 with the position coordinates of multiple reflection points P extracted by the reflection point extraction unit 26 to extract the three-dimensional shape of the target point cloud PG and determine the position and orientation of the target point cloud PG with respect to the virtual laser sensor 3A. The position coordinates of the reflection points P extracted by the reflection point extraction unit 26 are the position coordinates of the reflection points P when a predetermined position of the virtual laser sensor 3A is taken as the origin.

[0102] The determination unit 28B determines whether the virtual mobile object 2A is capable of moving by determining the accuracy of the self-position estimation of the virtual mobile object 2A based on the matching results from the matching unit 30. Specifically, the determination unit 28B uses the relative position and orientation of the target point cloud PG obtained by the matching unit 30 as the self-position estimation value of the virtual mobile object 2A, and determines the accuracy of the self-position estimation of the virtual mobile object 2A by comparing this self-position estimation value of the virtual mobile object 2A with a predetermined virtual travel path S0.

[0103] In this embodiment, by comparing the position coordinates of the reflection point P acquired by the point cloud acquisition unit 25 with the position coordinates of the reflection point P extracted by the reflection point extraction unit 26, the self-position estimation accuracy of the virtual moving object 2A can be easily determined, similar to the embodiment described above.

[0104] It should be noted that the present invention is not limited to the above embodiments. For example, in the above embodiments, the defined region Rs is set such that the area of ​​the plane M cut perpendicular to the center line CL of the virtual laser L0 increases continuously as the distance from the virtual laser sensor 3A increases, but the invention is not limited to such a form. The defined region Rs may be set such that the area of ​​the plane M cut perpendicular to the center line CL of the virtual laser L0 increases in steps as the distance from the virtual laser sensor 3A increases. In other words, the defined region Rs may have a cylindrical or prismatic shape in which the volume increases in steps as the distance from the virtual laser sensor 3A increases.

[0105] Furthermore, the specified region Rs may be set such that the area of ​​the plane M cut perpendicular to the center line CL of the virtual laser L0 remains constant, regardless of the distance from the virtual laser sensor 3A. In other words, the specified region Rs may have a cylindrical or prismatic shape with a constant volume.

[0106] Furthermore, in the above embodiment, the reflection point extraction unit 26 virtually drives the virtual mobile body 2A along a predetermined virtual travel path S0 determined based on the point cloud data acquired by the point cloud acquisition unit 25. However, the method for determining the virtual travel path S0 is not limited to using point cloud data acquired by the point cloud acquisition unit 25. For example, the reflection point extraction unit 26 may determine the virtual travel path S0 from previously obtained point cloud data, or it may determine the virtual travel path S0 without using point cloud data.

[0107] Furthermore, in the above embodiment, point cloud data of objects 7 surrounding the mobile body 2 is acquired using a point cloud acquisition device 10 mounted on the mobile body 2 while the mobile body 2 is driven by manual operation. However, the system is not limited to this configuration. For example, a laser sensor 3 and a point cloud acquisition controller 11 may be mounted on a trolley, and point cloud data of objects 7 surrounding the trolley may be acquired while an operator pushes the trolley to move it. In addition, if point cloud data of objects existing in the real environment to be simulated has been measured and acquired by someone other than the user, the user may obtain that point cloud data.

[0108] Furthermore, in the above embodiment, the distance to objects 7 surrounding the moving body 2 is detected by irradiating the moving body 2 with a laser L using the laser sensor 3 and receiving the reflected light of the laser L. However, the sensor used for the movement of the moving body is not limited to the laser sensor 3, and ultrasonic sensors, cameras, etc., may also be used. When an ultrasonic sensor is used, ultrasonic waves are emitted towards the moving body 2 as a distance measurement signal. For example, when a ToF camera is used as the camera, infrared light is emitted towards the moving body 2 as a distance measurement signal.

[0109] Furthermore, while the simulation system 20 of the above embodiment is a system that uses a laser sensor 3 to estimate the self-position of the mobile body 2 and performs a simulation of automatic driving in which the mobile body 2 automatically moves, the present invention can be applied to simulations of sensors used for the movement of the mobile body 2, other than self-position estimation.

[0110] The present invention can also be applied to simulations that detect obstacles present around a virtual moving object 2A, for example. For example, in the simulation system 20 shown in Figure 15, the determination unit 28B of the arithmetic processing unit 24B estimates the distance between the virtual moving object 2A and the obstacle from the relative position and orientation of the target point cloud PG obtained by the comparison unit 30, and determines the obstacle detection accuracy by comparing the distance to the obstacle with a predetermined virtual travel path S0. The point cloud PG obtained by the point cloud acquisition unit 25 is a point cloud of obstacles. In this case, the detection accuracy of obstacles present around the virtual moving object 2A can be easily determined by comparing the position coordinates of the reflection point P obtained by the point cloud acquisition unit 25 with the position coordinates of the reflection point P extracted by the reflection point extraction unit 26.

[0111] Furthermore, the present invention is not limited to simulations of autonomous driving, but can also be applied to simulations of manual driving. [Explanation of Symbols]

[0112] 2...Moving object, 2A...Virtual moving object, 3...Laser sensor (sensor), 3A...Virtual laser sensor (virtual sensor), 7...Object, 12...Point cloud data memory (point cloud data acquisition unit), 20...Simulation system, 25...Point cloud acquisition unit (point cloud data acquisition unit), 26...Reflection point extraction unit, 27,27A...Distance calculation unit, 30...Verification unit, F...Target point cloud position, L...Laser, L0...Virtual laser (virtual signal), CL...Centerline, M...Plane, P...Reflection point, PG...Point cloud, Rs...Specified area.

Claims

1. A simulation system for simulating sensors used in the movement of a mobile object, A point cloud data acquisition unit acquires point cloud data including the position coordinates of multiple reflection points that constitute the point cloud of an object existing in the real environment being simulated, A reflection point extraction unit extracts the position coordinates of multiple reflection points located within the three-dimensional defined region when a virtual signal having a three-dimensional defined region is emitted from a virtual sensor corresponding to the sensor toward the target point cloud while a virtual mobile body corresponding to the mobile body is virtually driven, A simulation system comprising: a distance calculation unit that calculates the distance from the virtual sensor to the position of the target point cloud based on the position coordinates of the plurality of reflection points extracted by the reflection point extraction unit.

2. The simulation system according to claim 1, wherein the defined region is a region centered on the center line of the virtual signal, and is set such that the area of ​​the surface cut perpendicular to the center line increases as the distance from the virtual sensor increases.

3. The simulation system according to claim 1, wherein the distance calculation unit calculates the average value of the position coordinates of the plurality of reflection points extracted by the reflection point extraction unit as the estimated coordinates of the position of the target point cloud, and calculates the distance from the virtual sensor to the position of the target point cloud based on the estimated coordinates.

4. The aforementioned defined region is the region centered on the center line of the virtual signal, The simulation system according to claim 1, wherein the distance calculation unit calculates the shortest distance between the center line of the virtual signal and the position coordinates of the plurality of reflection points, and calculates the distance from the virtual sensor to the position of the target point cloud by applying weights according to the shortest distance between the center line and the position coordinates of the reflection points.

5. The point cloud data further includes the reflection intensity of the plurality of reflection points, The simulation system according to claim 1, wherein the distance calculation unit calculates the distance from the virtual sensor to the position of the target point cloud based on the position coordinates and reflection intensity of the plurality of reflection points.

Images