Simulation system, simulation method, and simulation program

Abstract

The present invention makes it possible to confirm the operation of a moving body moving in real space without using the moving body. This simulation system performs an operation simulation of a moving body. The simulation system is provided with a virtual space generation unit, an object generation unit, and an operation generation unit. The virtual space generation unit generates a virtual space 60 which corresponds to real space and in which a virtual moving body 61 corresponding to the moving body is disposed. The object generation unit generates, in the virtual space 60, a virtual object corresponding to an object operating in real space. The operation generation unit generates a virtual operation of the virtual moving body 61 in relation to the virtual object.

Description

[Technical Field]

[0001] The technology disclosed herein relates to a simulation system, a simulation method, and a simulation program. [Background technology]

[0002] Patent Document 1 discloses an information processing device that provides a user with a virtual image. The information processing device includes an imaging device that generates a captured image by capturing an image of real space, and a controller that generates an output image by superimposing a virtually generated virtual object on the captured image. The virtual object superimposed on the captured image as real space moves in response to a user's operation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-149712 Summary of the Invention

[0004] Incidentally, the operation of a moving object moving in real space is confirmed by preparing an actual moving object and operating the actual object. For such operation confirmation, it is necessary to prepare an actual moving object. As a method for confirming the operation of a moving object without preparing an actual object, a method using a virtual image as described above is also conceivable. However, simply operating a virtual moving object in response to user operations as in Patent Document 1 makes it difficult to simulate the actual operation of the moving object, and the operation of the moving object cannot be confirmed.

[0005] The technology disclosed herein has been developed in consideration of these points, and its purpose is to enable confirmation of the movement of a moving object without moving the moving object in real space.

[0006] The simulation system disclosed herein is a simulation system that performs a motion simulation of a moving body, and includes a virtual space generation unit that generates a virtual space that corresponds to real space and in which a virtual moving body corresponding to the moving body is placed, an object generation unit that generates a virtual object in the virtual space that corresponds to an object moving in the real space, and an action generation unit that generates virtual actions of the virtual moving body relative to the virtual object.

[0007] The simulation system disclosed herein is a simulation system that performs a motion simulation of a moving body, and includes a virtual space generation unit that generates a virtual space that corresponds to real space and in which a virtual moving body corresponding to the moving body is placed, a detection calculation unit that calculates the detection results of a virtual sensor that is provided on the virtual moving body and detects virtual objects around the virtual moving body, and an action generation unit that generates virtual actions of the virtual moving body based on the detection results of the virtual sensor.

[0008] The simulation method disclosed herein is a simulation method for simulating the movement of a moving body, and includes generating a virtual space that corresponds to real space and in which a virtual moving body corresponding to the moving body is placed, generating a virtual object in the virtual space that corresponds to an object moving in the real space, and generating a virtual movement of the virtual moving body relative to the virtual object.

[0009] The simulation method disclosed herein is a simulation method for simulating the movement of a moving body, and includes generating a virtual space that corresponds to real space and in which a virtual moving body corresponding to the moving body is placed, calculating the detection results of a virtual sensor that is provided on the virtual moving body and detects virtual objects around the virtual moving body, and generating virtual movement of the virtual moving body based on the detection results of the virtual sensor.

[0010] The simulation program disclosed herein is a simulation program for simulating the movement of a moving body, and causes a computer to realize the following functions: generating a virtual space that corresponds to real space and in which a virtual moving body corresponding to the moving body is placed; generating a virtual object in the virtual space that corresponds to an object moving in the real space; and generating a virtual movement of the virtual moving body relative to the virtual object.

[0011] The simulation program disclosed herein is a simulation program for simulating the operation of a moving body, and includes a function for generating a virtual space that corresponds to real space and in which a virtual moving body corresponding to the moving body is placed, a function for calculating the detection results of a virtual sensor that is provided on the virtual moving body and detects virtual objects around the virtual moving body, and a function for generating virtual operation of the virtual moving body based on the detection results of the virtual sensor. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of the simulation system. [Figure 2] FIG. 2 is a schematic diagram showing a moving object in real space. [Figure 3] FIG. 3 is a perspective view of the moving body. [Figure 4] FIG. 4 is a block diagram showing a schematic hardware configuration of a mobile object. [Figure 5] FIG. 5 is a block diagram showing a schematic hardware configuration of the simulation system. [Figure 6] FIG. 6 is a functional block diagram showing a schematic software configuration of the control device. [Figure 7] FIG. 7 is a diagram of the virtual space generated by the simulation system. [Figure 8] FIG. 8 is a diagram showing the detection results of the virtual sensor. [Figure 9] FIG. 9 is a detailed functional block diagram of the action generation unit. [Figure 10] FIG. 10 is a functional block diagram showing a schematic software configuration of the display device. [Figure 11] FIG. 11 is a virtual image displayed by the simulation system. [Figure 12] FIG. 12 is a perspective view of the real space during the operation simulation. [Figure 13] FIG. 13 is a flowchart of the operation simulation. DETAILED DESCRIPTION OF THE INVENTION

[0013] An exemplary embodiment will be described in detail below with reference to the drawings. FIG. 1 is a schematic diagram of a simulation system 100. FIG. 2 is a schematic diagram showing a moving object 9 in a real space 80. The simulation system 100 performs a motion simulation of the moving object 9. The moving object 9 is an object that moves in a real space 80 where installed objects 82 such as walls or pillars may exist. The simulation system 100 realizes the motion simulation by simulating the operation of a virtual moving object corresponding to the moving object 9 in a virtual space corresponding to the real space 80. In this example, the moving object 9 moves autonomously in the real space 80, i.e., autonomously travels. During autonomous travel, the moving object 9 travels while avoiding, for example, installed objects 82 and people 83. The simulation system 100 causes the virtual moving object to autonomously travel in the virtual space in the same way as the moving object 9.

[0014] The simulation system 100 provides a user with a virtual image of a virtual moving object performing an action simulation using cross reality technology (XR technology). The simulation system 100 includes a control device 1 and a display device 3. The control device 1 generates simulated actions of the virtual moving object in a virtual space. The display device 3 provides a user with a virtual image including the virtual moving object performing the action generated by the control device 1. For example, the virtual image provided by the display device 3 is a mixed reality (MR) image. The display device 3 provides a mixed reality image in which the virtual moving object is superimposed on a real space 80.

[0015] First, the mobile body 9 will be described. FIG. 3 is a perspective view of the mobile body 9. FIG. 4 is a block diagram showing a schematic hardware configuration of the mobile body 9. The mobile body 9 in this example is a transport robot that transports articles. More specifically, the mobile body 9 is a transport robot that can travel autonomously. The mobile body 9 has a main body 90, a traveling unit 91, a driving unit 92, a sensor 94, a memory unit 95, and a controller 96.

[0016] An article can be mounted on the main body 90. In this example, an accommodation space capable of accommodating an article is formed inside the main body 90.

[0017] The running unit 91 propels the main body 90. In this example, the running unit 91 has left and right drive wheels 91a and left and right driven wheels 91b. The left and right drive wheels 91a are provided at the rear of the main body 90. The left and right driven wheels 91b are provided at the front of the main body 90.

[0018] The drive unit 92 drives the running unit 91. In this example, the drive unit 92 includes a left motor that rotationally drives the left drive wheel 91a and a right motor that rotationally drives the right drive wheel 91a. The rotational speeds of the left and right drive wheels 91a can be changed individually by changing the rotational speeds of the left and right motors. The left and right motors drive the left and right drive wheels 91a at the same rotational speed, causing the running unit 91 to run straight. The left and right motors drive the left and right drive wheels 91a at different rotational speeds, causing the running unit 91 to turn (for example, change direction). Note that the mechanism for turning the running unit 91 is not limited to this. For example, the drive unit 92 may achieve turning by driving the drive wheels 91a or the driven wheels 91b to rotate about a pivot extending in the vertical direction.

[0019] The sensor 94 detects objects around the mobile object 9 (hereinafter simply referred to as "peripheral objects"). In the present disclosure, "objects" include both inanimate objects and living objects. Peripheral objects may include an installed object 82 (see FIG. 2) installed in the real space 80 and an object that moves in the real space (hereinafter referred to as "moving object"). The moving object is, for example, a person 83 or another robot. The sensor 94 is provided, for example, on the main body 90. In this example, the sensor 94 is a distance measurement sensor that measures the distance from the sensor 94 to the peripheral object. Specifically, the sensor 94 is a LiDAR (Light Detection and Ranging) sensor. The sensor 94 has, for example, a light-emitting unit that emits laser light toward the periphery of the mobile object 9 and a light-receiving unit that receives the laser light reflected off the surface of the peripheral object. The sensor 94 measures the flight time of the laser light emitted from the light-emitting unit, hitting the surface of the peripheral object, and returning to the light-receiving unit. The sensor 94 measures the distance from the sensor 94 to the surface of a surrounding object based on the measured time of flight. The sensor 94 calculates point cloud data based on the measured distance. The point cloud data is three-dimensional position information of the surface of the surrounding object. The sensor 94 outputs the calculated point cloud data to the controller 96. When the mobile body 9 is autonomously traveling, it repeatedly detects surrounding objects at a predetermined detection cycle. Each time the sensor 94 detects a surrounding object, it outputs the detection result of the sensor 94, i.e., the point cloud data, to the controller 96.

[0020] The memory 95 stores various programs and various data. The data stored in the memory 95 includes a first map. The first map is a three-dimensional map of the real space 80. The first map is based on a real coordinate system, which is a coordinate system set in the real space 80. Hereinafter, in the description of the real space, when the term "position" is simply mentioned, it means an absolute position in the real coordinate system. The first map includes the positions of the installed objects 82. Furthermore, the first map includes first location information. The first location information includes the starting point and the arrival point of the mobile body 9 when traveling autonomously. The first map is used to control the autonomous traveling. The first map is stored in the memory 95 in advance.

[0021] The controller 96 reads the control program stored in the memory 95 and comprehensively controls each part of the mobile object 9. The controller 96 realizes autonomous driving by controlling the drive unit 92 based on the detection results of the sensor 94. In this example, the controller 96 realizes autonomous driving by controlling the drive unit 92 based on the first map and the detection results of the sensor 94.

[0022] The controller 96 calculates a global route based on the first map. The global route is a provisional and general route used when the mobile body 9 travels autonomously. The global route is a travel route from the starting point of the mobile body 9 to the destination point. The controller 96 stores the calculated global route in the memory 95.

[0023] The controller 96 acquires point cloud data from the sensor 94 and calculates the relative positions of surrounding objects from the mobile body 9 based on the acquired point cloud data. The surrounding objects may include the installed object 82 and the operating object.

[0024] The controller 96 calculates the position and attitude of the moving body 9 in the real coordinate system, which are estimated based on the relative position of the installed object 82 from the moving body 9 detected by the sensor 94. Hereinafter, these positions and attitudes of the moving body 9 will be simply referred to as "estimated positions." For example, the controller 96 determines the relative position of the installed object 82 from the moving body 9 based on the detection results of the sensor 94, and compares the relative position of the installed object 82 with the absolute position of the installed object 82 defined in the first map, thereby determining the absolute position and attitude of the moving body 9 in the real coordinate system based on the absolute position of the installed object 82. The absolute position and attitude of the moving body 9 determined in this way are the estimated position.

[0025] The controller 96 calculates a small area path based on the global path, the relative positions of surrounding objects from the mobile body 9, and the estimated position. The small area path is a deterministic path of the mobile body 9 generated based on the global path. Specifically, the controller 96 corrects the global path so that the mobile body 9 avoids contact with the installed objects 82 and the moving objects. The controller 96 generates a small area path that reduces the error between the estimated position and the global path, approaches the arrival point, and does not contact the installed objects 82 and the moving objects. The small area path is a movement path of a predetermined distance over a relatively short period of time.

[0026] The controller 96 calculates a command value for the drive unit 92 based on the calculated small area path. The command value is a command value input to the drive unit 92 to realize the traveling of the mobile object 9 from the start point to the end point of the small area path. Specifically, the command value is the rotation speed of the left motor and the rotation speed of the right motor.

[0027] The controller 96 outputs the calculated command value to the drive unit 92 to cause the traveling unit 91 to travel, thereby causing the mobile object 9 to travel along the small area route.

[0028] The controller 96 repeatedly performs the above-mentioned operations of acquiring point cloud data, calculating the relative positions of surrounding objects from the mobile body 9, calculating the estimated position of the mobile body 9, calculating the small area path, calculating command values, and outputting the command values ​​to the driving unit 92. As a result, the mobile body 9 repeatedly travels along the small area path and travels autonomously. The controller 96 stops the driving unit 92 when the calculated estimated position coincides with the arrival point. This ends the autonomous travel of the mobile body 9.

[0029] Next, the control device 1 and the display device 3 will be described in detail. Fig. 5 is a block diagram showing a schematic hardware configuration of the simulation system 100. The control device 1 generates a virtual space corresponding to the real space 80, a virtual installation corresponding to the installation 82, a virtual object corresponding to the operating object, a virtual moving body corresponding to the moving body 9, and a virtual action which is the action of the virtual moving body in the virtual space. The control device 1 is, for example, a computer. The control device 1 has a memory 11, a controller 12, and a communicator 13.

[0030] The storage device 11 stores various programs and various data and is formed of, for example, a magnetic disk such as a hard disk, an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a semiconductor memory.

[0031] The memory 11 stores a second map. The second map is a three-dimensional map in a virtual space. The second map is based on a virtual coordinate system, which is a coordinate system set in the virtual space. Unless otherwise specified, in the following description of the virtual space 60, when the term "position" is simply used, it means an absolute position in the virtual coordinate system. The virtual coordinate system corresponds to the real coordinate system. The second map includes the positions of virtual installations in the virtual space. Furthermore, the second map includes second location information. The second location information includes the starting point and the arrival point of the moving body 9 when autonomously traveling. The second location information may further include waypoints for the moving body 9 when autonomously traveling. The second map is used to generate the virtual space, virtual objects, virtual installations, virtual moving bodies, and virtual actions. The second map is stored in the memory 11 in advance.

[0032] The memory 11 further stores sensor performance. The sensor performance includes the detection error, resolution, and detectable range of the sensor 94. The sensor performance is used to generate the virtual operation. Note that the sensor performance may include only one or only two of the detection accuracy, resolution, and detectable range of the sensor 94.

[0033] The controller 12 reads and executes various programs stored in the memory 11. The controller 12 includes various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and / or a DSP (Digital Signal Processor), and various semiconductor memories such as a VRAM (Video Random Access Memory), a RAM (Random Access Memory), and / or a ROM (Read Only Memory).

[0034] The controller 12 reads the control program from the memory 11, controls each part of the control device 1 in an integrated manner, and generates a virtual space, a virtual object, a virtual moving body, and a virtual action.

[0035] The communicator 13 wirelessly transmits and receives signals to and from the display device 3. For example, the communicator 13 transmits the virtual action of the virtual moving object generated by the controller 12 to the display device 3.

[0036] The display device 3 generates a virtual image including a virtual moving object that performs virtual operations, and displays the generated virtual image. The display device 3 is a head mounted display that can be worn on the head of a user 81. As shown in FIG. 5 , the display device 3 has a display unit 31, a position detector 32, a memory unit 33, a controller 34, and a communication unit 35.

[0037] The display 31 displays a virtual image. In this example, the display 31 is an optical see-through display. The display 31 has a lens 31a (see FIG. 1) that is placed in the line of sight of a user 81 wearing the display device 3. The display 31 displays an image of a virtual moving object on the lens 31a, thereby providing a virtual image in which the virtual moving object is superimposed on the real space 80.

[0038] The position detector 32 detects the position and orientation of the display device 3, i.e., the position and orientation of the user 81 wearing the display device 3 (more specifically, the head of the user 81) in the real space 80. Hereinafter, the position and orientation of the user 81 will be simply referred to as the "user position." In this example, the display device 3 is an inside-out head-mounted display, and the position detector 32 detects the position of the display device 3 using sensors provided on the display device 3. The position detector 32 includes, for example, a gyro sensor that detects the angular velocity of the display device 3, an acceleration sensor that detects the acceleration of the display device 3, and a geomagnetic sensor that detects the geomagnetism at the position of the display device 3. The position detector 32 detects the user's position in the real coordinate system by cumulatively adding movement vectors obtained from the detection results of the gyro sensor, acceleration sensor, and geomagnetic sensor with respect to a reference point set in the real coordinate system. The position detector 32 is an example of a position detector that detects the position of an operating object operating in the real space 80.

[0039] The storage unit 33 stores various programs and various data. The storage unit 33 is formed, for example, by a magnetic disk such as a hard disk, an optical disk such as a CD or DVD, or a semiconductor memory. The storage unit 33 stores object definition data that defines objects included in the virtual image. In this example, the storage unit 33 stores object definition data of a virtual moving body.

[0040] The controller 34 reads and executes various programs stored in the memory 33. The controller 34 includes various processors such as a CPU, a GPU, and / or a DSP, and various semiconductor memories such as a VRAM, a RAM, and / or a ROM.

[0041] The controller 34 reads the control program from the storage device 33 and comprehensively controls each part of the display device 3. The controller 34 generates a virtual image and causes the display device 31 to display the generated virtual image.

[0042] The communicator 35 transmits and receives signals to and from the control device 1 (more specifically, the communicator 13) via wireless communication. For example, the communicator 35 transmits the user position detected by the position detector 32 to the control device 1. For example, the communicator 35 receives a virtual action transmitted from the control device 1.

[0043] 6 is a functional block diagram showing a schematic software configuration of the control device 1. The controller 12 has, as functional blocks, a virtual space generator 21, an object generator 22, a detection calculation unit 23, and an action generator 24. The functions of these functional blocks are realized by the controller 12 reading a control program from the memory 11.

[0044] FIG. 7 is a diagram showing an example of a virtual space 60 generated by the simulation system 100. The virtual space generation unit 21 generates the virtual space 60. The virtual space generation unit 21 generates the virtual space 60 based on a second map. The virtual space 60 is a space corresponding to the real space 80. Specifically, the virtual space 60 is a space in which virtual installations 63 corresponding to installations 82 in the real space 80 are arranged. That is, the virtual space generation unit 21 further generates the virtual installations 63 in the virtual space 60. The virtual space generation unit 21 generates the virtual installations 63 in the virtual space 60 at positions defined based on the second map. As a result, virtual walls and pillars having shapes similar to those of the walls and pillars in the real space 80 are formed as the virtual installations 63 at positions corresponding to those of the walls and pillars in the virtual space 60.

[0045] Furthermore, the virtual space generation unit 21 generates a virtual moving body 61 in the virtual space 60. The virtual moving body 61 corresponds to the moving body 9. The virtual moving body 61 has the same configuration and function as the moving body 9. For example, the virtual moving body 61 has a virtual main body corresponding to the main body 90, a virtual running unit corresponding to the running unit 91, a virtual driving unit corresponding to the driving unit 92, and a virtual sensor corresponding to the sensor 94.

[0046] Similar to the sensor 94, the virtual sensor detects virtual objects (hereinafter referred to as "virtual peripheral objects") that exist around the virtual moving body 61 in the virtual space 60. The virtual sensor is provided, for example, on the virtual main body. The virtual sensor has the same performance as the sensor 94. For example, the detection accuracy of the virtual sensor matches the detection accuracy of the sensor 94. The resolution of the virtual sensor matches the resolution of the sensor 94. The detection range of the virtual sensor based on the virtual main body matches the detection range of the sensor 94 based on the main body 90.

[0047] The generated virtual moving body 61 is placed in the virtual space 60. For example, the initial position of the virtual moving body 61 is the starting point defined in the second location information. The initial attitude of the virtual moving body 61 is also defined as the attitude at the starting point in the second location information. Thereafter, the position and attitude of the virtual moving body 61 in the virtual space 60 are updated according to the virtual movement. Hereinafter, the position and attitude of the virtual moving body 61 in the virtual space 60 will be simply referred to as the "virtual position."

[0048] The object generation unit 22 generates a virtual object in the virtual space 60 generated by the virtual space generation unit 21, the virtual object corresponding to the moving object in the real space 80. In this example, the virtual object is a virtual user 62 corresponding to the user 81. That is, the virtual object generated by the object generation unit 22 corresponds to a person moving in the real space 80.

[0049] Based on the detection result of the position detector 32, the object generation unit 22 generates a virtual user 62 at a position in the virtual space 60 that corresponds to the user's position in the real space 80. Specifically, the object generation unit 22 acquires the detection result of the position detector 32, i.e., the user's position in the real space 80. Based on the correspondence between the real coordinate system and the virtual coordinate system, the object generation unit 22 converts the user's position in the real space 80 into a user's position in the virtual space 60. The object generation unit 22 generates the virtual user 62 at the user's position in the virtual space 60. In other words, the position of the virtual user 62 in the virtual space 60 is the position obtained by converting the user's position in the real space 80 into the virtual space 60. The virtual user 62 is, for example, a cylindrical model.

[0050] The detection calculation unit 23 calculates the detection results of the virtual sensors that detect virtual peripheral objects. The virtual peripheral objects may include a virtual installation 63 and a virtual user 62.

[0051] As described above, the detection range of the virtual sensor is set based on the virtual moving object 61. The detection calculation unit 23 moves the detection range of the virtual sensor according to the virtual position of the virtual moving object 61. When the virtual user 62 or the virtual installation 63 is present within the detection range of the virtual sensor, the detection calculation unit 23 determines that the virtual sensor has detected the virtual user 62 or the virtual installation 63.

[0052] In this example, the detection calculation unit 23 calculates point cloud data (hereinafter referred to as "virtual point cloud data") 64 of virtual peripheral objects that are determined to have been detected by the virtual sensor as the detection result of the virtual sensor.

[0053] FIG. 8 is a diagram showing the position of virtual point cloud data 64 relative to the virtual space 60. Note that in FIG. 8, in order to clarify the position of the virtual point cloud data 64, not only the virtual point cloud data 64 but also the virtual installation 63, the virtual moving body 61, and the virtual user 62 are shown. The virtual point cloud data 64 corresponds to point cloud data acquired by the sensor 94. The virtual point cloud data 64 is three-dimensional position information of the surface of the virtual peripheral object. The virtual point cloud data 64 indicates the relative position of the virtual peripheral object with respect to the virtual moving body 61. The virtual point cloud data 64 may include virtual point cloud data 64a of the virtual installation 63 and virtual point cloud data 64b of the virtual user 62.

[0054] Specifically, the detection calculation unit 23 calculates the detection results of the virtual sensor, i.e., virtual point cloud data 64a, based on the virtual position of the virtual moving body 61, the position of the virtual installation 63, the user position, and sensor performance. The relative distance of the virtual installation 63 from the virtual moving body 61 is calculated from the virtual position of the virtual moving body 61 and the position of the virtual installation 63. The detection calculation unit 23 calculates the relative distance of the virtual installation 63 reflecting a sensing error caused by the sensor performance based on this relative distance and the sensor performance, and generates the virtual point cloud data 64a based on the relative distance of the virtual installation 63 reflecting the sensing error. For example, the detection calculation unit 23 may multiply the relative distance of the virtual installation 63 by a coefficient randomly selected from a value between 0.9 and 1.1 to calculate the relative distance of the virtual installation 63 reflecting the sensing error. Alternatively, the detection calculation unit 23 may obtain the relative distance of the virtual installation object 63 reflecting the sensing error by discarding any digits of the relative distance of the virtual installation object 63 that are smaller than the number of digits corresponding to the resolution of the sensor 94. Alternatively, when the relative distance of the virtual installation object 63 is not within the detectable range of the sensor 94, the detection calculation unit 23 may calculate virtual point cloud data 64a reflecting the sensing error as if the virtual sensor had not detected the virtual installation object 63. For example, the virtual point cloud data 64a may be positioned at a position shifted from the surface of the virtual installation object 63 by the amount of the reflected sensing error, as shown in FIG. 8 .

[0055] The detection calculation unit 23 generates the virtual point cloud data 64b in the same manner as the virtual point cloud data 64a. That is, the detection calculation unit 23 calculates the virtual point cloud data 64b based on the virtual position of the virtual moving body 61, the position of the virtual user 62, and sensor performance. The relative distance of the virtual user 62 from the virtual moving body 61 is calculated from the virtual position of the virtual moving body 61 and the position of the virtual user 62. The detection calculation unit 23 calculates the relative distance of the virtual user 62 reflecting the sensing error based on this relative distance and sensor performance, and generates the virtual point cloud data 64b based on the relative distance of the virtual user 62 reflecting the sensing error. The calculation method of the relative distance of the virtual user 62 reflecting the sensing error is the same as for the virtual point cloud data 64a. For example, the virtual point cloud data 64b may be positioned at a position shifted from the surface of the cylindrical virtual user 62 by an amount corresponding to the reflected sensing error, as shown in FIG. 8.

[0056] The action generation unit 24 generates a virtual action of the virtual moving body 61 in the virtual space 60. The virtual action is the same as the action of the moving body 9 in the real space 80. In this example, the virtual action is autonomous traveling of the virtual moving body 61 in the virtual space 60.

[0057] The virtual action generated by the action generation unit 24 is the action of the virtual moving body 61 corresponding to the virtual object. That is, the virtual moving body 61 performs an action in response to the virtual object as a virtual action. In the example of autonomous traveling, when the virtual moving body 61 approaches a virtual object, it travels in a manner that avoids contact with the virtual object.

[0058] The motion generation unit 24 generates a virtual motion based on the detection result of the virtual sensor calculated by the detection calculation unit 23. In this example, the motion generation unit 24 generates a path for the virtual moving body 61 in the virtual space 60 based on the detection result of the virtual sensor, and moves the virtual moving body 61 along the generated path. At this time, the motion generation unit 24 corrects the path of the virtual moving body 61 based on the detection result of the virtual sensor. The motion generation unit 24 calculates a command value (hereinafter referred to as a "virtual command value") to a virtual driving unit based on the generated path, and moves the virtual moving body 61 based on the calculated virtual command value. This movement, i.e., displacement, of the virtual moving body 61 is the virtual motion.

[0059] 9 is a detailed functional block diagram of the motion generation unit 24. The motion generation unit 24 includes a global path calculation unit 24a, an object position calculation unit 24b, an estimated position calculation unit 24c, a small area path calculation unit 24d, a command value calculation unit 24e, and a dynamic model 24f.

[0060] The global route calculation unit 24a calculates a virtual global route. The virtual global route corresponds to the global route calculated by the controller 96. The virtual global route is a provisional and general route used when the virtual moving body 61 travels autonomously. The virtual global route is a travel route from the departure point to the arrival point of the virtual moving body 61. The global route calculation unit 24a calculates the virtual global route based on the second map stored in the memory 11. Specifically, the global route calculation unit 24a calculates, as the virtual global route, a route for the virtual moving body 61 from the departure point to the arrival point so that the virtual moving body 61 does not interfere with the virtual installation 63, based on the positions and sizes of the virtual moving body 61 and the virtual installation 63, etc. Note that if there is a via point between the departure point and the arrival point, the virtual global route will be a route that passes through the via point.

[0061] The object position calculation unit 24b calculates the relative position (hereinafter referred to as the "detected position") of a virtual peripheral object detected by the virtual sensor with respect to the virtual moving body 61. The virtual peripheral object may include a virtual user 62 and a virtual installation 63. The object position calculation unit 24b calculates the detected position of the virtual peripheral object based on the virtual point cloud data 64 calculated by the detection calculation unit 23. As described above, the virtual point cloud data 64a reflects sensor performance and may be positioned at a position offset from the surface of the virtual installation 63. Therefore, the detected position of the virtual installation 63 is not the relative position of the virtual installation 63 with respect to the virtual moving body 61 defined based on the second map, but the position when the virtual sensor detects the virtual installation 63 positioned at a position defined based on the second map. Similarly, the detected position of the virtual user 62 is not the relative position of the virtual user 62 with respect to the virtual moving body 61 set based on the user position in the real space 80, but the position when the virtual sensor detects the virtual user 62 positioned at a position set based on the user position in the real space 80.

[0062] The estimated position calculation unit 24c calculates the position and orientation of the virtual moving body 61 in a virtual coordinate system, which are estimated based on the detection results of the virtual sensors. Hereinafter, the position and orientation of the virtual moving body 61 are simply referred to as the "virtual estimated position." The estimated position calculation unit 24c calculates the relative position of the virtual installation 63 from the virtual moving body 61 based on the detection results of the virtual sensors. The second map contains information on the absolute position of the virtual installation 63 in the virtual coordinate system. The estimated position calculation unit 24c compares the relative position of the virtual installation 63 from the virtual moving body 61 with the absolute position of the virtual installation 63 defined in the second map, thereby calculating the absolute position and orientation of the virtual moving body 61 based on the absolute position of the virtual installation 63. The absolute position and orientation of the virtual moving body 61 calculated in this way are the virtual estimated position. As described above, the relative position of the virtual installation 63 calculated by the object position calculation unit 24b may include sensing errors due to the sensor performance of the virtual sensors. Therefore, the virtual estimated position may deviate from the virtual position, which is the absolute position of the virtual moving object 61 in the virtual space 60.

[0063] The small area path calculation unit 24d calculates a virtual small area path. The virtual small area path corresponds to the small area path calculated by the controller 96. The virtual small area path is a deterministic path of the virtual moving object 61 that is generated based on the virtual global path. The small area path calculation unit 24d calculates the virtual small area path based on the virtual global path, the detected position of the virtual peripheral object, and the virtual estimated position.

[0064] Specifically, the small area path calculation unit 24d corrects the virtual large area path so that the virtual moving body 61 avoids contact with the virtual user 62 and the virtual installation 63. The small area path calculation unit 24d generates a virtual small area path that reduces the error from the virtual estimated position to the virtual large area path, approaches the destination point, and does not contact the virtual user 62 or the virtual installation 63. The virtual small area path is a movement path of a predetermined distance over a relatively short period of time. In this way, the small area path calculation unit 24d generates the virtual small area path little by little.

[0065] However, the starting point of the virtual small area path calculated by the small area path calculation unit 24d is not the virtual position of the virtual moving object 61 but the virtual estimated position. In other words, although the virtual moving object 61 is actually placed at the virtual position, the virtual small area path is generated from the virtual estimated position.

[0066] The command value calculation unit 24e calculates a virtual command value. The virtual command value corresponds to a command value to the drive unit 92 calculated by the controller 96. In other words, the virtual command value is a command value input to the virtual drive unit to realize travel of the virtual moving object 61 from the start point to the end point of the virtual small area path. The command value calculation unit 24e calculates the virtual command value based on the virtual small area path calculated by the small area path calculation unit 24d.

[0067] The dynamic model 24f generates a virtual movement. The dynamic model 24f is a motion model of the moving body 9. The dynamic model 24f generates a virtual movement based on a virtual command value calculated by the command value calculation unit 24e. In this example, the dynamic model 24f includes a drive unit model 24g and a vehicle body motion model 24h.

[0068] The driver model 24g is a motion model of the driver 92 modeled by mathematical modeling. The driver model 24g receives a virtual command value from the command value calculation unit 24e as an input and outputs a driving force of a virtual driver (hereinafter referred to as a "virtual driving force"). The virtual driving force corresponds to the driving force that the driver 92 outputs when a command value identical to the virtual command value is input from the controller 96 to the driver 92.

[0069] The vehicle body motion model 24h is a motion model of the main body 90 modeled by mathematical modeling. The vehicle body motion model 24h calculates the behavior of the main body 90 when a driving force equal to the virtual driving force is applied to the traveling unit 91. The vehicle body motion model 24h receives the virtual driving force from the driving unit model 24g as an input and outputs the displacement, i.e., movement, of the virtual moving body 61. The displacement output from the vehicle body motion model 24h corresponds to the displacement of the moving body 9 when the driving unit 92 generates a driving force equal to the virtual driving force. In detail, the vehicle body motion model 24h outputs, as virtual motion, the position and posture of the virtual moving body 61 after displacement when a virtual driving force is applied to the virtual moving body 61 located at a virtual position in the virtual space 60.

[0070] As described above, the virtual small area path is generated based on the virtual estimated position, and the virtual driving force is calculated based on the virtual small area path. Meanwhile, the dynamic model 24f moves the virtual moving object 61 located at the virtual position with the virtual driving force calculated based on the virtual small area path. Therefore, the virtual movement of the virtual moving object 61 may include sensing errors due to the sensor performance of the virtual sensor.

[0071] The motion generation unit 24 outputs the position and posture of the virtual moving body 61 calculated by the dynamic model 24f as virtual motion. The motion generation unit 24 updates the virtual position of the virtual moving body 61 to the calculated position and posture. By repeating this process, the motion generation unit 24 generates motion of the virtual moving body 61 that moves generally along a virtual global path while avoiding contact with the virtual user 62 and virtual installations 63 in the virtual space 60.

[0072] Furthermore, the action generation unit 24 periodically transmits the virtual position and posture of the virtual moving object 61 as the virtual action to the display device 3.

[0073] 10 is a functional block diagram showing a schematic software configuration of the display device 3. The controller 34 has, as functional blocks, an image generation unit 41 and a display unit 42. The functions of these functional blocks are realized by the controller 34 reading a control program from the storage unit 33.

[0074] FIG. 11 shows a virtual image 7 displayed by the simulation system 100. The image generation unit 41 generates the virtual image 7 including a virtual moving object 61 performing a virtual action. Specifically, the image generation unit 41 generates, as the virtual image 7, a mixed reality image in which the virtual moving object 61 is superimposed on a real space 80 seen by the user 81 through the lens 31a. The image generation unit 41 generates the virtual image 7 based on the virtual action generated by the action generation unit 24. Specifically, the image generation unit 41 generates an image of the virtual moving object 61. The image generation unit 41 acquires the virtual action generated by the action generation unit 24, i.e., the virtual position and orientation of the virtual moving object 61, and the user position in the real space 80 detected by the position detector 32. The image generation unit 41 converts the position and orientation of the virtual moving object 61 in the virtual space 60 into the position and orientation of the virtual moving object 61 in the real space 80 based on the correspondence between the real coordinate system and the virtual coordinate system. The image generation unit 41 generates an image of the virtual moving body 61 when the virtual moving body 61 is viewed in the real space 80 from the user's position as the starting point, based on the object definition data of the virtual moving body 61.

[0075] The display unit 42 displays the virtual image 7 generated by the image generation unit 41 on the display device 31. In this example, the display unit 42 provides the virtual image 7 to the user 81 by displaying the image of the virtual moving object 61 generated by the image generation unit 41 on the lens 31a of the display device 31. Therefore, the virtual image 7 displayed by the display device 31 may include not only the virtual moving object 61 but also an object 82 in the real space 80 that is visible through the lens 31a. The display unit 42 displays the image of the virtual moving object 61 on the lens 31a of the display device 31 so that the virtual moving object 61 is positioned in the same orientation as the virtual moving object 61 at a position in the real space 80 that corresponds to the virtual position of the virtual moving object 61 in the virtual space 60.

[0076] Next, the motion simulation will be described in detail. Fig. 12 is a schematic diagram showing an example of a real space 80 during the motion simulation. As shown in Fig. 12, the motion simulation is performed with the display device 3 attached to the head of a user 81. The motion simulation is started, for example, by the user 81 operating the control device 1 or the display device 3. Note that although a virtual moving object 61 is shown by a dashed line in Fig. 12, the virtual moving object 61 does not actually exist in the real space 80.

[0077] 13 is a flowchart of the operation simulation. When the operation simulation starts, in step S11, the virtual space generation unit 21 generates a virtual space 60, a virtual installation 63, and a virtual moving object 61 based on the second map. At this time, the virtual space generation unit 21 generates the virtual moving object 61 at a position in the virtual space 60 that corresponds to the starting point of the second location information. The virtual space generation unit 21 also transmits a command to the display device 3 via the communication device 13 to start detection by the position detector 32.

[0078] When the display device 3 receives a command to start detection by the position detector 32 via the communication device 35, the position detector 32 starts detecting the user's position in the real space 80 in step S21.

[0079] In step S22, the position detector 32 determines whether a predetermined detection period has arrived. The detection period is the detection period of the position detector 32. If the detection period has not arrived, the position detector 32 waits by repeating the process of step S22 until the detection period arrives.

[0080] When the detection period arrives, the position detector 32 detects the user position in step S23, and transmits the detected user position to the control device 1 via the communication device 35.

[0081] Meanwhile, in step S12, the object generation unit 22 of the control device 1 determines whether a predetermined calculation period has arrived. The calculation period is the period of calculation by the detection calculation unit 23. For example, the calculation period matches the detection period of the position detector 32. If the calculation period has not arrived, the object generation unit 22 waits by repeating step S12 until the calculation period arrives. Note that the calculation period of the detection calculation unit 23 may be longer or shorter than the detection period of the position detector 32.

[0082] When the calculation cycle arrives, the object generation unit 22 generates a virtual user 62 in the virtual space 60 based on the user position detected by the position detector 32 in step S13.

[0083] Next, in step S14, the detection calculation unit 23 calculates virtual point cloud data 64, which is the detection result of the virtual sensor, based on the position and orientation of the virtual moving object 61 in the virtual space 60.

[0084] In step S15, the motion generation unit 24 calculates a virtual command value based on the virtual point cloud data 64 calculated by the detection calculation unit 23. Specifically, if a virtual global path has not yet been calculated, the global path calculation unit 24a calculates a virtual global path based on the second map. The motion generation unit 24 stores the calculated virtual global path in the memory 11. The object position calculation unit 24b calculates the detected position of the virtual peripheral object based on the virtual point cloud data 64. The estimated position calculation unit 24c calculates the absolute position of the virtual moving body 61 in the virtual coordinate system as a virtual estimated position based on the relative position of the virtual installation 63 from the virtual moving body 61 and the absolute position of the virtual installation 63 on the second map. The small area path calculation unit 24d calculates a virtual small area path based on the relative positions of the virtual global path and the virtual peripheral object and the virtual estimated position. The command value calculation unit 24e calculates a virtual command value based on the virtual small area path.

[0085] In step S16, the movement generation unit 24 generates a virtual movement of the virtual moving body 61 based on the virtual command value. Specifically, the drive unit model 24g calculates a virtual driving force based on the virtual command value. The vehicle body motion model 24h calculates the position and posture of the virtual moving body 61 after displacement based on the virtual driving force. The movement generation unit 24 updates the virtual position and posture of the virtual moving body 61 in the virtual space 60 to the position and posture after displacement. The movement generation unit 24 transmits the virtual movement, i.e., the virtual position and posture of the virtual moving body 61 after displacement, to the display device 3 via the communication device 13.

[0086] When the display device 3 receives the virtual action from the control device 1 via the communication device 35, the image generation unit 41 generates an image of the virtual moving body 61 after displacement based on the virtual action, the object definition data of the virtual moving body 61, and the user position in step S24.

[0087] In step S25, the display unit 42 causes the display device 31 to display the virtual image 7 including the image of the virtual moving object 61 generated by the image generation unit 41.

[0088] After generating the virtual action, the action generation unit 24 of the control device 1 determines whether the autonomous traveling of the virtual moving body 61 has ended in step S17. Specifically, if the virtual estimated position and the arrival point match, the action generation unit 24 determines that the autonomous traveling of the virtual moving body 61 has ended, and if the virtual estimated position and the arrival point do not match, the action generation unit 24 determines that the autonomous traveling of the virtual moving body 61 has not ended. If the autonomous traveling of the virtual moving body 61 has not ended, the process returns to step S12. That is, the control device 1 repeatedly executes the processes of steps S12 to S17 to repeatedly generate virtual actions. As a result, the virtual moving body 61 in the virtual space 60 travels autonomously.

[0089] After step S25, the process of the display device 3 returns to step S22. That is, the display device 3 repeatedly executes the processes of steps S22 to S25 to repeatedly detect the user's position in the real space 80 and generate and display the virtual image 7. As a result, the virtual image 7 in which the autonomously traveling virtual moving object 61 is superimposed on the real space 80 is provided to the user 81 by the display device 31.

[0090] The calculation of the virtual global path by the motion generator 24 in step S15 is performed only when a virtual command value is calculated for the first time in the motion simulation. That is, when calculating a virtual command value for the second time or later, the motion generator 24 uses the virtual global path stored in the memory 11. The calculation of the virtual global path by the motion generator 24 only needs to be performed before the virtual command value is calculated for the first time in the motion simulation, and does not have to be performed in step S15.

[0091] In this example, the process in step S11 corresponds to generating a virtual space that corresponds to the real space and in which a virtual moving object corresponding to the moving object is placed. The processes in steps S23 and S13 correspond to generating a virtual object in the virtual space that corresponds to an object moving in the real space. The process in step S14 corresponds to calculating the detection result of a virtual sensor that is provided on the virtual moving object and detects virtual objects around the virtual moving object. The processes in steps S15 and S16 correspond to generating a virtual action of the virtual moving object with respect to the virtual object. The processes in steps S15 and S16 also correspond to generating a virtual action of the virtual moving object based on the detection result of the virtual sensor.

[0092] In this example, the simulation system 100 can virtually operate a virtual moving body 61 corresponding to the moving body 9 in a virtual space 60 in which virtual objects corresponding to objects operating in a real space 80 have been generated. At this time, the virtual moving body 61 can be caused to perform a virtual action corresponding to the virtual object. Therefore, the user 81 can confirm the action of the moving body 9 reacting to the object operating in the real space 80 by using the virtual moving body 61 without using the actual moving body 9.

[0093] Therefore, for example, in the development stage of the mobile body 9, it is possible to check the operation of the mobile body 9 without manufacturing a prototype of the mobile body 9. This reduces the development cost and development time of the mobile body 9, and also allows for flexible changes to the specifications of the mobile body 9. For example, it becomes possible to proceed with the development of the mobile body 9 by examining improvements to the mobile body 9 and carrying out development design, etc., without manufacturing a prototype of the mobile body 9.

[0094] Furthermore, the user 81 can safely check the safety of the moving body 9 by checking the virtual movement of the virtual moving body 61. For example, the safety of the moving body 9 approaching and colliding with surrounding objects can be checked without actually moving the moving body 9, which is a real machine. In a safety evaluation using a real machine, if the real machine collides with a surrounding object, there is a risk that damage will occur to the real machine or the surrounding object. In other words, even if a situation occurs that makes the safety evaluation unacceptable, no actual damage will occur if the virtual moving body 61 is placed virtually. As a result, the safety of the moving body 9 can be safely checked.

[0095] Furthermore, after the product of the mobile object 9 is completed, even if the actual mobile object 9 is not present at the site, the operation of the mobile object 9 can be demonstrated by the virtual mobile object 61. In other words, there is no need to transport the actual mobile object 9 to the demonstration site, which improves the efficiency of sales promotion.

[0096] Furthermore, in this example, the simulation system 100 calculates the detection results of a virtual sensor that detects a virtual object in the virtual space 60, and generates a virtual movement based on the calculated detection results of the virtual sensor. Therefore, the user 81 can confirm the virtual movement of the virtual moving body 61, thereby simulating the movement of the moving body 9 that moves autonomously based on the detection results of the sensor 94.

[0097] In particular, in this example, the virtual sensor corresponds to a sensor 94 provided on the moving body 9 and detecting objects around the moving body 9. Therefore, the virtual movement of the virtual moving body 61 becomes closer to the movement of the moving body 9 provided with the sensor 94, and the movement of the moving body 9 can be accurately reproduced. In other words, the simulation system 100 can perform a movement simulation of the moving body 9 that reflects the sensor performance of the sensor 94. Therefore, the user 81 can more appropriately grasp the actual movement of the moving body 9.

[0098] Furthermore, the virtual object generated in the virtual space 60 corresponds to a person moving in the real space 80. Therefore, the user 81 can appropriately understand the unpredictable behavior of the moving object 9 relative to a person.

[0099] Furthermore, the simulation system 100 generates a virtual object in the virtual space 60 based on the detection result of the position detector 32 that detects the position of an object moving in the real space 80. Therefore, the movement of the object moving in the real space 80 can be reflected in the movement of the virtual object in real time. Therefore, the user 81 can more appropriately check the movement of the moving object 9.

[0100] The virtual image 7 is a mixed reality image in which the virtual moving object 61 is superimposed on the real space 80. Therefore, the user 81 can visually recognize the virtual moving object 61 displayed as if it were moving in the real space 80, and can appropriately confirm the movement of the moving object 61.

[0101] In this example, the display device 3 that displays the virtual image 7 is a head-mounted display. Therefore, by checking the virtual image 7 displayed on the display device 3 worn on the head, the user 81 can properly check the movement of the moving object 9, with the feeling that the user 81 is actually in the space represented by the virtual image 7. In particular, when the display device 3 is provided with a position detector 32 as in this example, the user 81 can check the virtual image 7 including the virtual moving object 61 that virtually moves in response to the movement of the user 81.

[0102] As described above, the simulation system 100 is a simulation system that performs a motion simulation of a moving body 9, and includes a virtual space generation unit 21 that generates a virtual space 60 that corresponds to the real space 80 and in which a virtual moving body 61 corresponding to the moving body 9 is placed, an object generation unit 22 that generates a virtual user 62 (virtual object) in the virtual space 60 that corresponds to a user 81 (object) moving in the real space 80, and an action generation unit 24 that generates virtual actions of the virtual moving body 61 for the virtual user 62 in the virtual space 60.

[0103] In other words, the simulation method is a simulation method for simulating the operation of a moving body 9, and includes generating a virtual space 60 corresponding to a real space 80, generating a virtual user 62 (virtual object) in the virtual space 60 corresponding to a user 81 (object) operating in the real space 80, and generating virtual operations of the virtual moving body 61 for the virtual user 62.

[0104] In other words, the simulation program is a simulation program that simulates the operation of a moving body 9, and causes a computer to realize the following functions: generating a virtual space 60 corresponding to a real space 80; generating a virtual user 62 (virtual object) in the virtual space 60 corresponding to a user 81 (object) operating in the real space 80; and generating virtual operations of the virtual moving body 61 against the virtual user 62.

[0105] According to these configurations, it is possible to virtually move the virtual moving body 61 corresponding to the moving body 9 in the virtual space 60 in which the virtual user 62 corresponding to the user 81 moving in the real space 80 is generated. Therefore, by checking the virtual movement of the virtual moving body 61, the user 81 can check the movement of the moving body 9 moving relative to the user 81 moving in the real space 80 without actually moving the moving body 9.

[0106] The simulation system 100 further includes a detection calculation unit 23 that calculates the detection results of a virtual sensor that detects the virtual user 62 (virtual object), and the movement generation unit 24 generates a virtual movement based on the detection results of the virtual sensor.

[0107] According to this configuration, the virtual movement of the virtual moving object 61 can be a movement of autonomously moving based on the detection result of the sensor 94. Therefore, the user 81 can appropriately grasp the movement of the moving object 9 that moves autonomously based on the detection result of the sensor 94.

[0108] The virtual sensor corresponds to a sensor 94 that is provided on the moving body 9 and detects objects around the moving body 9.

[0109] According to this configuration, the virtual movement of the virtual moving object 61 becomes closer to the movement of the moving object 9 provided with the sensor 94. Therefore, the user 81 can more appropriately grasp the movement of the moving object 9 that moves based on the detection result of the sensor 94.

[0110] Moreover, the virtual object generated by the object generation unit 22 corresponds to a person moving in the real space 80.

[0111] According to this configuration, the user 81 can appropriately understand the unpredictable behavior of the moving object 9 toward people.

[0112] In addition, the simulation system 100 further includes a position detector 32 that detects the position of a user 81 (object) in the real space 80, and the object generation unit 22 generates a virtual user 62 (virtual object) in the virtual space 60 based on the detection result of the position detector 32.

[0113] According to this configuration, the movement of the virtual user 62 can be made closer to the movement of the user 81 moving in the real space 80. Therefore, the user 81 can more appropriately confirm the movement of the moving object 9 moving relative to the user 81 moving in the real space 80.

[0114] The simulation system 100 further includes a head-mounted display that provides a virtual image 7 including a virtual moving body 61 that performs a virtual action, and the position detector 32 detects the position of the head-mounted display in the real space 80, thereby detecting the position of the user 81 wearing the head-mounted display in the real space 80, and the object generation unit 22 generates a virtual user 62 corresponding to the user 81 at a position in the virtual space 60 that corresponds to the position of the user 81 based on the detection result of the position detector 32.

[0115] According to this configuration, by wearing the head-mounted display, the user 81 can check the virtual image 7 provided by the head-mounted display and can confirm the movement of the moving object 9. In addition, the user 81 can check the virtual image 7 including the virtual moving object 61 that moves virtually in response to the movement of the user 81.

[0116] The simulation system 100 further includes an image generation unit 41 that generates a virtual image 7 including a virtual moving object 61 that performs a virtual action.

[0117] According to this configuration, the user 81 can easily visually confirm the virtual movement of the virtual moving object 61.

[0118] The virtual image 7 is a mixed reality image in which a virtual moving object 61 is superimposed on a real space 80.

[0119] According to this configuration, the user 81 can see the virtual moving object 61 displayed as if it were moving in the real space 80, and can appropriately confirm the movement of the moving object 9.

[0120] The simulation system 100 further includes a display device 3 that displays a virtual image 7 .

[0121] According to this configuration, the user 81 can check the virtual image 7 displayed on the display device 3 and confirm the operation of the moving object 9.

[0122] The display device 3 is a head-mounted display.

[0123] According to this configuration, the user 81 can check the virtual image 7 displayed on the display device 3 attached to the head, and can properly check the movement of the moving object 9, with the feeling that the user 81 is actually present in the space represented by the virtual image 7.

[0124] The simulation system 100 is a simulation system that performs a motion simulation of a moving body 9, and includes a virtual space generation unit 21 that generates a virtual space 60 that corresponds to the real space 80 and in which a virtual moving body 61 corresponding to the moving body 9 is placed, a detection calculation unit 23 that calculates the detection results of a virtual sensor provided in the virtual moving body 61 that detects virtual objects around the virtual moving body 61, and an action generation unit 24 that generates virtual actions of the virtual moving body 61 based on the detection results of the virtual sensor.

[0125] In other words, the simulation method is a simulation method for simulating the operation of a moving body 9, and includes generating a virtual space 60 that corresponds to the real space 80 and in which a virtual moving body 61 corresponding to the moving body 9 is placed, calculating the detection results of a virtual sensor provided on the virtual moving body 61 that detects virtual objects around the virtual moving body 61, and generating virtual operations of the virtual moving body 61 based on the detection results of the virtual sensor.

[0126] In other words, it is a simulation program for simulating the operation of a moving body 9, and includes a function for generating a virtual space 60 that corresponds to the real space 80 and in which a virtual moving body 61 corresponding to the moving body 9 is placed, a function for calculating the detection results of a virtual sensor provided on the virtual moving body 61 that detects virtual objects around the virtual moving body 61, and a function for generating virtual operations of the virtual moving body 61 based on the detection results of the virtual sensor.

[0127] According to these configurations, it is possible to virtually operate a virtual moving body 61 corresponding to the moving body 9 in the virtual space 60. Therefore, by checking the virtual operation of the virtual moving body 61, the user 81 can appropriately check the operation of the moving body 9 operating in the real space 80 without using the moving body 9. In detail, the user 81 can appropriately check the operation of the moving body 9 moving based on the detection result of a sensor 94 that detects an object in the real space 80.

[0128] Other Embodiments As described above, the above embodiment has been described as an example of the technology disclosed in this application. However, the technology of the present disclosure is not limited to this and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, the components described in the above embodiment can be combined to create new embodiments. Furthermore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to exemplify the technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately determining that these non-essential components are essential.

[0129] The motion simulation executed by the simulation system 100 is not limited to the autonomous driving of the mobile object 9. For example, the motion simulation may be the autonomous driving of the mobile object 9 following a person detected by a sensor.

[0130] The moving body 9 that is the subject of the operation simulation is not limited to a transport robot, and may be an autonomous moving device other than a transport robot. Examples of autonomous moving devices other than transport robots include guide robots, nursing robots, patrol robots, humanoid robots, industrial robots, medical robots, automobiles, drones, construction machinery, and jet skis. The moving body 9 that is the subject of the operation simulation may be a moving device other than an autonomous moving device.

[0131] The sensor 94 is not limited to a LiDAR sensor. For example, the sensor 94 may be a camera, a global positioning system (GPS), or the like. The sensor 94 is not limited to a sensor that detects objects around the moving body 9. For example, the sensor 94 may be a sensor that detects other physical quantities such as the length, mass, time, or speed of the moving body 9 or an object other than the moving body 9. For example, the sensor 94 may be a sensor provided on an object other than the moving body 9. The same applies to a virtual sensor corresponding to the sensor 94.

[0132] The control device 1 may be a single computer or may be formed by multiple computers.

[0133] The display device 3 may be a tablet, a stationary display, a projector, etc. The simulation system 100 does not necessarily have to include the display device 3.

[0134] The display 31 may be a video-through display that displays the virtual moving object 61 on an image of the real space 80 captured by a camera.

[0135] The position detector 32 may use only an acceleration sensor and a gyro sensor, or only an acceleration sensor and a geomagnetic sensor, among an acceleration sensor, a gyro sensor, and a geomagnetic sensor. The position detector 32 may also be a radar, a GPS (Global Positioning System), a camera, an infrared sensor, or the like. The position detector 32 is not limited to being provided in the display device 3. For example, the position detector 32 may be an outside-in type detector that detects the position of the display device 3 using a sensor provided separately from the display device 3. The position detector 32 is not limited to being a detector that detects the position of the display device 3. For example, the position detector 32 may be a detector that directly detects a moving object using a sensor provided on a wall, a pillar, a ceiling, a floor, or an installation attached to these. The position detector 32 may be omitted.

[0136] The object generation unit 22 may generate multiple virtual users 62 in the virtual space 60. In this case, the simulation system 100 includes multiple display devices 3. The object generation unit 22 generates virtual users 62 corresponding to users 81 wearing each display device 3 in the common virtual space 60. The action generation unit 24 generates virtual actions of the virtual moving object 61 for the multiple virtual users 62. In this case, each user 81 can see how the virtual moving object 61 moves while avoiding other users 81.

[0137] The virtual object generated by the object generation unit 22 is not limited to the virtual user 62. For example, the virtual object may be a person other than the virtual user 62. The virtual object is not limited to a person. For example, the virtual object may be a robot separate from the moving body 9. The object generation unit 22 can be omitted. In this case, for example, the virtual space generation unit 21 may generate a virtual object in the virtual space 60 that moves based on a preset movement. The movement set for the virtual object is determined, for example, by measuring in advance the movement of an operating object that moves in the real space 80.

[0138] The detection calculation unit 23 may calculate the virtual point cloud data 64a of the virtual installation 63 based on the position and orientation of the virtual moving body 61 and the position of the virtual installation 63 in the virtual space 60 without using the sensor performance.

[0139] The movement generating unit 24 may generate the virtual movement based on the virtual command value without using the dynamic model 24f.

[0140] The virtual image 7 generated by the image generation unit 41 is not limited to a mixed reality image. The virtual image 7 may be, for example, another cross-reality image such as a virtual reality image (VR (Virtual Reality) image). The simulation system 100 does not necessarily have to include the image generation unit 41.

[0141] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, application-specific integrated circuits (ASICs), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or processor. [Explanation of symbols]

[0142] 100 Simulation Systems 21 Virtual space generation unit 22 Object generation section 23 Detection and calculation unit 24 Motion generator 3 Display device 32 Position detector 41 Image generation unit 60 Virtual Space 61 Virtual Mobile 62 Virtual User (Virtual Object) 7 Virtual Images 80 Real Space 81 User (Object) 9 Mobile 94 Sensors

Claims

1. A simulation system for performing a motion simulation of a moving object, a virtual space generation unit that generates a virtual space that corresponds to real space and in which a virtual moving object corresponding to the moving object is placed; an object generation unit that generates a virtual object in the virtual space corresponding to an object that operates in the real space; a position detector that detects the position of the object in the real space; a motion generation unit that generates a virtual motion of the virtual moving body with respect to the virtual object, The object generation unit generates the virtual object at a position in the virtual space that corresponds to the position of the object in the real space based on the detection result of the position detector.

2. 2. The simulation system according to claim 1, a detection calculation unit that calculates a detection result of the virtual sensor that detects the virtual object; The action generation unit is a simulation system that generates the virtual action based on the detection result of the virtual sensor.

3. 3. The simulation system according to claim 2, A simulation system in which the virtual sensor corresponds to a sensor provided on the moving body and that detects objects around the moving body.

4. 2. The simulation system according to claim 1, A simulation system in which the virtual object generated by the object generation unit corresponds to a person moving in the real space.

5. A simulation system for performing a motion simulation of a moving object, comprising: a virtual space generation unit that generates a virtual space that corresponds to real space and in which a virtual moving object corresponding to the moving object is placed; an object generation unit that generates a virtual object in the virtual space corresponding to an object that operates in the real space; a position detector that detects the position of the object in the real space; a motion generation unit that generates a virtual motion of the virtual moving body with respect to the virtual object; a head-mounted display that provides a virtual image including the virtual moving object that performs the virtual action, the position detector detects a position of the head mounted display in the real space, thereby detecting a position of a user wearing the head mounted display in the real space; The object generation unit generates a virtual user corresponding to the user at a position in the virtual space that corresponds to the position of the user in the real space based on the detection results of the position detector.

6. 2. The simulation system according to claim 1, The simulation system further comprises an image generation unit that generates a virtual image including the virtual moving object performing the virtual action.

7. 7. The simulation system according to claim 6, A simulation system in which the virtual image is a mixed reality image in which the virtual moving object is superimposed on the real space.

8. 8. The simulation system according to claim 6 or 7, The simulation system further comprises a display device for providing the virtual image.

9. 9. The simulation system according to claim 8, A simulation system in which the display device is a head-mounted display.

10. A simulation method for performing a motion simulation of a moving object, comprising: generating a virtual space corresponding to a real space and in which a virtual moving object corresponding to the moving object is placed; generating a virtual object in the virtual space corresponding to an object operating in the real space; generating a virtual action of the virtual moving body with respect to the virtual object; A simulation method in which the virtual object is generated in the virtual space at a position in the virtual space that corresponds to the position of the object in the real space, based on the detection result of a position detector that detects the position of the object in the real space.

11. A simulation program for performing a motion simulation of a moving object, a function of generating a virtual space corresponding to the real space and in which a virtual moving object corresponding to the moving object is placed; a function of generating a virtual object in the virtual space corresponding to an object operating in the real space; a function of generating a virtual action of the virtual moving body relative to the virtual object; The function of generating the virtual object in the virtual space is a simulation program that generates the virtual object at a position in the virtual space that corresponds to the position of the object in the real space, based on the detection results of a position detector that detects the position of the object in the real space.

Images