Automatic driving control system, automatic driving control device, and automatic driving control method using a digital twin
The automated driving control system enhances obstacle avoidance by using a digital twin space to integrate real-time IoT sensor data and geometry, enabling safe navigation through dynamic environments.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- HITACHI SOFTWARE ENG
- Filing Date
- 2022-09-26
- Publication Date
- 2026-06-16
Smart Images
Figure 0007874494000001 
Figure 0007874494000002 
Figure 0007874494000003
Abstract
Description
Technical Field
[0001] The present invention relates to an automatic driving control system, an automatic driving control device, and an automatic driving control method.
Background Art
[0002] Patent Document 1 discloses a simulation system that performs autonomous movement simulation of a robot. As described below, the simulation system performs autonomous movement simulation. That is, the simulation system creates a two-dimensional environmental map in which the attributes of the point cloud corresponding to the object are reflected, and performs correction according to the sensor data correction parameter on the point cloud containing the attributes. The simulation system outputs corrected sensor data, estimates the position and orientation of the robot in the two-dimensional environmental map by collating the output corrected sensor data and the attribute-reflecting two-dimensional environmental map, determines the next destination, and updates the virtual robot position and orientation according to the determined value.
[0003] Patent Document 2 calculates the operation amount by simulation in the virtual space in the path instruction and control of an articulated robot, and performs a correction operation according to the position error between the actual machine position (calculated from the mounted camera) and the position by simulation.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Patent Document 2
Summary of the Invention
Problems to be Solved by the Invention
[0005] The technology described in Patent Document 1 is not a technology that controls the robot's movement in real time in conjunction with simulation results. In the technology described in Patent Document 2, correction operations are performed to track the target path in response to positional errors, but the path itself cannot be corrected or corrected. Furthermore, since the technology described in Patent Document 2 is not a technology that performs simulations in a virtual space where information from the real space is reflected in real time, it is not possible to dynamically create a path that avoids dynamic obstacles whose positions change according to time. Therefore, Conventional Technology 1 and Conventional Technology 2 may have low obstacle avoidance performance.
[0006] This invention was made to solve the above problems. Specifically, one of the objectives of this invention is to provide an automatic driving control system, an automatic driving control device, and an automatic driving control method that can improve the obstacle avoidance performance of an automatic driving vehicle. [Means for solving the problem]
[0007] To solve the above problems, the present invention provides an automated driving control system that includes an IoT sensor installed in real space and acquiring IoT sensor data which is information about objects present in real space; an information processing device to which the IoT sensor data, geometry data for representing features present in real space in a virtual 3D space corresponding to real space, and information for defining spatial attributes and virtual objects for any region of the virtual space corresponding to real space are input; and the information processing device controls a moving body present in real space to move to a predetermined destination. The information processing device generates a digital twin space which is a 3D space in which features of real space, spatial attributes, and virtual objects are represented based on the IoT sensor data, geometry data, and information for defining spatial attributes and virtual objects, generates a virtual moving body corresponding to the moving body on the digital twin space, and performs a real-time driving simulation in which the virtual moving body moves on the digital twin space so that the virtual moving body and the moving body move in conjunction.
[0008] The present invention provides an automated driving control device which includes an information processing device that receives IoT sensor data, which is information about objects existing in the real space acquired by IoT sensors installed in the real space; geometry data for representing features existing in the real space in a virtual 3D space corresponding to the real space; and information for defining spatial attributes and virtual objects for any region of the virtual space corresponding to the real space. The information processing device controls a moving body existing in the real space to move to a predetermined destination. The information processing device generates a digital twin space, which is a 3D space in which features of the real space, spatial attributes, and virtual objects are represented, based on the IoT sensor data, geometry data, and information for defining spatial attributes and virtual objects. The device generates a virtual moving body corresponding to the moving body on the digital twin space and performs a real-time driving simulation in which the virtual moving body moves on the digital twin space in conjunction with the real body.
[0009] The present invention provides an automated driving control method that uses an IoT sensor installed in a real space to acquire IoT sensor data which is information about objects present in the real space, an information processing device which receives the IoT sensor data, geometry data for representing features present in the real space in a virtual 3D space corresponding to the real space, and information for defining spatial attributes and virtual objects for any region of the virtual space corresponding to the real space, and controls a moving body present in the real space to move to a predetermined destination using the information processing device, wherein the information processing device generates a digital twin space which is a 3D space in which features of the real space, spatial attributes, and virtual objects are represented based on the IoT sensor data, geometry data, and information for defining spatial attributes and virtual objects, generates a virtual moving body corresponding to the moving body on the digital twin space, and performs a real-time driving simulation in which the virtual moving body moves on the digital twin space so that the virtual moving body and the moving body move in conjunction. [Effects of the Invention]
[0010] According to the present invention, the obstacle avoidance performance of an autonomously driven vehicle can be improved. [Brief explanation of the drawing]
[0011] [Figure 1] Figure 1 is a schematic diagram showing an example of the configuration of an automated driving control system according to an embodiment of the present invention. [Figure 2] Figure 2 is a schematic block diagram illustrating the geometry data generation function. [Figure 3A] Figure 3A is a diagram illustrating an example of the configuration of a real robot. [Figure 3B] Figure 3B is a diagram illustrating an example of the configuration of a real robot. [Figure 4] Figure 4 is a diagram illustrating an example of the hardware configuration of an information processing device. [Figure 5]FIG. 5 is a diagram for explaining the digital twin generation function. [Figure 6] FIG. 6 is a sequence diagram showing an example of a processing flow executed by the automatic driving control system. [Figure 7] FIG. 7 is a diagram for explaining the route correction control. [Figure 8] FIG. 8 is a flowchart showing the processing flow executed by the simulation function. [Figure 9] FIG. 9 is a flowchart showing the processing flow executed by the simulation function. [Figure 10] FIG. 10 is a diagram for explaining the details of the obstacle determination executed by the simulation function. [Figure 11] FIG. 11 is a flowchart showing the processing flow of the obstacle determination executed by the simulation function. [Figure 12] FIG. 12 is a diagram for explaining the real object linkage function.
Mode for Carrying Out the Invention
[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings of the embodiments, the same or corresponding parts may be denoted by the same reference numerals.
[0013] In the following description, when explaining the identification information, expressions such as "ID" are used, but the identification information may be other expressions such as an identification number, a name, etc. Further, in the following description, when explaining the processing with a functional block as the subject, the subject of the processing may be a CPU or a device instead of the functional block.
[0014] <<Embodiment>> FIG. 1 is a schematic configuration diagram showing a configuration example of an automatic driving control system according to an embodiment of the present invention. As shown in FIG. 1, the automatic driving control system includes an input device 100, a virtual space manager 200, a control application 300, a real machine robot 400, and a display device 500. In FIG. 1, the number of real machine robots 400 is one, but the number of real machine robots 400 may be plural. Further, the control target may not be only a robot, but may be, for example, a navigation instruction for a person by displaying a route on a display device.
[0015] The automatic driving control system according to an embodiment of the present invention controls a plurality of real machine robots 400 by using a digital twin space DS1. The digital twin space DS1 is a three-dimensional space in which static geometric information of the real space RS1, real-time and dynamic information (IoT sensor data IT1) of people and objects detected in the real space RS1 by an IoT sensor SN1, etc., and virtual objects, space attributes, avatars, etc. arranged on the virtual space VS1 are superimposed, and the real and virtual overlap.
[0016] The digital twin space DS1 is a three-dimensional space (virtual space) corresponding to the real space RS1, in which the information of the real space RS1 is reflected in real time (immediately), and any virtual object, any space attribute, and avatar set in the virtual space VS1 corresponding to the real space RS1 are reflected. It can also be said that it is a three-dimensional space.
[0017] In the digital twin space DS1, by utilizing a general game engine, the position and shape of an object including a feature in the real space RS1 are expressed, the space attributes, virtual objects, avatars of the control target, etc. set in the virtual space VS1 are expressed, and the real-time movement of people and objects detected by an IoT sensor SN1, etc. can be expressed. In the digital twin space DS1, by utilizing a game engine, it is possible to create a route for avoiding obstacles, create the shortest route from the destination to the departure place, etc.
[0018] As will be explained in more detail later, in the automated driving control system, the virtual space manager 200 generates an avatar (virtual robot) that is a replica of the actual robot 400. For convenience, the "virtual robot" is sometimes referred to as a "virtual mobile entity." When the virtual space manager 200 is to have the actual robot 400 drive automatically from the starting point to the destination point, it simulates the automated driving of the virtual robot in the digital twin space DS1.
[0019] Furthermore, the virtual space manager 200 performs a real-time driving simulation in the digital twin space DS1, so that the virtual robot in the digital twin space DS1 and the actual robot 400 in the real space RS1 are linked in real time.
[0020] As a result, the actual robot 400 can automatically travel from the starting point to the destination point while avoiding obstacles, including static obstacles whose position does not change over time and dynamic obstacles whose position changes over time.
[0021] Furthermore, this automated driving control system can also control the actual robot 400, which is inexpensive and lacks sensors to perceive its surroundings, by utilizing the digital twin space DS1, so that the robot 400 can drive safely on its own.
[0022] The virtual space manager 200 can receive information from the input device 100, the geometry data generation function 110, and the IoT sensor SN1 for generating the digital twin space DS1 and for setting up virtual objects, etc., to be represented in the digital twin space D1.
[0023] The input device 100 is a device for inputting information for placing spatial attributes, virtual objects, and avatars such as the avatar to be controlled onto the virtual space VS1, which corresponds to the real space RS1, into the virtual space manager 200.
[0024] The input device 100 is configured to input information to the virtual space manager 200 for setting up virtual robots, spatial attributes, virtual objects, etc., in the virtual space VS1. The input device 100 is, for example, an operating device such as a terminal, keyboard, or mouse.
[0025] The virtual space VS1, with its spatial attributes, virtual objects, and avatars configured, is input to the virtual space manager 200. The spatial attributes, virtual objects, and avatars configured for the virtual space VS1 are reflected (represented) in the digital twin space DS1.
[0026] Spatial attributes are attributes related to the control of the actual robot 400 and can be set for the virtual space VS1. For example, attributes related to the control of the actual robot 400 include at least one of the following: no entry, speed limit, limit on the number of robots that can be present, limit on the length of time that can be present, usage restrictions, "time limits on whether the space can be used or not," and reservation, and can be set for the virtual space VS1.
[0027] Details of each spatial attribute are as follows. Note that areas with spatial attributes set are sometimes also referred to as "spatial attribute setting areas."
[0028] Entry Prohibited: Virtual robots are prohibited from entering the spatial attribute setting area.
[0029] Speed Limit: When the virtual robot travels within the spatial attribute setting area, its speed is limited to the set speed range.
[0030] Dwelling Limit: The number of virtual robots present within the spatial attribute setting area is limited to a set dwelling limit or less.
[0031] Dwelling Time Limit: The dwell time of virtual robots within the spatial attribute setting area is limited to be less than or equal to the set dwell time.
[0032] Usage Restrictions: Uses other than those specified within the spatial attribute setting area are prohibited.
[0033] Time restrictions on the availability or unavailability of the space: The time range in which the space attribute setting area can be used or unavailable is restricted.
[0034] Reservation: You can reserve a time range for using the spatial attribute setting area.
[0035] An avatar is an object in the virtual space VS1, such as a robot in the virtual space VS1 that is represented as a doppelganger of the actual robot 400, a robot in the virtual space VS1 that is represented as a doppelganger of a person, or a person. For example, a virtual robot is a robot in the virtual space VS1 that is represented as a doppelganger of the actual robot 400 that exists in the real space RS1 corresponding to the virtual space VS1.
[0036] A virtual object is an object (other than an avatar) that exists in the virtual space VS1 and is set up (represented) in the virtual space VS1, but does not actually exist in the corresponding real space RS1.
[0037] The input device 100 may also input an environment template to the virtual space manager 200. An environment template is information for setting a predetermined situation according to a desired verification scene. More specifically, an environment template is template information for setting specific virtual events that do not actually occur in the real space RS1 and specific virtual objects that do not actually exist in the real space RS1. For example, an environment template is information for setting a situation in the virtual space VS1 where people who do not actually exist in the real space RS1 are moving around, and information for setting a situation in the virtual space VS1 where there is a fire or injured people, such as in a fire drill. The environment template (specific virtual events and virtual objects) set for the virtual space VS1 is reflected (represented) in the digital twin space DS1.
[0038] IoT sensors SN1 include, for example, multiple cameras, LiDAR, and ToF sensors installed in the real-world space RS1. The data acquired (detected) by IoT sensors SN1 is sometimes referred to as "IoT sensor data IT1." IoT sensor data IT1 is real-time and dynamic information (e.g., 3D point cloud data) of objects (including people) detected from the real-world space RS1 by IoT sensors SN1. Objects detected from the real-world space RS1 by IoT sensors SN1 are reflected (represented) in the digital twin space DS1.
[0039] The geometry data generation function 110 generates geometry data representing the shape, position, etc., of objects (features) existing in the real space RS1, and outputs (provides) it to the virtual space manager 200. The geometry data generation function 110 selects and integrates geometry data that meets the requirements of the control application 300 from a geometry data set GT1 containing a large amount of geometry data in various formats for dynamic placement in the digital twin space DS1, and generates geometry data for the application. Integration includes coordinate system transformation to represent geometry data represented in different coordinate systems in a unified coordinate system, and unification of data formats.
[0040] Figure 2 is a schematic block diagram illustrating the geometry data generation function 110. As shown in Figure 2, the geometry data generation function 110 includes a geometry selection unit 111, a geometry integration unit 112, and a quality verification unit 113.
[0041] The geometry data set GT1 is a collection of geometry data representing objects existing in the real-world space RS1. The geometry data set GT1 includes a large amount of geometry data in various formats for dynamically placing features (static objects) existing in the real-world space RS1 into a 3D space (digital twin space DS1) corresponding to the real-world space RS1. Examples of geometry data include 3D point cloud data and surface mesh data, and are used in BIM (Building Information Modeling) / CIM (Construction Information Modeling / Management) and PLATEAU urban models.
[0042] The geometry selection unit 111 selects a combination of geometry data from the geometry data set GT1 (multiple geometry data) that is suitable for the geometry generation requirement RQ1 of the target application (hereinafter also referred to as the "target application") for performing a specific service, based on the geometry generation requirement RQ1.
[0043] In this example, the "service" is an "automatic driving control service" that controls the automatic driving of multiple actual robots 400, and the "target application" is a "control application 300" that executes the service. "Geometry generation requirement RQ1" is the geometry data generation requirement for the target application, which defines the level of detail (LOD, etc.) of the geometry data required by the target application and the types of features to be included in the digital twin space DS1.
[0044] The geometry integration unit 112 has the function of seamlessly connecting (integrating) geometry data with different LOD (Level of Detail), scale, and coordinate systems. The geometry integration unit 112 generates geometry (geometry data representing the shape and position of features (static objects) in real space RS1) according to the target application by connecting (integrating) each geometry data that satisfies the geometry generation requirement RQ1 of the selected target application.
[0045] The quality verification unit 113 estimates the range in which the application can operate using the simulation unit 114, outputs a quality map MP1 which is information indicating the area / range that satisfies the requirements required by the target application, and also outputs geometry data AGT1 for the application which is geometry data that satisfies the geometry generation requirement RQ1 of the target application.
[0046] Returning to Figure 1, the virtual space manager 200 manages the dynamic creation, updating, and destruction of multiple virtual spaces (digital twin spaces DS1). The virtual space manager 200 performs state synchronization, visualization, prediction, simulation, and feedback (control, nudge) of multiple virtual spaces (digital twin spaces DS1) on the target application. The virtual space manager 200 includes a digital twin generation function 210, a simulation function 220, and a real object linkage function 230.
[0047] The digital twin generation function 210 generates a digital twin space DS1 based on IoT sensor data IT1, geometry data (for example, geometry data AGT1 for the application), and information input from the input device 100 (information for generating avatars, virtual objects, and spatial attributes of the virtual space VS1).
[0048] Simulation function 220 includes the ability to perform simulations such as real-time driving simulations of a virtual robot in the digital twin space DS1.
[0049] The real object linkage function 230 controls the actual robot 400 corresponding to the virtual robot in the digital twin space DS1 based on the results of the simulation of the virtual robot by the simulation function 220.
[0050] The control application 300 controls the automated movement of multiple physical robots 400 via the virtual space manager 200. The control application 300 manages the physical robots 400 by referring to management information for controlling (managing) the multiple physical robots 400. The management information includes information that associates the robot ID, which is identification information for each physical robot 400, with the current position of each physical robot 400. The management information is contained in the virtual space manager 200.
[0051] An example of the actual robot 400 is a wheelchair-type robot (mobile robot). Figures 3A and 3B are diagrams illustrating an example of the configuration of the actual robot 400. As shown in Figures 3A and 3B, the actual robot 400 includes the control system 410 shown in Figure 3B, a drive unit (not shown), a control unit (not shown) that controls the drive unit, a left front wheel (not shown) and a right front wheel 401 and wheel axle (not shown), a left rear wheel (not shown) and a left drive axle (not shown), and a right rear wheel 402 and a right drive axle (not shown). Note that the left and right front wheels and left and right rear wheels are referred to as "wheels" unless there is a need to distinguish between them, and the left drive axle and right drive axle are referred to as "drive axles" unless there is a need to distinguish between them. The left and right rear wheels are also referred to as drive wheels.
[0052] The control system 410 includes a first communicator 411, a path-following control and turn control unit 412, a control command generation unit 413, an odometry receiving unit 414, and a second communicator 415. In this example, the control system 410 is mounted on the actual robot 400, but it may also be located outside the actual robot 400.
[0053] The first communication device 411 acquires the travel path (planned travel path) from the virtual space manager 200 and outputs it to the path-following control and turn control unit 412 (hereinafter also referred to as the "travel control unit 412"). The travel control unit 412 generates control commands (target speed and yaw rate (instructed amounts for speed and yaw rate)) so that the actual robot 400 travels along the travel path, and outputs the control commands via the second communication device 415 to the control unit that controls the drive system of the actual robot 400.
[0054] The control unit is connected to a drive unit that drives the wheels. The drive unit is, for example, two motors. Each of the two motors generates torque that rotates (drives) the drive shaft corresponding to each motor, power supplied to the motor from a power supply unit (not shown). The motors can use this torque to rotate the drive wheels corresponding to each drive shaft.
[0055] The control unit is a control device that controls the power supplied to the motor. By controlling the power supplied to the motor, the control unit can adjust the magnitude and direction of the torque generated by the motor, thereby freely controlling the rotational speed and direction of the drive wheels.
[0056] The control unit controls the movement of the actual robot 400 by controlling the motors and driving the drive shafts so that the speed and yaw rate of the actual robot 400 become the target speed and yaw rate, based on the control command 313 (target speed and yaw rate). In other words, the control unit can freely control the movement of the actual robot 400 to perform actions such as moving forward, backward, turning, curve driving, adjusting the direction of travel, avoiding obstacles, and stopping. For convenience, the device including the motors, drive shafts, and drive wheels may also be referred to as the "mobilization device".
[0057] The display device 500 is, for example, a display capable of displaying images. The virtual space manager 200 is configured to output and display a screen (image) that visualizes the digital twin space DS1 to the display device 500.
[0058] Each functional block shown in Figures 1 and 2 can be realized by applying hardware such as an information processing device and a storage device.
[0059] Figure 4 is a block diagram illustrating an example of the hardware configuration of an information processing device (computer). An example of a virtual space manager 200 can be configured by an information processing device 2000. The virtual space manager 200 may also be composed of multiple information processing devices 2000 connected via a network. The information processing device 2000 may also be a virtual information processing device and storage device on the cloud.
[0060] The information processing device 2000 includes a CPU 2001, ROM 2002, RAM 2003, storage device 2004, network interface 2005, and input / output interface 2006, etc. These are connected to each other via a bus 2007 so that they can communicate with one another.
[0061] The CPU 2001 loads various programs (not shown) stored in the ROM 2002 and / or the storage device 2004 into the RAM 2003, and then executes the programs loaded into the RAM 2003 to realize various functions. As described above, the RAM 2003 is loaded with various programs to be executed by the CPU 2001, and temporarily stores data used by the CPU 2001 when executing these programs. The ROM 2002 is a non-volatile storage medium that stores various programs. The storage device 2004 is a non-volatile storage medium that allows data to be read and written. The network interface 2005 is an interface for the information processing device 2000 to connect to a network. The input / output interface 2006 is an interface for connecting to external devices (e.g., keyboards, mice and other operating devices, and displays (display devices)).
[0062] Figure 5 is a diagram illustrating the digital twin generation function 210. The digital twin generation function 210 can define attributes (spatial attributes) related to the control of the actual robot 400 in the virtual space VS1. By dynamically defining the spatial attributes, the digital twin generation function 210 can reflect these in the automatic driving (control) of the actual robot 400.
[0063] As shown in Figure 5, the digital twin generation function 210 includes a destination and travel route setting unit 211, a spatial attribute setting unit 212, and a spatial attribute modification unit 213.
[0064] The destination and travel route setting unit 211 is capable of setting destinations and generating travel routes for the virtual space VS1. The destinations and travel routes set for the virtual space VS1 are reflected in the digital twin space DS1.
[0065] As shown in specific example 221, once a destination and travel path are set for the virtual space VS1, they are reflected in the digital twin space DS11. Then, the simulation function 220 and the real object linkage function 230 can be used to reflect this in the automatic driving of the actual robot 400 so that it automatically travels along the travel path from the destination.
[0066] The spatial attribute setting unit 212 is capable of setting attributes for any region (any 2D region and any 3D space) in the virtual space VS1. For example, the input device 100 inputs coordinate information representing a specific region of the virtual space VS1 and information indicating the attributes of the specified region to the spatial attribute setting unit 212, and the spatial attributes are set based on the input information.
[0067] Spatial attributes set for the virtual space VS1 are reflected in the digital twin space DS1. As shown in specific example 222, for example, if an area in the virtual space VS1 is set as "no entry" as a spatial attribute, it is reflected in the digital twin space DS1. Then, the simulation function 220 and the real object linkage function 230 can be used to ensure that the actual robot 400 does not enter the area where "no entry" is set, and this can be reflected in the automatic driving of the actual robot 400.
[0068] The spatial attribute modification unit 213 can change at least one of the attributes and size of regions (2D regions and 3D spaces) on which spatial attributes have been set for the virtual space VS1. For example, as shown in specific example 223, if the attribute (e.g., no entry) of an area (on the travel path) on which an attribute has been set for the virtual space VS1 is changed to slow-speed travel, the changed attribute is reflected in the digital twin space DS1. Then, the simulation function 220 and the real object linkage function 230 can be used to reflect this in the automatic driving of the actual robot 400 so that it drives at a slow speed when traveling through that area.
[0069] Figure 6 is a sequence diagram showing an example of the processing flow performed by the automated driving control system. The automated driving control system performs the processes S601 to S620 described below in order.
[0070] S601: The control application 300 transmits the robot ID and initial position of the physical robot 400 to the virtual space manager 200. The virtual space manager 200 registers the robot IDs and positions (initial positions) of multiple physical robots 400 in its management information.
[0071] S602: The control application 300 receives a dispatch request for the actual robot 400.
[0072] S603: The control application 300 transmits to the virtual space manager 200 the robot ID and destination of the actual robot 400 that has been requested to be dispatched.
[0073] S604: The virtual space manager 200 uses the simulation function 220 to generate a virtual robot corresponding to the actual robot 400 indicated by the robot ID for which dispatch was requested, and searches for the shortest path from the virtual robot's initial position to the destination on the digital twin space DS1. For example, the virtual space manager 200 uses a game engine to find the shortest path from the initial position to the destination in the navigation mesh representing the passable area on the digital twin space DS1. * The search is performed using the (A-star) algorithm.
[0074] S605: The virtual space manager 200 executes a driving simulation in the digital twin space DS1, in which the virtual robot is driven along the shortest path that has been found.
[0075] S606: The virtual space manager 200 notifies the simulation results (e.g., time taken and whether or not there were collisions with obstacles).
[0076] S607: Upon receiving the simulation results, the control application 300 sends an instruction to the virtual space manager 200 to start the actual robot 400's movement.
[0077] S608: When the virtual space manager 200 receives a command to start driving from the control application 300, it transmits position information representing the driving route to the actual robot 400.
[0078] S609: The actual robot 400 starts autonomous driving in the real space RS1, following the designated driving path.
[0079] S610: The actual robot 400 notifies the virtual space manager 200 that it has started autonomous driving. The actual robot 400 also estimates its own position and yaw rate sequentially (at predetermined intervals) and transmits the estimated position and yaw rate to the virtual space manager 200. Information including the estimated position and yaw rate is also referred to as "odometry data" or "odometry".
[0080] S611: The virtual space manager 200 performs real-time driving simulations of the virtual robot in the digital twin space DS1. The virtual space manager 200 also synchronizes the real robot 400 and the virtual robot by moving the virtual robot in the digital twin space DS1 based on odometry received from the real robot 400.
[0081] S612: When the IoT sensor SN1 detects a marker (not shown) on the actual robot 400 moving in the real space RS1, and detects the position of the actual robot 400, the IoT sensor SN1 transmits the position of the actual robot 400 to the virtual space manager 200. This position is transmitted at predetermined intervals during the period in which the IoT sensor SN1 is detecting the position of the actual robot 400.
[0082] S613: When the virtual space manager 200 obtains the position of the actual robot 400, it performs path correction control. Details of the path correction control will be described later.
[0083] S614: The IoT sensor SN1 detects an obstacle on the travel path, and the virtual space manager 200 receives information indicating the obstacle on the travel path.
[0084] S615: The virtual space manager 200 notifies the actual robot 400 of the presence of an obstacle and initiates obstacle avoidance control. Details of the obstacle avoidance control will be described later.
[0085] S616: The virtual space manager 200 searches for an avoidance path to avoid obstacles in the digital twin space DS1.
[0086] S617: If the virtual space manager 200 can find a circuit path, it instructs the actual robot 400 to take an avoidance route.
[0087] S618: The actual robot 400 avoids obstacles by traveling along the avoidance path.
[0088] S619: The virtual space manager 200 moves the virtual robot along an avoidance path in the digital twin space DS1 in conjunction with the movement of the actual robot 400.
[0089] S620: When the virtual robot arrives at its destination in the digital twin space DS1, the virtual space manager 200 sends a notification of arrival to the control application 300. The virtual space manager 200 requests the control application 300 to perform the following tasks.
[0090] <Route Correction Control> Figure 7 is a diagram illustrating the details of the path correction control described above. Figure 7 shows the two-dimensional coordinate plane (2D position coordinates) when the virtual robot performs a realistic simulation run.
[0091] In Figure 7, "Virtual (X, Y)" indicates the X and Y coordinate positions of the virtual robot on a two-dimensional coordinate plane in the digital twin space DS1. "Actual (x, y)" indicates the x and y coordinate positions of the actual robot 400 on a two-dimensional coordinate plane with the initial position as the origin, and represents the position obtained based on the self-position estimation results of the actual robot 400. Self-position estimation is performed based on the amount of movement (direction and distance) of the actual robot 400. The actual robot 400 acquires odometry (self-position and yaw rate) by self-position estimation at predetermined intervals.
[0092] Region R1 is a region where the position of the virtual robot can be detected with high accuracy by detecting the position of the actual robot 400 using the IoT sensor SN1.
[0093] The simulation function 220, in real-time driving simulation, acquires the virtual (X, Y) position of the virtual robot from its initial position based on the odometry of the actual robot 400 at predetermined intervals.
[0094] In such real-time driving simulations, there may be discrepancies between the virtual robot's position in the virtual space VS1 and the position in the virtual space VS1 that corresponds to the actual position of the real robot 400.
[0095] For example, in the example shown in Figure 7, as indicated by the callout BL1, the initial position of the virtual robot is virtual (X, Y) = (100, 100), and the initial position of the actual robot 400 is actual (x, y) = (0, 0).
[0096] As shown in callout BL2, when the actual robot 400 enters the high-precision detection area, the position of the virtual robot before correction at a certain detection point is virtual (X, Y) = (150, 150), and the position of the actual robot 400 is actual (x, y) = (50, 50).
[0097] At this point in time, the correct position of the virtual robot detected by the IoT sensor SN1 is virtual (X, Y) = (190, 160), as shown in callout BL3. This is an error between this and the virtual (X, Y) = (150, 150) position of the virtual robot at the time of detection.
[0098] In this case, if the real-time driving simulation is performed with the error still present, when the virtual robot reaches the virtual destination (X,Y)=(320,310) on the 2D coordinate plane of the digital twin space DS1, the actual robot 400 in the real space RS1 may reach the wrong destination because it moved along the dashed arrow (callout BL4). In other words, even though the virtual robot reaches the correct destination (callout BL5) in the digital twin space DS1, the actual robot 400 in the real space RS1 may not reach the correct destination.
[0099] Therefore, the virtual space manager 200 performs path correction control as described below.
[0100] When the actual robot 400 is moving within the high-precision position detection area R1, the virtual space manager 200 can accurately acquire the position of the actual robot 400 using the IoT sensor SN1.
[0101] The virtual space manager 200 performs route correction control to correct the travel route by recreating the travel route from the high-precision detected position to the destination (blower BL6).
[0102] Specifically, the virtual space manager 200 recreates the travel path to the destination based on the position of the actual robot 400 acquired by the IoT sensor SN1 when the actual robot 400 is traveling in the high-precision position detection area R1. The virtual space manager 200 corrects the existing travel path by replacing the existing travel path with the recreated travel path. The virtual space manager 200 transmits the corrected travel path to the actual robot 400. The actual robot 400 travels along the corrected travel path.
[0103] This allows the virtual space manager 200 to control the actual robot 400 so that it reaches the correct destination.
[0104] The details of the processing performed by the simulation function 220 described above will be explained using Figure 8. Figure 8 is a flowchart showing the processing flow performed by the simulation function 220. The simulation function 220 starts processing from step 800 in Figure 8, and after sequentially executing the processes from steps 805 to 820 described below, proceeds to step 825.
[0105] Step 805: The simulation function 220 receives dispatch instructions (robot ID and initial position) from the control application 300.
[0106] Step 810: The simulation function 220 searches for a path on the virtual space VS1 using the game engine's navigation AI.
[0107] Step 815: The simulation function 220 performs a driving simulation using a behavior model of the actual robot 400. That is, it uses a behavior model that models the behavior of the actual robot 400 (position of the actual robot 400, speed of the actual robot 400, yaw rate of the actual robot 400, etc.) in response to control commands (operated variables (velocity and yaw rate)) to perform a driving simulation of the actual robot 400 in the virtual space VS1.
[0108] Step 820: The simulation function 220 transmits the simulation results (time taken, whether or not there was a collision, etc.) to the control application 300.
[0109] When the simulation function 220 proceeds to step 825, it determines whether or not it has received a command to start driving from the control application 300.
[0110] If the control application 300 has not received a command to start driving, the simulation function 220 determines "No" in step 825 and proceeds to step 830, terminating this processing flow.
[0111] When the control application 300 receives a command to start driving, the simulation function 220 determines "Yes" in step 825 and proceeds to step 835, starting a real-time driving simulation in the virtual space VS1. The real-time driving simulation is mainly performed to make the actual robot 400 drive in accordance with the situation that changes over time. By performing a real-time driving simulation, the actual robot 400 can avoid colliding with dynamic obstacles whose position changes over time.
[0112] When the simulation function 220 proceeds to step 840, it determines whether or not the IoT sensor SN1 has detected an obstacle, based on the obstacle detection described later.
[0113] If the IoT sensor SN1 detects an obstacle, the simulation function 220 determines "Yes" in step 840 and proceeds to step 845, executes obstacle avoidance control, and returns to step 840.
[0114] If the IoT sensor SN1 does not detect an obstacle, the simulation function 220 determines "No" in step 840 and proceeds to step 850 to determine whether the virtual robot has arrived at its destination.
[0115] If the virtual robot has not reached its destination, the simulation function 220 determines "No" in step 850 and returns to step 840.
[0116] If the virtual robot arrives at its destination, the simulation function 220 determines "Yes" in step 850 and proceeds to step 895, terminating this processing flow.
[0117] The details of the obstacle avoidance control performed by the simulation function 220 described above will be explained using Figure 9. Figure 9 is a flowchart showing the processing flow performed by the simulation function 220. The simulation function 220 starts processing from step 900 in Figure 9 and proceeds to step 905, where it acquires the location information of the obstacle detected by the IoT sensor SN1.
[0118] Subsequently, the simulation function 220 proceeds to step 910, where it determines whether the obstacle detected by the IoT sensor SN1 is an obstacle on the travel path. Details of this obstacle determination will be described later.
[0119] If the obstacle detected by the IoT sensor SN1 is not an obstacle on the travel path, the simulation function 220 determines "No" in step 910 and proceeds to step 915, terminating this processing flow.
[0120] If the obstacle detected by the IoT sensor SN1 is an obstacle on the driving path, the simulation function 220 determines "Yes" in step 910 and proceeds to step 920, and if the TTC (Collision Time (Collision Prediction Time)) is within the acceptable range (more than the time required to avoid a collision, and a collision occurs) Avoid The system determines whether the collision avoidance time is within the acceptable range (which is less than or equal to the judgment time). The TTC (Time To Cycle) can be calculated, for example, based on the distance between the obstacle and the virtual robot and the speed of the virtual robot. The collision avoidance time is set to be shorter than the collision avoidance judgment time.
[0121] If the TTC is outside the acceptable range (less than the time required to avoid a collision), the simulation function 220 determines "No" in step 920 and proceeds to step 925, instructing the actual robot 400 to stop or limit its speed, and then proceeds to step 930.
[0122] If the TTC is within the acceptable range, the simulation function 220 determines "Yes" in step 920, and after sequentially executing the processes in steps 930 and 935 described below, proceeds to step 940.
[0123] Step 930: Simulation function 220 performs rerouting by navigation AI on the virtual space VS1.
[0124] Step 935: The simulation function 220 performs a driving simulation using a behavior model of the actual robot 400.
[0125] When the simulation function 220 proceeds to step 940, it determines, based on the results of the driving simulation, whether or not the actual robot 400 is likely to collide with other obstacles on the avoidance path.
[0126] If there is a possibility that the actual robot 400 will collide with another obstacle on its avoidance path, the simulation function 220 determines "No" in step 940 and proceeds to step 945, instructing the actual robot 400 to stop and notifying the control application 300 that the actual robot 400 has stopped.
[0127] If there is no possibility of the actual robot 400 colliding with other obstacles on its avoidance path, the simulation function 220 determines "Yes" in step 940, sends the avoidance path to the actual robot 400, and proceeds to step 995 to terminate this processing flow.
[0128] The details of the obstacle detection described above will be explained using Figures 10 and 11. Figure 10 is a diagram illustrating the details of the obstacle detection performed by the simulation function 220. As shown in Figure 10, the navigation AI generates initial waypoints P1 at predetermined intervals along the travel path of the virtual robot VB1. Waypoints P1 are points along the path.
[0129] The simulation function 220 generates waypoint P2 at a predetermined interval a between adjacent initial waypoints P1. The simulation function 220 generates a circle with radius c centered on waypoint P2. The simulation function 220 sets the area contained within a predetermined number (b) of circles, counting sequentially from the current position, as the obstacle detection area P3. The simulation function 220 can reduce the computational load by setting the obstacle detection area P3 as the minimum necessary detection area. The simulation function 220 may also change the predetermined number according to the speed. The simulation function 220 performs obstacle detection by determining whether or not an obstacle detected by the IoT sensor SN1 is present within the obstacle detection area P3.
[0130] Figure 11 is a flowchart showing the obstacle detection processing flow performed by the simulation function 220. The simulation function 220 starts processing from step 1100, sequentially executes the processes described below in steps 1105 to 1115, and then proceeds to step 1195 to terminate this processing flow.
[0131] Step 1105: The simulation function 220 divides the initial waypoints P1 by a predetermined length a. That is, the simulation function 220 creates waypoints P2 at positions that divide the initial waypoints P1 by a predetermined length (predetermined interval a).
[0132] Step 1100: The simulation function 220 extracts routes from the current location that include a predetermined number (b) waypoints P1 and waypoint P2.
[0133] Step 1115: The simulation function 220 sets the area contained within multiple circles of radius c drawn with each of the extracted b waypoints P1 and P2 as the obstacle detection area P3, and determines objects (obstacles) within the obstacle detection area P3 as obstacles.
[0134] Figure 12 is a diagram illustrating the real object linkage function 230. As shown in Figure 12, the virtual space manager 200 communicates with the first communication device 411 using the communication device COM1 to exchange information necessary for the real object linkage function 230.
[0135] The virtual space manager 200 transmits the travel path (path data) created by the simulation function 220 to the control system 410 of the actual robot 400. Upon receiving the path data, the control system 410 performs path-following control to control the actual robot 400 to move along the path data. Specifically, the travel control unit 412 generates control commands (velocity and yaw rate) for moving along the travel path at predetermined intervals and transmits them to the control unit of the actual robot 400 via the second communication device 415. The control unit controls the drive device based on the control commands. The control unit of the actual robot 400 performs self-position estimation to estimate odometry based on the amount of movement and transmits it to the control system 410 via the second communication device 415.
[0136] When the odometry receiving unit 414 of the control system 410 receives odometry from the control unit of the actual robot 400 via the second communicator 415, it communicates with the communicator COM1 using the first communicator 411 and transmits the odometry to the virtual space manager 200. Based on the odometry, the virtual space manager 200 moves the position of the virtual robot in the digital twin space DS1.
[0137] The virtual space manager 200 outputs virtual space data to the display device 500, enabling the display of a visualization screen that visualizes the digital twin space DS1. An example of the visualization screen includes a person image HU1 corresponding to a person in the real space RS1, a virtual robot AB1, a virtual object OB1, and a line TD1 indicating path data. The person image HU1 is generated based on IoT sensor data IT1 related to the person detected by the IoT sensor SN1. The virtual robot AB1 is generated based on the odometry position. The line TD1 indicating path data is generated based on the travel path. The virtual object OB1 is generated based on spatial attributes.
[0138] <Effects> As described above, the automated driving control system according to the embodiment of the present invention is a digital twin space DS1 which is a virtual space VS1 corresponding to the real space RS1, where information from the real space RS1 is reflected in real time (immediately), and any virtual object and the attributes of any space can be set. The system can control the movement of a real robot 400 that actually travels in the real space RS1 in conjunction with the movement of a virtual robot that moves to follow a driving path dynamically set to avoid obstacles present in the digital twin space DS1. As a result, the automated driving control system can improve the obstacle avoidance performance of the real robot 400 during automated driving.
[0139] Furthermore, the automated driving control system according to the embodiment of the present invention also provides the following effects.
[0140] In other words, conventional autonomous robots are equipped with multiple LiDAR sensors and cameras for each robot to perceive their surroundings, resulting in a very expensive and complex system configuration. Because each robot has a different sensor configuration and installation conditions, each possesses its own unique environmental map, making it difficult to share spatial information. Furthermore, verifying safety in scenarios that cannot or are difficult to test in real space (such as collisions) remains a challenge.
[0141] In contrast, the automated driving control system according to the embodiment of the present invention can provide a control method that enables safe movement even for inexpensive robots that do not have sensors to perceive their surroundings, by utilizing the digital twin space DS1. Furthermore, the automated driving control system according to the embodiment of the present invention can improve the efficiency of pre-verification and learning of the automated driving of the actual robot 400 by providing a digital twin space DS1 in which virtual objects that do not exist in real space are placed.
[0142] <<Variation>> The present invention is not limited to the embodiments described above, and various modifications can be adopted within the scope of the present invention.
[0143] In the above embodiment, the number of wheels of the actual robot 400 is not limited to the above, and may be, for example, two wheels, and a steering mechanism for steering the wheels that can be controlled by the control unit may be provided. The actual robot 400 is not limited to a mobile body whose mobility mechanism is wheels, but may be a mobile body whose mobility mechanism is other than wheels. The mobile body may be, for example, a bipedal robot. [Explanation of Symbols]
[0144] 100…Input device, 110…Geometry data generation function, 200…Virtual space manager, 210…Digital twin generation function, 220…Simulation function, 230…Real object linkage function, 300…Control application, 400…Actual robot, 500…Display device
Claims
1. An IoT sensor installed in real space that acquires IoT sensor data which is information about objects present in that real space, An information processing device that receives the IoT sensor data, geometry data for representing features existing in the real space in a virtual three-dimensional space corresponding to the real space, and information for defining spatial attributes and virtual objects for any region of the virtual space corresponding to the real space as input. A storage device that stores environment templates, which are templates of virtual objects and spatial attributes corresponding to a desired verification scene, Includes, An automated driving control system that uses the information processing device to control a moving object in the real space to move to a predetermined destination, The aforementioned information processing device is Based on the IoT sensor data, the geometry data, and the information for defining the spatial attributes and the virtual object, a digital twin space is generated, which is a three-dimensional space in which the features of the real space, the spatial attributes, and the virtual object are represented. A virtual mobile body corresponding to the mobile body is generated in the digital twin space, and a real-time driving simulation is performed in which the virtual mobile body moves in the digital twin space so that it moves in conjunction with the mobile body. It is configured in such a way, The aforementioned information processing device is Using the environment template, define the virtual objects and spatial attributes in the virtual space according to the desired verification scene. It is configured in such a way. Automated driving control system.
2. In the automated driving control system according to claim 1, The moving body includes a moving device capable of moving the moving body, which is controlled by a control device, in a desired direction of movement. The aforementioned information processing device is A driving route to a predetermined destination is generated in the digital twin space, A simulation is performed on the digital twin space to show the virtual mobile object moving along the travel path, and if the simulation shows that the virtual mobile object can travel along the travel path, the travel path is transmitted to the control device. It is configured in such a way, The control device controls the moving device so that the moving body moves along the travel path. It is configured in such a way. Automated driving control system.
3. In the automated driving control system according to claim 2, The aforementioned information processing device is As the aforementioned spatial attributes, we define attributes related to the control of the moving object. Configured Automated driving control system.
4. In the automated driving control system according to claim 3, The attributes relating to the control of the aforementioned moving object include at least one of the following: no entry, speed limit, limit on the number of vehicles allowed, limit on the length of stay, restriction on use, availability of space, time limit for availability or non-availability, and reservation. Automated driving control system.
5. In the automated driving control system according to claim 2, The IoT sensor is configured to acquire the position of the moving object as information about the object. The information processing device corrects the position of the virtual moving object based on the position of the moving object acquired by the IoT sensor. It is configured in such a way. Automated driving control system.
6. In the automated driving control system according to claim 5, The aforementioned information processing device is When the mobile body is traveling in an area with high position detection accuracy, if an error occurs between the high-precision position in the digital twin space corresponding to the position of the mobile body acquired by the IoT sensor and the position of the virtual mobile body, the position of the virtual mobile body is corrected to the high-precision position, a new corrected travel route is created from the high-precision position to the destination, and the travel route is corrected using the created corrected travel route. It is configured in such a way. Automated driving control system.
7. In the automated driving control system according to claim 2, The aforementioned information processing device is When the real-time driving simulation is being performed, if an obstacle exists in a predetermined obstacle detection area on the driving path of the virtual mobile body and in front of it in the digital twin space, an obstacle avoidance simulation is performed to cause the virtual mobile body to move to avoid the obstacle, an obstacle avoidance path is generated based on the results of the obstacle avoidance simulation, and the obstacle avoidance path is transmitted to the control device. It is configured in such a way, The control device controls the moving device so that the moving body moves along the obstacle avoidance path. It is configured in such a way. Automated driving control system.
8. In the automated driving control system according to claim 7, The aforementioned information processing device is A specific area along the aforementioned travel path is set as an obstacle detection area. It is configured in such a way. Automated driving control system.
9. In the automatic driving control system according to claim 8, The aforementioned information processing device is Circles of a predetermined radius are generated centered on points plotted at predetermined intervals along the aforementioned travel path, and the region enclosed by a predetermined number of circles, counting from the side closest to the virtual moving object, is defined as the obstacle detection region. It is configured in such a way. Automated driving control system.
10. In the automatic driving control system according to claim 8, The aforementioned information processing device is If the presence of an obstacle is detected in the obstacle detection area based on the IoT sensor data, the collision prediction time of the virtual moving body with the obstacle is calculated. If the collision prediction time is within an acceptable range that is greater than or equal to the collision avoidance time and less than or equal to the collision avoidance determination time, the obstacle avoidance simulation is performed. If the collision prediction time is less than the time available for collision avoidance, the control device will cause the moving body to either stop or limit its speed. It is configured in such a way. Automated driving control system.
11. In the automated driving control system according to claim 1, Includes a display device capable of displaying images, The aforementioned information processing device is The image visualizing the digital twin space is displayed on the display device. It is configured in such a way. Automated driving control system.
12. IoT sensor data, which is information about objects present in the real space acquired by IoT sensors installed in the real space, Geometric data for representing features existing in the real space in a virtual three-dimensional space corresponding to the real space, Information for defining spatial attributes and virtual objects for any region of the virtual space corresponding to the aforementioned real space, An information processing device into which the following is input, A storage device that stores environment templates, which are templates of virtual objects and spatial attributes corresponding to a desired verification scene, Includes, An automatic driving control system that uses the information processing device to control a moving object in the real space to move to a predetermined destination, The aforementioned information processing device is Based on the IoT sensor data, the geometry data, and the information for defining the spatial attributes and the virtual object, a digital twin space is generated, which is a three-dimensional space in which the features of the real space, the spatial attributes, and the virtual object are represented. A virtual mobile body corresponding to the mobile body is generated in the digital twin space, and a real-time driving simulation is performed in which the virtual mobile body moves in the digital twin space so that it moves in conjunction with the mobile body. It is configured in such a way, The aforementioned information processing device is Using the environment template, define the virtual objects and spatial attributes in the virtual space according to the desired verification scene. It is configured in such a way. Automatic driving control system.
13. An IoT sensor installed in real space that acquires IoT sensor data which is information about objects present in that real space, An information processing device that receives the IoT sensor data, geometry data for representing features existing in the real space in a virtual three-dimensional space corresponding to the real space, and information for defining spatial attributes and virtual objects for any region of the virtual space corresponding to the real space as input. A storage device that stores environment templates, which are templates of virtual objects and spatial attributes corresponding to a desired verification scene, Use An automated driving control method for controlling a moving object in the real space to move to a predetermined destination using the information processing device, The information processing device, Based on the IoT sensor data, the geometry data, and the information for defining the spatial attributes and the virtual object, a digital twin space is generated, which is a three-dimensional space in which the features of the real space, the spatial attributes, and the virtual object are represented. A virtual mobile body corresponding to the mobile body is generated in the digital twin space, and a real-time driving simulation is performed in which the virtual mobile body moves in the digital twin space so that it moves in conjunction with the mobile body. The information processing device, Using the environment template, define the virtual objects and spatial attributes in the virtual space according to the desired verification scene. Automatic driving control method.