Information processing device, interference control system, method and program

Abstract

This enables high-precision control with ample time to avoid interference between objects and people in real space. [Solution] The configuration includes a processor that executes a program, acquires information on at least one movable object in real space, performs a simulation on the digital twin of the model of at least one movable object reproduced on the digital twin which maps real space to virtual space, verifies whether at least a part of the model of at least one movable object may interfere with the model on the digital twin which corresponds to at least one other object included in real space by a predetermined future time after the time the information was acquired, and if interference is found with the model of at least one movable object by the future time, determines countermeasures to avoid the interference and instructs the at least one movable object in real space to take the interference avoidance measures.

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, an interference control system, a method, and a program. [Background technology]

[0002] In workplaces where multiple robotic arms, mobile robots, and human workers are working together, it is necessary to prevent accidents and improve work efficiency by preventing interference between robots or between robots and humans. Interference here may include, for example, cases where multiple objects or parts of them overlap (collide) with each other in a three-dimensional coordinate system, or are located in close proximity, or where one object is located within the space occupied or mutually exclusive of the other.

[0003] A technology is known that predicts events that may occur at a future point in time on a digital twin, which is a virtual reproduction of objects and people in real space. For example, Patent Document 1 discloses a configuration that predicts the positions of a target vehicle and objects at a predetermined time based on a traffic digital twin that is time-synchronized with the real space and constructed in virtual space based on vehicle data received by a communication unit. The generation of the traffic digital twin includes data about objects other than the vehicle itself, such as other vehicles, buildings, and pedestrians that exist around the vehicle, and various sensors mounted on each vehicle can be used to acquire this information and data. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2024-87504 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] In workplaces where multiple robotic arms and human workers are involved, it is necessary to prevent interference between objects and between objects and people in order to prevent accidents and improve work efficiency. In other words, it is desirable to control interference avoidance with high precision and with sufficient time to spare.

[0006] One of the purposes of this disclosure is to provide an apparatus, system, method, or program that solves the above-mentioned problems. [Means for solving the problem]

[0007] According to one embodiment of the present disclosure, the information processing device includes at least one processor and at least one memory storing a program executable by the at least one processor, wherein the at least one processor executes the program, acquires information on at least one movable object in real space, performs a simulation on the digital twin on a model of the at least one movable object reproduced on a digital twin which maps the real space to a virtual space, verifies whether at least a part of the model of the at least one movable object may interfere with a model on the digital twin corresponding to at least one other object included in the real space by a predetermined future time after the time the information was acquired, determines countermeasures to avoid the interference if such interference exists with respect to the model of the at least one movable object by the future time, and instructs the at least one movable object in real space to take the measures to avoid the interference.

[0008] According to one embodiment of the present disclosure, the system comprises a plurality of movable objects, a first information processing device, and a second information processing device that controls at least one of the plurality of movable objects. The first information processing device acquires information on the at least one movable object, performs a simulation on the digital twin on a model of the at least one movable object reproduced on a digital twin that maps the real space in which at least the plurality of movable objects are located to a virtual space, verifies whether at least a part of the model of the at least one movable object may interfere with a model on the digital twin corresponding to at least one other object included in the real space by a predetermined future time after the time the information was acquired, determines measures to avoid the interference with the model of the at least one movable object by the future time, transmits a command to the second information processing device instructing the measures to avoid the interference, and the second information processing device instructs the at least one movable object to take the measures to avoid the interference.

[0009] According to one embodiment of the present disclosure, information on at least one movable object in real space is obtained, A simulation is performed on the digital twin, which is a digital twin that maps the real space to a virtual space, to verify whether at least a part of the model of the at least one movable object may interfere with the model on the digital twin that corresponds to at least one other object included in the real space, by a predetermined future time point prior to the time when the information was acquired. If the interference occurs with respect to the model of at least one of the movable objects by the future time, measures to avoid the interference will be determined. The system instructs at least one movable object in the real space to take measures to avoid the interference.

[0010] According to the program of one aspect of the present disclosure, information on at least one movable object in a real space is acquired, A simulation on a digital twin is executed for the model of the at least one movable object reproduced on the digital twin obtained by mapping the real space to a virtual space, and it is verified whether at least a part of the model of the at least one movable object may interfere with a model on the digital twin corresponding to at least one other object included in the real space, by a predetermined future point in time that is later than the point in time when the information is acquired If the interference exists by the future point in time for the model of the at least one movable object, a measure for avoiding the interference is determined and the processor is caused to execute a process of instructing the interference avoidance measure to the at least one movable object in the real space. According to another aspect of the present disclosure, a computer-readable non-transitory medium having the program recorded thereon is provided. Effects of the Invention

[0011] According to the present disclosure, interference avoidance between objects or between an object and a person in a real space can be controlled with high accuracy while securing a time margin. Brief Description of the Drawings

[0012] [Figure 1] It is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 2] It is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 3] (A) and (B) are diagrams illustrating at least one example of several embodiments of the present disclosure. [Figure 4] It is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 5] It is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 6](A) and (B) are diagrams illustrating at least one example of several embodiments of the present disclosure. [Figure 7] This is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 8] This is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 9] This is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 10] (A) and (B) are diagrams illustrating at least one example of several embodiments of the present disclosure. [Figure 11] This is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 12] This is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 13] This is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 14] This is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 15] (A) to (C) are diagrams illustrating at least one example of several embodiments of the present disclosure. [Figure 16] This is a diagram illustrating at least one example of several embodiments of the present disclosure. [Figure 17] This is a diagram illustrating at least one example of several embodiments of the present disclosure. [Mode for Carrying Out the Invention]

[0013] Several embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0014] Figure 1 is a schematic diagram illustrating some examples of several embodiments of a system related to information processing in this disclosure. Figure 1 is an example of a non-limiting system configuration, and can correspond to system configurations such as Figure 7, which will be described later. Referring to Figure 1, in this system 100, multiple robots 201-203 and a person (not shown) are placed in a real space such as a work site, and a sensor group 210 is provided to acquire information about the real space such as the work site. It should be noted that the work site is merely an example, and in this disclosure, the real space to be controlled is not limited to a work site. Although not particularly limited, the sensor group 210 may include multiple image sensors, ultrasonic distance sensors, TOF (Time of Flight) sensors, etc., installed in the work site. It may also be a LiDAR (Light Detection And Ranging) that acquires information from the entire object using a TOF sensor, saves it as point cloud data, and enables the creation of a 3D model. At least one of the sensor group 210 may be an IoT (Internet of Things) device.

[0015] Figure 1 illustrates robots 201 to 203 as examples of multiple robots. Each of these robots includes the three elements of a robot: control systems 201c to 203c, drive systems 201d to 203d, and sensor systems 201s to 203s. The control systems 201c to 203c process information from the sensor systems 201s to 203s to control the whole system (they include a communication system for communicating with a higher-level device). The drive systems 201d to 203d include actuators, power transmission mechanisms, arms, hands, and movement mechanisms. The sensor systems 201s to 203s may include at least one of the following: image sensors, pressure sensors, thermal sensors, humidity sensors, acceleration sensors, gyroscopes, ultrasonic sensors, proximity sensors, displacement sensors, timers, antenna elements, etc. Note that the work site is assumed to be indoors and positioning systems such as GPS (Global Positioning System) cannot be used. However, this disclosure does not exclude the use of GPS sensors (receivers), etc., by robots in cases such as when the work site is outdoors.

[0016] Robots 201-203 may be industrial robots such as autonomous mobile robots (AMRs), transport robots, serving robots, and autonomous or semi-autonomous robotic arms, or they may be unmanned aerial vehicles (UAVs) or other types of drones. Furthermore, robots that have moving parts (moving parts, moving units) but do not move, such as stationary robotic arms, as well as autonomous mobile robots (wheeled robots for movement only, legged robots for walking only (drones, etc. (Unmanned Auto Vehicles: UAVs))), and mobile robots with moving parts, may be collectively referred to as movable objects. Also, drones are not limited to autonomous unmanned aerial vehicles or legged robots, and may be unmanned aerial vehicles that perform tasks such as transporting goods at a work site under remote control by a controller.

[0017] The system may include controllers (information processing devices) 301-303, each installed, for example, at or near the work site, for multiple robots 201-203 at the work site, and communicating with each robot 201-203 using a predetermined protocol to control the robots 201-203. Note that one controller may control multiple robots, or multiple controllers may control one robot. The controllers (information processing devices) 301-303 may be connected to a higher-level information processing device 310 via, for example, a local area network, and the information processing device 310 may be connected to the information processing device 110 via at least one of the following: a local area network, a mobile communication network, a wide area network, etc. While not particularly limited, the robot software may be implemented using open-source ROS 2, etc.

[0018] The sensor group 210 may upload acquired image information, distance information, etc., to the information processing device 310, associating them with the identification information (ID) of the sensor group 210. Similarly, information acquired by the sensor systems 201s to 203s of robots 201 to 203, and the current state information of the control systems 201c to 203c and drive systems 201d to 203d of robots 201 to 203 may be uploaded to the information processing device 310, associating them with the identification information (Identity Information: ID) of robots 201 to 203.

[0019] The information processing device 110 receives data about the real space (e.g., a work site) collected from the sensor group 210 by the information processing device 310, and constructs a digital twin space by mapping the real space to a virtual space.

[0020] The information processing device 110 further performs simulations of objects and people on the digital twin. Models of objects and people are represented, for example, in a 3D digital space and reproduced as digital models on the digital twin. Based on real-time information from the real space, events that may occur in the real space at a future point in time are simulated in the virtual space, making them predictable in advance.

[0021] According to this disclosure, by simulating objects or people that are to be controlled on a digital twin, which is a replica of the real space, it is possible to predict the occurrence of interference between such objects or people, determine interference avoidance measures, and control the objects in the real space via the corresponding controller through the information processing device 310.

[0022] Figure 1 illustrates an example of a hierarchical structure from the perspective of the transmission and reception of information (commands, data) and control systems. However, depending on the implementation, for example, in the case of an intelligent robot equipped with AI (Artificial Intelligence) (e.g., an autonomous robot), it is of course possible to communicate directly with the information processing device 310.

[0023] Furthermore, in Figure 1, the information processing device 310 may also be configured to be integrated with the information processing device 110. Alternatively, if the computational load (computational amount) of the digital simulation is large on the information processing device 110, another information processing device may be provided as a backend, and the digital simulation may be executed by the backend information processing device (backend machine). When the digital simulation is executed in the backend, it may be configured as a BaaS (Backend as a Service), with the cloud-side application server handling the task. In addition, if the processing performance of the information processing device 110 is high and the number of robots to be controlled is relatively small, the information processing device 110 may be configured to perform digital twin construction, simulation, and control of multiple robots all on a single unit.

[0024] The information processing device 110 constructs an interference region (interference region) where interference occurs with a controlled object such as a robot (for example, when its position in 3D coordinates overlaps with that of other objects or people) based on a simulation on the digital twin. For example, it acquires information such as the position, velocity (acceleration), motion, and posture of a robot device in real space, reproduces (reflects) it in a model in virtual space, and from the time of information acquisition, it runs a simulation of the robot's motion, movement, etc. on the digital twin at a desired time scale to a desired future point in time, and acquires the time progression of the interference region for the model of the controlled robot on the digital twin.

[0025] The information processing device 110 acquires positional information and motion / movement information in real space for other objects and people that may be obstacles, and based on the acquired information, it similarly simulates their motion and movement for models on the digital twin, and determines from the interference region of the model of the robot to be controlled on the digital twin whether or not there is a possibility of interference between the model of the robot to be controlled and the models of other objects and people that may be obstacles.

[0026] It is also possible to simulate the robot's operation and verify whether the robot arm's links or end-effector (hand) interfere with (collide with) the floor, sides (side walls or installed objects), or top surface installed objects. In other words, other obstacles may be stationary objects, may interfere with the arms of other robots depending on the robot's operation, or may interfere with workers, etc. Interference may include mechanical collisions, but it may also include other objects entering an exclusive area (2D or 3D space) that one object should occupy, or intrusion into the vicinity of said exclusive area (2D or 3D space). In other words, this exclusive area may be called an interference area. The interference area may include areas where a loose contact occurs between a part of the controlled object and a part of another object or person, or areas where there is no contact, but which are further away from the outer edge of the robot's workspace, for example from the perspective of worker safety. Therefore, when simulating the occurrence of interference on a digital twin, the parameters used to determine the interference region of the controlled object may be appropriately determined according to the robot being controlled, such as the outer edge (outermost periphery) of the three-dimensional space formed by the arm trajectory + α. The simulation may also be performed using existing robot simulators (robotics simulation software products) that create, analyze, and verify the robot's motion paths and processes.

[0027] The information processing device 110, based on the movement and motion information of the 3D model of the robot to be controlled in the virtual space and the action plan of the robot to be controlled, predicts that interference (collision) with other obstacle models may occur at a certain point in the future. In such cases, it attempts to avoid the interference on the digital twin and determines a control method for the robot. It may also output avoidance measures for the interference based on a learning model created by machine learning using various interference occurrence situations that may occur in the work site as training signals.

[0028] The information processing device 110 reschedules the action plan of the on-site (robot) based on the avoidance measures determined on the digital twin, and controls the robot based on this rescheduling. The rescheduling of the robot's action plan based on the interference avoidance measures may be automatically generated based on a machine learning model or rule-based system. Alternatively, the information processing device 110 may display one or more candidate interference avoidance measures on the screen of the display device of the information processing device 110 that displays the digital twin space, and the operator may set (select) the interference avoidance measures for the robot on the screen. Based on this operation, the information processing device 110 may automatically modify a part of the action plan for the robot to be controlled (for example, the action plan for future times after the time when the data (work site data) used in the simulation was sampled).

[0029] Figure 2 is a schematic diagram illustrating an example of the processing of the information processing device 110 of the system 100 described with reference to Figure 1. Referring to Figure 2, the information processing device 110, before the start of operation, refers to the storage device 101 which stores data collected from the real world, constructs a digital twin, and generates an action plan, etc., using the digital twin (step S10).

[0030] For example, the information processing device 110 creates a digital twin model of a person based on design information of a model (3D model) corresponding to an object in real space (such as a robot), and if necessary, stores it in the storage device 103. The information processing device 110 then generates a digital twin of the real space (for example, a work site) and the objects in the real space, replacing them with 3D models in a virtual space, and stores it in the storage device 104.

[0031] The information processing device 110 displays a digital twin, which is a virtual reproduction of the work site where people and objects exist, on a display device (not shown).

[0032] For example, when an operator (user) creates or modifies an action plan for a controlled object (e.g., a robot), they may specify the starting position and destination position of the 3D model of the controlled object (e.g., a robot) on a digital twin displayed on a display device (not shown) using, for example, a mouse, to automatically generate a movement path, create an action plan (movement path, movement speed, arrival time at the destination, etc.) for the controlled object (e.g., a robot), and store the formulated action plan in the storage device 102. In generating the movement path for the controlled object (e.g., a robot), the system may verify through simulation on the digital twin that interference with other obstacles can be avoided before generating the movement path. For example, if there are only a few controlled objects (e.g., robots) at the work site, the operator (user) may set the movement path from the starting position to the destination position on the digital twin by dragging and dropping the 3D model of the controlled object (e.g., a robot) using a mouse.

[0033] The formulation and generation of action plans differ depending on the type of robot. For example, in the case of a robotic arm, the starting point (initial posture) and ending point (posture where the end effector matches the target value) can be specified within the free region of the joint space on a digital twin. A path connecting the starting and ending points can then be simulated on the digital twin to generate the robotic arm's action plan, thereby automatically generating the action plan for the robotic arm. In this case, the set of points (space) where the robotic arm interferes with obstacles is defined as the interference region.

[0034] Regarding humans, it is not necessary to create an action plan on the digital twin. This is because, in terms of interference avoidance, humans are the top priority (in principle, if interference with humans is possible, the robot should take measures to avoid interference), humans can move freely according to their own will and judgment, and if they determine that interference with robots may occur in the future, they can decide on avoidance measures themselves. In addition, the information processing device 110 may use AI / machine learning-based object detection technology to recognize humans at the work site from image data (video or still images) from image sensors that capture images of the work site.

[0035] Figure 3(A) shows a simulated example of a work site. A mobile robot 20 (actual robot) equipped with two robot arms moves between side walls (or equipment or shelves) 21A, 21B, and 21C. Image sensors 22A to 22C each capture images of a predetermined area of ​​the work site. Image sensors 22A to 22C may be configured to change their posture in response to the movement of the object being tracked and to follow the object. Other sensors, etc., are omitted in Figure 3(A).

[0036] Figure 3(B) schematically shows an example of displaying a digital twin 400, which maps the work site 200 of Figure 3(A) into a virtual space, on a display terminal. On the screen displayed on the digital twin 400, the initial position 45 and target position 46 of the mobile robot (3D model) 40 (a 3D model in virtual space of the mobile robot 20 (actual robot) of the work site 200 in Figure 3(A)) can be positioned using the mouse cursor 27, and the path plan from the initial position 45 to the target position 46 can be automatically calculated by simulation on the digital twin 400. For example, a potential function can be defined between the target position 46 and the positions of obstacles such as walls and shelves 41A, 41B, 41C, etc., and a path that avoids obstacles and moves to the target position can be derived by, for example, using the gradient of the potential function as the direction of travel. The digital twin 400 model stores information on the position and size (width, length, height, etc.) of walls and shelves 41A, 41B, 41C, etc.

[0037] As described above, if there are few robots in the actual work site 200, the operator (user) may set the movement path 43 (the path from the initial position to the target position) by specifying it with a mouse or the like on a screen displaying a digital twin 400 that reproduces the work site 200 in a virtual space. Depending on the implementation, the operator (user) may also create and edit an action plan by referring to the action plan (path plan) of other robots and appropriately setting and inputting arrival times, etc., at multiple points up to the target position 46 from a display terminal or the like.

[0038] The information processing device 110 may perform a simulation of the movement of the mobile robot (3D model) 40 on the digital twin 400 based on the created action plan for the robot to be controlled, verify that there is no interference with other robots or stationary objects, and then finalize the action plan. For example, if the number of autonomous mobile robots increases, it may become difficult to formulate (optimize) the movement plan for each robot on the display terminal screen. In such cases, the information processing device 110 may be configured to automatically generate the action plan for the robot to be controlled by simulation on the digital twin 400. Alternatively, the action plan for the robot to be controlled may be generated by using a combination of simulation on the digital twin 400 and settings input by the operator (user).

[0039] In the information processing device 110, the action plan created on the digital twin 400 is transmitted, for example, to the information processing device 310, which then sets it on the target robot in the real space via the corresponding controller, and allows it to be operated in the work site 200 in Figure 3(A) in accordance with the movement path 43 in Figure 3(B). In Figure 3(B), the mobile robot (3D model) 40 is shown in the same form as the mobile robot 20 (actual robot) simply for the sake of drawing convenience, but a surface model, solid model, wireframe model, etc. may be used as the 3D model, and of course, a simplified model such as a box may also be used for display. Furthermore, the side walls, equipment, sensors (cameras), etc., indicated by reference numerals 21A~21C, 22A~22C, etc. in Figure 3(A) do not need to be reproduced and displayed if they are not required for operation on the digital twin 400.

[0040] Referring again to Figure 2, after the above pre-operation processing, the information processing device 110 performs a simulation on the digital twin during the operation process (step S20).

[0041] In the operational process, information from the real space is acquired at a predetermined sampling time and reflected in a 3D model on the digital twin. n-k If there was no interference with the controlled robot arm, etc. at (k≧1), then at the sampling time (t n Based on the motion and movement information (position information, velocity vector information, etc.) of the controlled object acquired by ), a future time t n+m The operation and movement of the controlled object are simulated at a future time point t (m≧1, preferably m>2), and it is verified whether interference occurs between the robot's arm and other obstacle models on the digital twin. n+m The simulation can be performed with a resolution of several times the sampling time (several times the sampling time), or with a resolution of less than the sampling time (for example, if the sampling time is 60 seconds, the simulation can be performed every 30 seconds to check for interference).

[0042] The information processing device 110 determines the sampling time (t n ) against a certain future point in time t n+m If it is determined that interference may occur (m≧1), interference avoidance measures are taken. The information processing device 110 rewrites (updates) the action plan for the controlled object at a future point in time in response to the interference avoidance measures, and stores the revised action plan in the storage device 102. Note that storage devices 101-104 may be one storage device or multiple storage devices. Storage devices 101-104 may be semiconductor memory such as RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc.

[0043] Figure 4 is a diagram illustrating in detail an example of processing step S10 in the information processing device 110 in Figure 2. Referring to Figure 4, the information processing device 110 receives the action plan created on the digital twin from the storage device 102 (step S101).

[0044] The information processing device 110 reflects data acquired from the actual work site using sensors and other means into individual 3D models on the digital twin, and simulates whether or not interference occurs with respect to the robot being controlled (step S102).

[0045] If there is a possibility of interference occurring between the 3D model of the robot to be controlled and the 3D model of another obstacle at a certain point in the future (Yes branch in step S103), the information processing device 110 generates interference avoidance measures for the controlled object where interference may occur, and the action plan from the current sampling time onward is revised (rescheduled) (step S104).

[0046] The information processing device 110 simulates whether or not interference occurs with respect to the robot to be controlled, according to the rescheduled action plan (step S102).

[0047] If there is no possibility of interference between the 3D model of the robot to be controlled and other models (No branch in step S103), operation is started based on the original action plan or the action plan modified in step S104 (step S200).

[0048] In addition, the rescheduling of the action plan in step S104 of Figure 4 may modify the action plans of other obstacles that have been determined to potentially cause interference, as well as the robot being controlled. Furthermore, if the interference of the robot being controlled is between the robot arm and a fixed side wall, or between the tip of the robot arm and the passageway (floor), it is not necessary to modify the action plans of other obstacles. However, if the side wall (for example, at least one of 21A to 21C in Figure 3(A) (at least one of 41A to 41C in Figure 3(B), which are 3D models of these on the digital twin 400)) is movable, such as a sliding wall or a movable partition storage unit, the arrangement of the side wall may be changed if possible.

[0049] Figure 5 is a diagram illustrating in detail an example of processing step S20 in the information processing device 110 shown in Figure 2. Referring to Figure 5, the information processing device 110 collects data from the real space upon the start of operation (step S201).

[0050] Based on the collected data and action plan, the information processing device 110 simulates on the digital twin whether interference may occur with the controlled object due to the actions, movements, etc., of the controlled object at a future point in time (step S202).

[0051] If the information processing device 110 determines that there is no possibility of interference (No branch in step S203), it determines whether all actions in the action plan have been completed (step S204). If they have been completed (Yes branch in step S204), the operation ends (step S205). If all actions in the action plan have not been completed (No branch in step S204), processing from the collection of real-space data in step S201 is performed.

[0052] If the information processing device 110 determines in step S203 that there is a possibility of interference (Yes branch in step S203), it performs control to temporarily stop the operation of the controlled object in the real space (step S206). The information processing device 110 sends a control signal (command) to the information processing device 310 in Figure 1, and the information processing device 310 sends a command to the corresponding controlled object robot 201 via the controller 301 to forcibly stop it temporarily.

[0053] Next, the information processing device 110 determines interference avoidance measures for the controlled object on the digital twin (step S207).

[0054] Then, the information processing device 110 reviews (reschedules) the current action plan based on the action plan that would be implemented if the interference avoidance measures were taken (step S208).

[0055] The information processing device 110 performs a simulation again on the digital twin based on real-space sampling data and the revised action plan (including interference avoidance actions for the controlled object) to verify whether interference may occur at a future point in time (step S209).

[0056] If the information processing device 110 determines that there is a possibility of interference based on the simulation results on the digital twin (Yes in step S210), it may select a different interference avoidance measure (step S207).

[0057] The information processing device 110 modifies the action plan based on the action plan when interference avoidance measures are taken (step S208).

[0058] The information processing device 110 performs another simulation on the digital twin to determine if there is a possibility of interference (step S209).

[0059] If the information processing device 110 determines that there is no risk of interference up to a predetermined future time (No. in step S210), it sets the revised action plan to the robot to be controlled in the real space (S211) and resumes operation (step S212).

[0060] In an implementation where interference avoidance measures have been determined and it has been confirmed that the possibility of interference occurring to the controlled robot until a predetermined time thereafter is extremely small, the information processing device 110 may determine the interference avoidance measures for the occurrence of interference on the digital twin, omit the simulation of the interference avoidance measures on the digital twin (step S209 in Figure 5), and set them on the controlled robot in the real space.

[0061] In step S206 of Figure 5, if it is determined that interference is possible, the operation of the robot being controlled in the real space is temporarily stopped. However, depending on the implementation, if the simulation results on the digital twin indicate that the timing at which interference may occur is far in the future from the current sampling time (for example, tens to hundreds of times the sampling time, tens of seconds to several minutes ahead), then instead of step S206, the system may decide on interference avoidance measures on the digital twin, revise the action plan, and then perform verification through simulation.

[0062] If the timing at which interference may occur in the simulation is in the near future from the current sampling point (for example, several times the sampling time, or several seconds later), the process may include a determination process to temporarily suspend the operation of the robot being controlled in the real space. Note that the processing flow in Figure 5 is just one example, and various modifications may be included as long as they do not deviate from the purpose of ensuring safety through interference avoidance.

[0063] When the controlled robot receives control information, including an action plan, from the information processing device 110, it performs actions, movements, etc., in accordance with the action plan under the control of the information processing device 110.

[0064] As an autonomous mobile robot, there is also a technology that allows the robot to autonomously avoid obstacles while moving, using distance sensors such as cameras and LiDAR. In this disclosure, an action plan, including a pre-set travel path and speed information, is set in the information processing device 110 for the robot to be controlled (201-203 in Figure 1). The robot to be controlled (201-203 in Figure 1) may be a mobile (traveling) robot that autonomously travels from an initial position to a target position by following a set path, or it may be a robot arm that moves its arm along a trajectory according to the action plan. In the case of a traveling robot, it can be considered an optimization problem that includes path planning (path, speed) and moves from a starting point to a destination point while passing between obstacles such as the sides. The position and size of the obstacles are assumed to be reflected on the digital twin from real space. Although not particularly limited, there are also stationary robot arms that can determine a target trajectory and move while avoiding obstacles when a start point and an end point, or multiple points, are specified, but here we are focusing on robots that operate based on an action plan (target trajectory) set from the controller side. In this case, the action plan may include a plan of how the joint will move (generating the trajectory of the joint space).

[0065] Figure 6 illustrates the relationship between the travel distance of the mobile robot (3D model) 40 based on the action plan set in the information processing device 110 and the actual travel distance of the mobile robot (actual robot) 20 in real space. Figure 6(A) shows the correspondence between the action plan created on the digital twin and the mobile robot (3D model) 40. The sampling time t for collecting data in real space. nIn this scenario, as shown in Figure 6(B), the actual robot (real robot) 20 may move later or earlier than the planned distance based on the action plan due to differences in planned values ​​such as the coefficient of friction between the robot's wheels and the floor, and differences in planned values ​​for the robot's load. The information processing device 110 may change the speed of the actual robot (real robot) 20 based on simulations on a digital twin using collected real-world data. In the simulation on the digital twin, in addition to the interference simulation described above, the device may calculate the arrival time at the target location based on the current position information, speed information, route information, etc., and adjust the speed for each pre-set route section, or, in some cases, change from a route with many obstacles to a route with fewer obstacles, in order to arrive at the time of the action plan.

[0066] Figure 7 is a schematic diagram showing an example of a system according to some embodiments of the present disclosure. The work site 200 includes a remotely operated robot (stationary robot arm) 201, an autonomous robot arm 202, an autonomous mobile robot 203, and a person (worker) 204 who transports goods (luggage) 205. The autonomous mobile robot 203 receives an action plan, such as a travel route and speed information, which is set in advance on the information processing device 110, and autonomously travels from the initial position to the target position by following the route set in the action plan. Data from the work site 200 is collected by sensing information from IoT sensors such as an image sensor (camera) 210A, LiDAR 210B, and ultrasonic sensor 210C, and transmitted to the information processing device 110. The information processing device 110 places the 3D models 401, 402, 403, and 404 of the remotely operated robot 201, autonomous robotic arm 202, autonomous mobile robot 203, and human (worker) 204 on the digital twin 400 at positions corresponding to the work site 200, based on the collected real-time data (motion information and position information). Note that the 3D models 401, 402, and 403 may be generated using wireframes or the like from 3D CAD (Computer Aided Design) data of the robots 201, 202, and 203. Alternatively, even more simplified models may be used.

[0067] Furthermore, when identifying robots 201, 202, and 203 and calculating their position information using image data from image sensors (cameras), identifiers (markers) may be attached to predetermined positions on robots 201, 202, and 203. Augmented Reality (AR) markers may be used as markers. However, any marker can be used as long as it is possible to calculate the relative position between the camera and the marker. For the problem of calculating the relative coordinates between the camera and the robot from the image data captured by the camera that photographed the robot, the PnP (Perspective-n-Point) problem-solving method can be used (Reference 1). Alternatively, reference markers may be attached to stationary objects such as side walls and shelves in the real space (work site), and the relative position of the robot to the reference markers may be determined from the reference markers included in the image data captured by the robot's camera. The absolute coordinate coordinates of the robot may then be determined from the absolute coordinate coordinates of the reference markers and the relative position of the robot to the reference markers.

[0068] Figure 8 illustrates how the presence or absence of interference with the controlled object is simulated based on data collected from real space. The remotely operated robot 201, the autonomous robotic arm 202, and the human 204 are controlled by information processing devices 301, 302, and 304, respectively. Although not particularly limited, in the example of Figure 8, the autonomous mobile robot 203 is controlled by information processing device 310. Furthermore, information processing device 311, which receives instruction input from operator 312, transmits remote operation instructions to information processing device 301, and information processing device 301 controls the remotely operated robot 201 (task control, etc.) based on these instructions. Information processing device 301 may also transmit information received from the remotely operated robot 201 (e.g., status information such as position, movement, and task completion) to information processing device 311. The information received from the remotely operated robot 201 may be displayed on, for example, an unshown display terminal of information processing device 311, which can be monitored by operator 312. Furthermore, the information processing device 301 may transmit information received from the remotely operated robot 201 to the information processing device 310. The information processing device 310 may be configured to act as a higher-level device for the information processing devices 301, 302, 304, and 311, receiving information data from the information processing devices 301, 302, 304, and 311 and transmitting it to the information processing device 110. The information processing device 310 may also receive instructions from the information processing device 110 and transmit instructions to each of the information processing devices 301, 302, 304, and 311.

[0069] The information processing device 110 is, for example, • Generation of a digital twin by mapping real space onto virtual space. • Real-time updates of the digital twin based on data collected periodically from the real world. • Simulation of whether or not interference occurs based on a 3D model of the controlled object on a digital twin. • Simulation results indicate a possibility of interference, and the selection of interference avoidance measures. • Reschedule and verify the action plan, and then instruct the control of the controlled object based on the rescheduled action plan.

[0070] Reference numeral 411 represents an interference region between the arm trajectories 401a, 402a of the three-dimensional models 401, 402 of the teleoperated robot 201 and the autonomous robot arm 202. Reference numeral 412 represents an interference region between the arm trajectory of the three-dimensional model 402 of the autonomous robot arm 202 and the planned travel path of the three-dimensional model 403 of the autonomous mobile robot 203. Reference numeral 413 represents an interference region with the three-dimensional model 404 of the human 204 at the planned travel destination of the three-dimensional model 403 of the autonomous mobile robot 203.

[0071] Figure 9 is a diagram for explaining simulation of interference regions in the digital twin 400 of the three-dimensional models 401, 402 of the teleoperated robot 201 and the autonomous robot arm 202 in Figure 8.

[0072] Referring to Figure 9, at the sampling time t n-k Based on the motion plan of the teleoperated robot 201 from the positions and motions of the links in the real space acquired at (k≧1), the arm trajectory up to the future time t after the current sampling time tn n+m (m≧1) 401a (point cloud) is calculated. At the sampling time t n Based on the motion plan of the autonomous robot arm 202 from the positions and motions of the links in the real space acquired at, the arm trajectory up to the future time t after the sampling time tn n+m (m≧1) 402a (point cloud) is calculated. The three-dimensional shape of the trajectory of the hand (gripper, fingers) of the robot arm is a torus shape. In the example of Figure 9, the arm trajectory 401a and the arm trajectory 402a are set as circular trajectories at substantially the same height, and if the operation is continued as it is, at the future time t n+m An interference region 431 occurs at the tip of the gripper (fingers) of the arm.

[0073] Figure 10(A) is a diagram illustrating an embodiment. Reference numeral 434 indicates the interference region between the arm trajectory of the 3D model 401 of the remotely operated robot 201 and the planned (future) travel path of the 3D model 403 of the autonomous mobile robot 203. Reference numeral 435 indicates the interference region between the 3D model 404 of the autonomous mobile robot 203 and the 3D model 404 of the human 204 on the planned travel path of the 3D model 403 of the autonomous mobile robot 203.

[0074] Figure 10(B) is a two-dimensional matrix showing the past, present, and future interference correspondence between two controlled objects A and B.

[0075] • No interference countermeasures are necessary for the past of object A and object B (1).

[0076] • B takes emergency action against interference between the past of object A and the present of object B (2) (this is avoided in this disclosure). Interference between the present of object B and the past of object A corresponds to the case where A is a stationary object, etc., and is in the same position in the past and present.

[0077] • Regarding the past of object A and the future of object B, B will deal with them in accordance with this disclosure (3). Interference between the future of object B and the past of object A corresponds to the case where A is in the same position in the past and future.

[0078] • Object A takes emergency action in response to interference between the present state of object A and the past state of object B. This applies when B is in the same position in the past and present (4).

[0079] • Emergency response is taken by A and B in response to the current interference between object A and object B (5) (this is avoided in this disclosure).

[0080] • Regarding the present state of object A and the future state of object B, B will deal with this in accordance with the disclosure (6). This corresponds to the case where A is in the same position in the present and future.

[0081] • The future of object A and the past of object B are addressed by object A (7). This is done when B is in the same position in the past and future.

[0082] • The future of object A and the present of object B will be dealt with by object A (8).

[0083] • Depending on the policy, the future interference between object A and object B will be addressed on either side A or B (9). If A is a person and B is an object such as a robot, and human priority is given, then interference avoidance measures will be taken on side B. Priority will be determined according to the object being worked on. In optimal control, it is permissible to include human control as an option. For example, human work at the work site will be given priority. Human work will always be given priority and will not be stopped.

[0084] Figure 11 is a schematic diagram illustrating an example of control over a real-world device when the information processing device 110 determines that interference exists through simulation on a digital twin.

[0085] The information processing device 110 determines that interference exists through simulation on the digital twin 400 (Simulation & Interference Determination in Step S501).

[0086] The information processing device 110 selects an interference avoidance measure and verifies whether interference avoidance is achieved through simulation on the digital twin 400 (verification regarding the elimination of interference risk in step S502).

[0087] The information processing device 110 reschedules its action plan in response to interference avoidance measures (step S503).

[0088] The information processing device 110 instructs the information processing device 310 to change the action plan (step S504).

[0089] The information processing device 310 instructs the information processing device 311, which controls the remotely operated robot 201, to perform a rollback, returning the arm of the remotely operated robot 201 to its previous position (step S505).

[0090] In this case, the information processing device 311, which controls the remotely operated robot 201 via the information processing device 301, may allow the operator 312 to control the remotely operated robot 201 and issue control instructions from the information processing device 311 to the information processing device 301 (step S506). In this case, for example, the operator 312 may set a rollback operation for the remotely operated robot 201 on a display terminal (not shown) of the information processing device 311 and transmit this setting information to the information processing device 301 as control instruction information.

[0091] The information processing device 310 instructs the autonomous robot arm 202 to stop operating via the information processing device 302 that controls the autonomous robot arm 202 (step S507).

[0092] The information processing device 310 instructs the autonomous mobile robot 203 to deviate from the path (step S508).

[0093] The information processing device 310 may also notify the human 204 via the information processing device 304 that the autonomous mobile robot 203 is taking a detour by voice or other means (step S509).

[0094] Figure 12 illustrates an example of interference avoidance measures in the digital twin 400 when the passageway is narrow. If the simulation on the digital twin 400 predicts that the mobile robot (3D model) 40 will interfere with a human (3D model) 404 at a certain point on its planned travel path, the mobile robot (3D model) 40 may reduce its speed, allow the human 404 to pass the predicted interference point first, and then return to its original speed to allow the human 404 to pass the point. As an example, Figure 12 shows a scenario in real space where the human cannot perceive the mobile robot's movement due to walls, etc., and the human unexpectedly encounters the mobile robot and interferes (collides) with it at the predicted interference point. If interference 433 between a human (3D model) 404 and the robot (3D model) is predicted to occur at a future point in time, it is possible to increase the speed of the mobile robot (3D model) 40 so that it passes the point before the human 404 reaches it. However, as a general rule, the policy is not to increase the speed, such as when an upper limit on the mobile speed is set within the work site.

[0095] In the example in Figure 9, the sampling time t is as follows: n Based on the data collected, simulations on the digital twin 400 show that future time point t n+m If the possibility of interference between the hand of robot arm 401 (3D model of remotely controlled robot 201 in Figure 8) and the hand of robot arm 402 (3D model of autonomous robot arm 202 in Figure 8) is detected, as a countermeasure to avoid interference, the movement speed of robot arm 402 is reduced or stopped, and after the arm of robot arm 401 passes through the interference region 431, the movement speed of robot arm 402 is increased so that it passes through the interference region 431 along the trajectory 402a. In this case, depending on the sampling time, timing constraints such as the time required to change the action plan (movement) of robot arm 402 and control the corresponding autonomous robot arm 202 (Figure 8) may be considered, and the future time point t n+m Preferably, the next sampling time t n+1This is set to a point in time later than (m≧2). Note that the control of the above movement speed may be performed by the robot arm 401.

[0096] Figure 13 illustrates an example of interference avoidance measures in the digital twin 400 when the passageway is narrow. If a simulation on the digital twin 400 predicts that the mobile robot (3D model) 40 will interfere 433 (collision) with a human (3D model) 404 at a certain point on the planned travel path, the mobile robot 40 (3D model) is controlled to perform a rollback operation and return along the path it came from as an interference avoidance measure. The mobile robot in the real space is instructed to perform the same rollback operation as the mobile robot (3D model) 40 on the digital twin 400. If a simulation on the digital twin 400 confirms that no interference will occur by a predetermined future time, the mobile robot in the real space may be instructed to start traveling along the planned travel path.

[0097] In the example in Figure 9, for example, the next sampling time t n+1 Based on the data collected, simulations on the digital twin 400 show that future time point t n+m If, for example, m≧1, it is detected that there is a possibility of interference between the hand of robot arm 401 (3D model of remotely operated robot 201 in Figure 8) and the hand of robot arm 402 (3D model of autonomous robot arm 202 in Figure 8), then as a measure to avoid interference, robot arm 402 is rolled back so that after the arm of robot arm 401 passes through the interference region 431, it passes through the interference region 431 along its original trajectory 402a. Note that the above rollback control may also be performed on robot arm 401.

[0098] Figure 14 illustrates an example of interference avoidance measures when the passage in the digital twin 400 is wider than the passage in Figure 13. If a simulation on the digital twin 400 predicts that the mobile robot (3D model) 40 will have an interference 433 (collision) with a human (3D model) 404 at a certain point on its planned travel path, an interference avoidance measure may be selected in which the mobile robot 40 (3D model) travels by bypassing the point where interference 433 (collision) is predicted to occur from the side. In this case, the action plan for the mobile robot 40 (3D model) is modified to include a change in its route, such as a detour to the left before the predicted point of interference on its planned travel path, and the mobile robot in the real space is instructed to travel by bypassing the point where interference (collision) is predicted to occur from the side based on this action plan.

[0099] In the example in Figure 9, for example, a simulation on the digital twin 400 based on the data collected at the current sampling time tn will show a future time t n+m If the possibility of interference between the hand of robot arm 401 (3D model of remotely controlled robot 201 in Figure 8) and the hand of robot arm 402 (3D model of autonomous robot arm 202 in Figure 8) is detected, the response to avoid interference is to modify the trajectory 401a of robot arm 401 so that it passes over (detours) the interference region 431. In this case, depending on the sampling time, timing constraints such as the time required to change the action plan (movement) of robot arm 401 and control the corresponding remotely controlled robot 201 (Figure 8) will be considered, and the future time point t n+m Preferably, the next sampling time t n+1 This is set to a point in time later than (m≧2). Note that the above detour control may also be performed on the robot arm 402 side.

[0100] In the above embodiments, several examples of interference between the mobile robot and the planned travel path of a human (Figures 12 to 14) have been described. Interference due to the robot's movement will be explained with reference to Figure 15. Figure 15 shows a motion simulation on the digital twin 400. As shown in Figure 15(A), the mobile robot 40 (3D model) equipped with a robot arm 40A is assumed to have a pre-programmed action plan to grasp an item from the floor and load it onto a platform 40D (or place an item from the platform onto the floor). In Figure 15(B), the space where the robot arm 40A in Figure 15(A) was located becomes free space, and in this state, the space is not an interference area and is passable. Subsequently, if another robot 42 or the like travels to the position where the robot arm 40A in Figure 15(A) was located (or remains at that position), and the robot arm 40A moves back towards the floor as shown in Figure 15(C), interference 423 may occur. The hand (end effector) of the robot arm 40A is equipped with an image sensor (camera) and proximity sensors for grasping objects, but it is generally difficult for image sensors and proximity sensors to recognize objects passing from the side. If an obstacle is placed directly beneath or passes by the hand of the robot arm 40A while the hand is grasping an object on the floor, interference 423 may occur. According to this disclosure, a simulation of the movement of another robot model (not shown) that will be an obstacle and the movement of the mobile robot 40 (3D model) is performed based on their current positions, movements, and action plans, and if the occurrence of interference as shown in Figure 15(C) is predicted, the movement of the mobile robot 40 (3D model) can be rolled back to the state shown in Figure 15(B) to avoid the interference.

[0101] Figure 16 is a schematic diagram illustrating the configuration of the information processing device 110. The information processing device 110 includes at least one processor 501 (a configuration with multiple processors is also possible, as shown by the dashed line), memory (storage device) 502, display device 503, storage 504, and communication interface (IF) 505, which are capable of sending and receiving control signals and data via a bus system 506 (the bus system 506 may be configured to include an internal bus and an external bus). The memory 502 may be configured to include semiconductor storage such as RAM (Random Access Memory), ROM (Read Only Memory), or EEPROM (Electrically Erasable and Programmable ROM), or an HDD (Hard Disk Drive), CD (Compact Disc), DVD (Digital Versatile Disc), etc. At least one processor 501 executes a program (not shown) stored in the memory 502 to realize the processing described with reference to each figure. The communication interface 505 may be configured to include a network interface card, transceiver, etc., and to communicate with other information processing devices and sensors via a LAN (Local Area Network), a WAN (Wide Area Network) such as the Internet, a wireless LAN, a mobile communication network, etc.

[0102] Figure 17 is a diagram showing an example of the module configuration of a program stored in the memory (storage device) 502, as explained with reference to Figure 16, and shows the functions realized by execution by the processor 501. Note that each module in Figure 17 may share functions with, for example, the information processing device 310 in Figure 1. Referring to Figure 17, the program modules executed by the information processing device 110 include: an action plan creation unit 111 that generates an action plan for the robot to be controlled; a data collection unit 112 that collects data from sensors, etc., in real time; a model creation unit 113 that creates a virtual model (3D digital model) of real objects and people; a digital twin construction / update unit 114 that constructs a digital twin and updates objects and people on the digital twin based on the data (real-time data) collected by the data collection unit 112 at predetermined sampling times; and a simulation unit 115 that performs simulations of the position and movement of objects and people at future points in time on the digital twin in order to verify whether or not interference occurs. The system includes an interference avoidance countermeasure selection unit 116 that selects interference avoidance countermeasures if the simulation results predict interference, an action plan modification unit 117 that modifies the action plan of the controlled object according to the interference avoidance countermeasures, a verification unit 118 that verifies whether interference will be cleared at a future point in time by performing a simulation on a digital twin if the selected interference avoidance countermeasures are implemented, a control information setting unit 119 that sets control information for the controlled object in the real space to operate according to the modified action plan if the verification result of the verification unit 118 is OK, and a communication control unit 120 that receives sensing signals (digital data) from the sensor group 210 in Figure 1 and sends and receives signals and information with the information processing device 310 via the communication interface 505 in Figure 16 according to a predetermined protocol. The simulation unit 115 may be executed by another information processing device (not shown) to obtain the simulation results.

[0103] [Reference 1] International Publication No. 2020 / 138345

[0104] The embodiments / examples described above may be noted as follows (but are not limited to the following):

[0105] (Note 1) The information processing device includes at least one processor and at least one memory storing a program executable by the at least one processor. The at least one processor executes the program, acquires information on at least one movable object in real space, performs a simulation on the digital twin on a model of the at least one movable object reproduced on a digital twin which maps the real space to a virtual space, verifies whether at least a part of the model of the at least one movable object may interfere with a model on the digital twin corresponding to at least one other object included in the real space by a predetermined future time after the time the information was acquired, determines measures to avoid the interference if such interference exists with respect to the model of the at least one movable object by the future time, and instructs the at least one movable object in real space to take the measures to avoid the interference.

[0106] (Note 2) In the information processing apparatus of Note 1, at least one processor is As the aforementioned information, the position and / or motion information of the at least one movable object is obtained, Based on a pre-set action plan for at least one movable object, the system simulates the position and / or movement of the model of the at least one movable object up to the future point in time to construct an interference region, and determines whether or not interference is possible by checking whether the interference region overlaps with or is adjacent to at least a part of the model of the at least one other object.

[0107] (Note 3) In the information processing device of Note 1 or 2, the at least one processor is configured to select the interference avoidance measures and, when the selected interference avoidance measures are taken, to verify whether at least a portion of the model of the at least one movable object on the digital twin may interfere with the model of one or more other objects on the digital twin that are included in the real space at some point in the future.

[0108] (Note 4) In the information processing device described in Note 3, at least one processor is configured to modify the action plan for the at least one movable object included in the real space in accordance with the interference avoidance measures based on the interference verification results, and to perform control over the at least one movable object included in the real space based on the modified action plan.

[0109] (Note 5) In any of the information processing devices described in Notes 1 to 4, if at least one processor determines that the interference may occur up to a predetermined time, it is configured to issue an instruction to at least one movable object included in the real space to temporarily stop its operation.

[0110] (Note 6) In any of the information processing devices in Notes 1 to 5, the movable object is This includes at least one of the following: a stationary robot with movable parts, a mobile robot, a mobile robot with movable parts, an autonomous mobile robot, an unmanned aerial vehicle, an unmanned transport vehicle, and a human.

[0111] (Note 7) The interference control system includes a plurality of movable objects, a first information processing device, and a second information processing device. The first information processing device acquires information on at least one movable object, performs a simulation on the digital twin on a model of the at least one movable object reproduced on a digital twin that maps the real space where at least the plurality of movable objects are located to a virtual space, verifies whether at least a part of the model of the at least one movable object may interfere with a model on the digital twin corresponding to at least one other object included in the real space by a predetermined future time after the time the information was acquired, and if such interference occurs with respect to the model of the at least one movable object by the future time, determines measures to avoid the interference and transmits a signal to the first information processing device instructing the measures to avoid the interference. The second information processing device, in response to the signal from the first information processing device, instructs the at least one movable object to take measures to avoid the interference.

[0112] (Note 8) In the interference control system of Note 7, the first information processing device is: As the aforementioned information, position and / or motion information of the at least one movable object is acquired, and based on a pre-set action plan for the at least one movable object, the position and / or motion of the model of the at least one movable object up to the future time point is simulated to construct an interference region, and the presence or absence of the possibility of interference is determined by checking whether the interference region overlaps with or is adjacent to at least a part of the model of the at least one other object.

[0113] (Note 9) The interference control method acquires information on at least one movable object in real space, A simulation is performed on the digital twin, which is a digital twin that maps the real space to a virtual space, to verify whether at least a part of the model of the at least one movable object may interfere with the model on the digital twin that corresponds to at least one other object included in the real space by a predetermined future time point prior to the time when the information was acquired. If such interference is found to exist for the model of the at least one movable object by the predetermined future time point, measures to avoid the interference are determined, and the at least one movable object in the real space is instructed to implement the measures to avoid the interference.

[0114] (Note 10) The program obtains information on at least one movable object in real space, A simulation is performed on the digital twin, which is a digital twin that maps the real space to a virtual space, to verify whether at least a part of the model of the at least one movable object may interfere with the model on the digital twin that corresponds to at least one other object included in the real space, by a predetermined future time point prior to the time when the information was acquired. If the interference occurs with respect to the model of at least one of the movable objects by the future time, measures to avoid the interference will be determined. The at least one movable object in the real space is instructed to take measures to avoid the interference. It consists of a program that causes the processor to execute a process.

[0115] Furthermore, the disclosures in the above-mentioned patent documents and references are incorporated herein by reference. Within the framework of the disclosure of this application (including the claims), further modifications, adjustments, and combinations of embodiments or examples are possible based on the fundamental technical concept. Also, within the framework of the claims of this disclosure, various combinations or selections of various disclosed elements (including each element in each appendix, each element in each example, each element in each drawing, etc.) are possible. In other words, this disclosure, including the entire disclosure including the claims, naturally includes various modifications and alterations that a person skilled in the art could make in accordance with the technical concept. [Explanation of Symbols]

[0116] 12A, 12B Robot Arms 21A, 21B, 21C Side wall (shelf, equipment) 22A, 22B, 22C Image Sensor (Camera) 20 Mobile robots 27 Mouse Cursor 40 Mobile robots 40A Robot Arm 40D cargo bed 41A, 41B, 41C Side wall (shelf, equipment) 42 Robots 43. Travel Path (Route) 45 Initial position 46 Target position 100 Systems 101, 102, 103, 104 Storage device 110, 310, 311 Information Processing Devices 111 Action Plan Creation Department 112 Data Collection Unit 113 Model Creation Department 114 Digital Twin Construction and Update Department 115 Simulation Department 116 Interference Avoidance Measures Selection Section 117 Action Plan Revision Department 118 Verification Department 119 Control Information Setting Unit 120 Communication Control Unit 200 work sites 201, 202, 203 Robots 201c, 202c, 203c control systems 201d, 202d, 203d drivetrain 201s, 202s, 203s Sensor System 204 Humans 205 Goods 210 Sensors (IoT Sensors) 211 Image sensor (camera) 212 LiDAR 213 Ultrasonic distance sensor 301, 302, 304 Controllers (Information Processing Units) 312 Operator 400 Digital Twin 401, 402, 403, 404 Robots (Models) 401a, 402a Arm Tracks 405 Items (Models) Interference regions 411, 412, 413 412A, 412B, 412C side wall (shelf) Interference regions 413, 423, 433 422A~422C Side wall (shelf, equipment) Interference regions 431, 434, 435 433 Interference 500 Computers 501 Processor 502 memory 503 Display device 504 Storage 505 Communication Interface 506 Bus System

Claims

1. At least one processor, At least one memory containing a program executable by the at least one processor, Includes, The at least one processor executes the program, Obtain information on at least one movable object in real space, A simulation is performed on the digital twin for the model of at least one movable object that is reproduced on the digital twin, which maps the real space onto a virtual space. Verify whether, by a predetermined future time point prior to the time when the aforementioned information was acquired, at least a portion of the model of the at least one movable object may interfere with the model on the digital twin corresponding to at least one other object included in the real space. If the interference exists with respect to the model of the at least one movable object by the future time, measures to avoid the interference are determined, and the at least one movable object in real space is instructed to take the measures to avoid the interference. An information processing device configured to perform processing.

2. The at least one processor is As the aforementioned information, the position and / or motion information of the at least one movable object is obtained. The information processing apparatus according to claim 1, configured to simulate the position and / or movement of the model of the at least one movable object up to the future time point based on a pre-set action plan for the at least one movable object to construct an interference region, and to determine whether or not interference is possible by checking whether the interference region overlaps with or is adjacent to at least a part of the model of the at least one other object.

3. The information processing apparatus according to claim 1, wherein the at least one processor is configured to select the interference avoidance measures, and, if the selected interference avoidance measures are taken, to verify whether at least a portion of the model of the at least one movable object on the digital twin may interfere with the model of one or more other objects on the digital twin that are included in the real space at some point in the future on the digital twin.

4. Based on the results of the interference verification, the at least one processor modifies the action plan for the at least one movable object included in the real space in response to the interference avoidance measures. The information processing apparatus according to claim 3, configured to perform control over at least one movable object included in the physical space based on the modified action plan.

5. The information processing apparatus according to claim 1, wherein the at least one processor is configured to issue an instruction to the at least one movable object in the real space to temporarily stop its operation if it determines that the interference may occur up to a predetermined time.

6. The aforementioned movable object is Stationary robot with movable parts, A mobile robot with movable parts, Autonomous mobile robots, unmanned aircraft, Automated transport vehicles The information processing apparatus according to claim 1, comprising at least one human.

7. Multiple movable objects, The first information processing device and A second information processing device, Equipped with, The first information processing device acquires information on the at least one movable object, performs a simulation on the digital twin on a model of the at least one movable object reproduced on a digital twin that maps the real space where at least the plurality of movable objects are located onto a virtual space, verifies whether at least a part of the model of the at least one movable object may interfere with a model on the digital twin corresponding to at least one other object included in the real space by a predetermined future time after the time the information was acquired, determines measures to avoid the interference with the model of the at least one movable object by the future time, and transmits a signal to the first information processing device instructing the measures to avoid the interference. The second information processing device is an interference control system that, in response to the signal from the first information processing device, instructs the at least one movable object to take measures to avoid the interference.

8. The first information processing device is As the aforementioned information, the position and / or motion information of the at least one movable object is obtained. The interference control system according to claim 7, configured to simulate the position and / or movement of the model of the at least one movable object up to the future time point based on a pre-set action plan for the at least one movable object to construct an interference region, and to determine whether there is a possibility of interference by checking whether the interference region overlaps with or is close to at least a part of the model of the at least one other object.

9. Obtain information on at least one movable object in real space, A simulation is performed on the digital twin, which is a digital twin that maps the real space to a virtual space, to verify whether at least a part of the model of the at least one movable object may interfere with the model on the digital twin that corresponds to at least one other object included in the real space, by a predetermined future time point prior to the time when the information was acquired. If the interference occurs with respect to the model of at least one of the movable objects by the future time, measures to avoid the interference will be determined. An interference control method for instructing at least one movable object in the real space to take measures to avoid the interference.

10. Obtain information on at least one movable object in real space, A simulation is performed on the digital twin, which is a digital twin that maps the real space to a virtual space, to verify whether at least a part of the model of the at least one movable object may interfere with the model on the digital twin that corresponds to at least one other object included in the real space, by a predetermined future time point prior to the time when the information was acquired. If the interference occurs with respect to the model of at least one of the movable objects by the future time, measures to avoid the interference will be determined. The system instructs the at least one movable object in the real space to take measures to avoid the interference. A program that instructs the processor to perform a task.

Images