A robot system based on digital twinning

Abstract

The application provides a robot system based on digital twinning, and belongs to the technical field of robot control. The robot system comprises a service gateway, a building information modeling (BIM) model, a BIM engine module, a service request distribution module, a micro-service pool, a service processing module and at least one robot agent instance. The robot system can realize digital twinning of a regional inspection robot in a BIM in an indoor or outdoor scene, so as to realize data interaction between the inspection robot and equipment, data exchange between robots, and visualization of the position and state of the robot in a virtual space. The application can be used in places harmful to human bodies such as equipment rooms, nuclear power stations, garbage stations, transformer substations and infectious disease hospitals, and provides digital twinning for robot equipment inspection, so as to expand the application scene of the robot and protect the health of the inspection personnel.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a robot system based on digital twins. Background Technology

[0002] Traditional equipment inspection robots cannot present inspection results, operational status, and location information in a 3D visualization. This results in central data center managers being unaware of the robot's operational status and location, requiring them to conduct on-site inspections. Furthermore, these robots cannot read BIM data, interact with equipment, or be remotely controlled. All of these limitations restrict the robot's application scenarios. For example, when inspecting robots in harsh environments such as those with radiation, noise, high temperature, high humidity, or in toxic or dusty conditions, manual on-site verification of their operational status, such as location, battery level, and angle, is often necessary. This seriously impacts the health of inspection personnel.

[0003] For example, Chinese patent application CN 111438677 A discloses an unmanned data center inspection robot that can automatically inspect and monitor equipment in the data center at preset inspection times and points, and can remotely control the robot. Chinese patent application CN113472079 A provides a power distribution station operation and maintenance monitoring cloud robot system, background processing, and task assignment methods. The operation and maintenance monitoring system, together with the cloud robot terminal and cloud resource library, constitutes the power distribution station cloud robot system architecture, integrating cloud computing and intelligent robot technologies with power distribution station operation tasks.

[0004] Among them, Chinese patent application CN 111438677 A lacks the concept of a cloud-based robot. The actual robot can only read limited device data using a camera and cannot interact with the device to read its spatial data, thus limiting its application scenarios. CN 113472079 A requires operation tickets in Word format to be input into the system, and their content must be translated into a language that the robot can recognize and execute after semantic parsing. This also limits its application scenarios. Furthermore, the robots in these two patent applications cannot access BIM data, nor can they display the actual robot's location information and inspection results in virtual space, thus limiting their application scenarios. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a robot system based on digital twins, which provides digital twins for robot equipment inspection, expands the application scenarios of robots, and protects the health of inspection personnel.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solutions:

[0007] A digital twin-based robot system includes: a service gateway, a Building Information Modeling (BIM) model, a BIM engine module, and at least one robot agent instance, wherein:

[0008] The BIM model is used to construct a virtual space corresponding to the physical space;

[0009] The first robot agent instance is used to receive the self-state information of the corresponding first inspection robot, convert the self-state information into the first virtual state information in the virtual space, and map the first virtual state information to the first digital twin robot in the virtual space to construct the digital twin robot of the first inspection robot in the virtual space.

[0010] The BIM engine module is used to perform visualization processing on the first digital twin robot in the virtual space and generate first image data;

[0011] The service gateway is used to send the first image data to the first terminal so as to display the first digital twin robot on the first terminal side.

[0012] Optionally, the robot system further includes: a service request allocation module, a microservice pool, and a service processing module, wherein:

[0013] The service gateway is also configured to receive a first service request sent by the second terminal and store the first service request in a service request message queue.

[0014] The service request allocation module is used to read the first service request in the service request message queue, allocate and start the second robot agent instance to process the first service request, and generate the first subscription task of the first service request. The first subscription task is used to subscribe to the first field status information published by the second inspection robot corresponding to the second robot agent instance.

[0015] The second robot agent instance is used to send the first service request to the service processing module, receive the list of executable first microservices returned by the service processing module, and send the list of first microservices to the second inspection robot corresponding to the second robot agent instance, so that the second inspection robot executes the microservices in the list of first microservices;

[0016] The service processing module is used to analyze the first service request and convert it into a list of executable first microservices by calling microservices in the microservice pool.

[0017] Optionally, the second robot agent instance is further configured to receive the first field status information of the first inspected device published by the second inspection robot, convert the first field spatial coordinates of the first inspected device into the first virtual spatial coordinates of the first digital twin device in the virtual space, and map the first field status information to the first digital twin device.

[0018] The BIM engine module is also used to perform visualization processing on the first digital twin device based on the first site status information and the first virtual space coordinates, and generate second image data.

[0019] The service gateway is also used to send the second image data to the second terminal to display the first digital twin device on the second terminal side.

[0020] Optionally, the service gateway is further configured to receive a second service request sent by a third terminal and store the second service request in a service request message queue;

[0021] The service request allocation module is also used to read the second service request in the service request message queue, allocate and start multiple robot agent instances to process the second service request, and generate a second subscription task for the second service request. The second subscription task is used to subscribe to the second field status information published by multiple inspection robots corresponding to the multiple robot agent instances.

[0022] Each of the plurality of robot agent instances is configured to send the second service request to the service processing module, receive the list of executable second microservices returned by the service processing module, and send the list of second microservices to the inspection robot corresponding to the robot agent instance, so that the inspection robot executes the microservices in the list of second microservices.

[0023] The service processing module is further configured to analyze the second service request and convert it into a list of executable second microservices by calling microservices in the microservice pool.

[0024] Optionally, each of the plurality of robot agent instances is further configured to receive the second field status information of the second inspected device published by the corresponding inspection robot, convert the second field spatial coordinates of the second inspected device into the second virtual space coordinates of the second digital twin device in the virtual space, and map the second field status information to the second digital twin device.

[0025] The BIM engine module is also used to perform visualization processing on the second digital twin device based on the second site status information and the second virtual space coordinates, and generate third image data;

[0026] The service gateway is also used to send the third image data to a third terminal to display the second digital twin device on the third terminal side.

[0027] Optionally, the robot system further includes a speech recognition server and a semantic management module; wherein,

[0028] The service gateway is also used to receive a first voice command sent by a fourth terminal;

[0029] The speech recognition server is used to recognize the first speech command and convert it into the first text content;

[0030] The semantic management module is used to parse the first text content and obtain the keywords therein;

[0031] The third robot agent instance is used to call the corresponding microservice according to the preset instruction matched by the keyword, and send the identifiable instructions and data corresponding to the microservice to the corresponding third inspection robot to complete the relevant work.

[0032] Optionally, the robot system further includes:

[0033] The fourth robot agent instance is used to receive the third service request sent by the corresponding fourth inspection robot, the third service request being used to obtain relevant information about the fifth inspection robot; and to send the third service request to the fifth robot agent instance corresponding to the fifth inspection robot.

[0034] The fifth robot agent instance is used to request relevant information from the fifth inspection robot according to the third service request, receive the relevant information returned by the fifth inspection robot, and forward it to the fourth robot agent instance;

[0035] The fourth robot agent instance is also used to send the relevant information to the fourth inspection robot.

[0036] Optionally, the inspection robot's own status information includes at least one of the following: inspection robot number, moving speed, on-site empty coordinates, battery level, angle after reaching the point, inspection point information, inspection route, current timestamp, map ID, whether the inspection robot is online, inspection robot status, inspection robot name, task type, work completion rate, current task stage, estimated completion time, and working mode.

[0037] Optionally, when the inspection robot moves from the first inspection point to the second inspection point in a straight line and the time taken is the first time, the moving speed of the digital twin robot of the inspection robot in the virtual space is calculated based on the first virtual distance and the first time. The first virtual distance is the Euclidean distance between the first virtual inspection point and the second virtual inspection point, and the first virtual inspection point and the second virtual inspection point are respectively the coordinate points of the first inspection point and the second inspection point in the virtual space.

[0038] Optionally, the movement speed of the inspection robot in the virtual space, as represented by its digital twin, is calculated using the following formula:

[0039] Speed_twin__robot =

[0040] in, )and ) are the coordinates of the first virtual inspection point and the second virtual inspection point in the virtual space, respectively, and time represents the first time.

[0041] Optionally, the angle of the digital twin robot in the virtual space is the angle of the inspection robot corresponding to the digital twin robot in the physical space.

[0042] Optionally, the inspection robot and its corresponding robot agent instance can interact with each other via asynchronous message queue telemetry transmission MQTT protocol, real-time HTTP protocol, or TCP protocol.

[0043] The embodiments of the present invention have the following beneficial effects:

[0044] The robot system of this invention can realize the digital twin of an area inspection robot in indoor or outdoor scenarios within BIM, thereby enabling data interaction between the inspection robot and equipment, data exchange between robots, and visualization of the robot's position and status in virtual space. This invention can be used in equipment rooms, nuclear power plants, waste stations, substations, infectious disease hospitals, and other places where human health is hazardous, providing digital twins for robot equipment inspection, thus expanding the application scenarios of robots and protecting the health of inspection personnel. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of a robot system based on digital twins according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram illustrating data interaction between the robot and the cloud robot system in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the registration and publishing process of cloud services in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the microservice invocation process in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram illustrating the data interaction between the inspection robot and the robot agent instance in an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram of a flowchart of a microservice call in an embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram illustrating information interaction between robots in an embodiment of the present invention. Detailed Implementation

[0052] To make the technical problems, technical solutions and advantages of the embodiments of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0053] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein.

[0054] This invention provides a digital twin inspection robot system based on Building Information Modeling (BIM). This system runs in the cloud, acting as a bridge connecting cloud resources with real inspection robots. It achieves a digital twin effect of the real inspection robot in virtual space, allowing inspection personnel to understand the status of the real robot without going to the site. This enables automated robot inspections and visualization of inspection results in the digital twin space, thus protecting the health of inspection personnel. For example, with the help of the virtual digital twin robot, the real-world inspection robot can read real-time equipment data from the BIM, thereby judging the operating status of the equipment and achieving real-time interaction between the real-world robot and the equipment. Furthermore, with the help of the virtual digital twin robot, real-world robots can also exchange information, enabling multi-robot collaboration. Finally, with the help of the virtual digital twin robot, real-world inspection robots can upload inspection results to the cloud and utilize cloud computing power to analyze the results.

[0055] Please refer to Figure 1This invention provides a robot system based on digital twins, comprising: a service gateway, a BIM model, a BIM engine module, and at least one robot agent instance, each robot agent instance corresponding to a physical inspection robot. The at least one robot agent instance may include a first robot agent instance. Wherein:

[0056] The BIM model is used to construct a virtual space corresponding to the physical space. The physical space can specifically be an indoor space or an outdoor space inspected by the inspection robot.

[0057] The first robot agent instance is used to receive its own status information of the corresponding first inspection robot, convert the own status information into first virtual status information in the virtual space, and map the first virtual status information to the first digital twin robot in the virtual space to construct the digital twin robot of the first inspection robot in the virtual space.

[0058] The BIM engine module is used to perform visualization processing on the first digital twin robot in the virtual space and generate first image data;

[0059] The service gateway is used to send the first image data to the first terminal so as to display the first digital twin robot on the first terminal side.

[0060] Through the processing of the above modules, the robot system of this embodiment of the invention realizes the visualization of the robot's position and status in virtual space. By providing a digital twin for robot equipment inspection, this embodiment of the invention expands the application scenarios of the robot and can protect the health of inspection personnel.

[0061] Here, the inspection robot's own status information may include at least one of the following: inspection robot number, moving speed, on-site empty coordinates, battery level, angle after reaching the point, inspection point information, inspection route, current timestamp, map ID, whether the inspection robot is online, inspection robot status, inspection robot name, task type, work completion rate, current task stage, estimated completion time, working mode, etc.

[0062] When the inspection robot moves from the first inspection point to the second inspection point in a straight line and the time taken is the first time, the movement speed of the digital twin robot of the inspection robot in the virtual space is calculated based on the first virtual distance and the first time. The first virtual distance is the Euclidean distance between the first virtual inspection point and the second virtual inspection point. The first virtual inspection point and the second virtual inspection point are the coordinate points of the first inspection point and the second inspection point in the virtual space, respectively.

[0063] Specifically, the movement speed of the inspection robot in the virtual space, as represented by its digital twin, is calculated using the following formula:

[0064] Speed_twin__robot =

[0065] in, )and ) are the coordinates of the first virtual inspection point and the second virtual inspection point in the virtual space, respectively, and time represents the first time.

[0066] The angle of the digital twin robot in the virtual space is the angle of the inspection robot corresponding to the digital twin robot in the physical space.

[0067] In addition, the inspection robot and its corresponding robot agent instance interact with each other through asynchronous Message Queuing Telemetry Transport (MQTT) protocol, real-time HTTP protocol, or TCP protocol.

[0068] In this embodiment of the invention, Figure 1 The robot system shown also includes: a service request allocation module, a microservice pool, and a service processing module. The at least one robot agent instance may further include a second robot agent instance, wherein:

[0069] The service gateway is also used to receive a first service request (single service request) sent by the second terminal and store the first service request in a service request message queue.

[0070] The service request allocation module is used to read the first service request in the service request message queue, allocate and start the second robot agent instance to process the first service request, and generate the first subscription task of the first service request. The first subscription task is used to subscribe to the first field status information published by the second inspection robot corresponding to the second robot agent instance.

[0071] The second robot agent instance is used to send the first service request to the service processing module, receive the list of executable first microservices returned by the service processing module, and send the list of first microservices to the second inspection robot corresponding to the second robot agent instance, so that the second inspection robot executes the microservices in the list of first microservices.

[0072] The service processing module is used to analyze the first service request and convert it into a list of executable first microservices by calling microservices in the microservice pool.

[0073] In this embodiment of the invention, the second robot agent instance is further configured to receive the first on-site status information of the first inspected device published by the second inspection robot, convert the first on-site spatial coordinates of the first inspected device into the first virtual spatial coordinates of the first digital twin device in the virtual space, and map the first on-site status information to the first digital twin device.

[0074] The BIM engine module is also used to perform visualization processing on the first digital twin device based on the first site status information and the first virtual space coordinates, and generate second image data.

[0075] The service gateway is also used to send the second image data to the second terminal to display the first digital twin device on the second terminal side.

[0076] Through the cooperation of the above modules, the robot system of this embodiment can process single service requests and control a single inspection robot to perform related services. In addition, it can also realize data interaction between the inspection robot and the equipment.

[0077] In this embodiment of the invention, the service gateway is further configured to receive a second service request sent by a third terminal and store the second service request in a service request message queue;

[0078] The service request allocation module is further configured to read the second service request in the service request message queue, allocate and start multiple robot agent instances to process the second service request, and generate a second subscription task for the second service request. The second subscription task is used to subscribe to the second field status information published by multiple inspection robots corresponding to the multiple robot agent instances.

[0079] Each of the plurality of robot agent instances is configured to send the second service request to the service processing module, receive the list of executable second microservices returned by the service processing module, and send the list of second microservices to the inspection robot corresponding to the robot agent instance, so that the inspection robot executes the microservices in the list of second microservices.

[0080] The service processing module is further configured to analyze the second service request and convert it into a list of executable second microservices by calling microservices in the microservice pool.

[0081] Here, each of the plurality of robot agent instances is further configured to receive the second field status information of the second inspected device published by the corresponding inspection robot, convert the second field spatial coordinates of the second inspected device into the second virtual space coordinates of the second digital twin device in the virtual space, and map the second field status information to the second digital twin device;

[0082] The BIM engine module is also used to perform visualization processing on the second digital twin device based on the second site status information and the second virtual space coordinates, and generate third image data;

[0083] The service gateway is also used to send the third image data to a third terminal to display the second digital twin device on the third terminal side.

[0084] Through the cooperation of the above modules, the robot system of this embodiment can process multiple service requests and control multiple inspection robots to perform related services. In addition, it can also realize data interaction between the inspection robots and the equipment. Of course, multiple service requests can also be implemented by a single inspection robot by performing multiple services.

[0085] In this embodiment of the invention, the at least one robot agent instance may further include a third robot agent instance, and the robot system may further include a speech recognition server and a semantic management module; wherein,

[0086] The service gateway is also used to receive a first voice command sent by a fourth terminal.

[0087] The speech recognition server is used to recognize the first speech command and convert it into the first text content;

[0088] The semantic management module is used to parse the first text content and obtain the keywords therein.

[0089] The third robot agent instance is used to call the corresponding microservice according to the preset instruction matched by the keyword, and send the identifiable instructions and data corresponding to the microservice to the corresponding third inspection robot to complete the relevant work.

[0090] With the cooperation of the above modules, this embodiment of the invention enables the terminal to control the inspection robot to perform related tasks via voice commands.

[0091] In this embodiment of the invention, the at least one robot agent instance may further include a fourth robot agent instance and a fifth robot instance. Through the interaction between the robot agents, this embodiment of the invention can realize information exchange between inspection robots.

[0092] Specifically, the fourth robot agent instance is used to receive a third service request sent by the corresponding fourth inspection robot, the third service request being used to obtain relevant information about the fifth inspection robot; and to send the third service request to the fifth robot agent instance corresponding to the fifth inspection robot.

[0093] The fifth robot agent instance is used to request relevant information from the fifth inspection robot according to the third service request, receive the relevant information returned by the fifth inspection robot, and forward it to the fourth robot agent instance;

[0094] The fourth robot agent instance is also used to send the relevant information to the fourth inspection robot.

[0095] The following section, in conjunction with the accompanying drawings, further details... Figure 1 The overall process of the digital twin-based virtual inspection robot system will be explained in more detail below.

[0096] A) Interaction between inspection robots and cloud-based robot systems

[0097] After the inspection robot establishes a long connection with the robot agent instance in the cloud, all requests and data transmissions from the inspection robot are uploaded to the robot system in the cloud through the robot agent instance. Figure 1 Step 13 describes the process of data interaction between the robot and its corresponding robot agent instance.

[0098] Figure 2 It provides a process for inspection robots to interact with a cloud-based robot system. Specifically, the inspection robot interacts with its cloud-based robot agent instance to obtain information about the equipment in the virtual space and maps its real-time status to a digital twin robot in the virtual space.

[0099] B) Service Registration and Posting

[0100] Figure 3 This demonstrates the registration and publishing process for cloud services. Users (terminals), inspection robots, and microservices all need to register in advance with the service registration portal in the cloud. When a service requester wants to obtain a target service, they search for that target service in the service registration portal. The service registration portal determines the service provider of the target service and publishes the required target service to that service provider (robot agent instance), thereby enabling the service provider (robot agent instance) and its corresponding inspection robot to execute the corresponding target service. The services that the service provider (robot agent instance) and its corresponding inspection robot can provide include inspection services, voice services, navigation services, and other services.

[0101] The cloud stores the actions, maps, and navigation resources needed for the inspection robot to complete its tasks, and continuously updates these resources as the robot performs its tasks. Information exchange between the inspection robot and the cloud is accomplished through robot agent instances. Furthermore, the inspection robot can offload a significant amount of computation to the cloud. This interaction is also achieved through robot agent instances in the cloud.

[0102] C) Microservice call process

[0103] Figure 4 This invention illustrates a flowchart of a microservice invocation in an embodiment of the present invention. Figure 1 The process of issuing instructions to the inspection robot via the terminal is broken down into steps.

[0104] The process involves either the request allocation logic (service request allocation module) or the inspection robot compiling request information to submit a service request to a robot agent instance. The robot agent instance receives the service request, decodes it, encodes at least one microservice request, and sends a microservice invocation command to the microservice pool. The microservice pool receives and parses the microservice invocation command, runs the corresponding microservice, encodes the microservice execution result, and returns the microservice result to the robot agent instance. The robot agent instance receives the microservice result, encodes the return parameters, and sends these parameters to the request allocation logic (service request allocation module) or the inspection robot. Thus, upon receiving the return parameters, the request allocation logic (service request allocation module) or the inspection robot can decode the service result.

[0105] D) Single-service requests, multi-service requests, and robot service requests

[0106] 1. Single service request:

[0107] (1) The second terminal sends a single service request, which is sent to the message queue by the service gateway, see reference. Figure 1 Step 11 in the process.

[0108] (2) Service Request Allocation Module (Request Allocation Logic): Select and start a robot agent instance to complete the subscription task. Refer to [link / reference] Figure 1 Step 12 in the process.

[0109] (3) The robot agent instance communicates with the corresponding inspection robot to complete the task, see reference. Figure 1 Step 13 in the process.

[0110] 2. Multiple service requests

[0111] (1) The third terminal sends a multi-service request, which is published to the message queue by the service gateway, see reference. Figure 1 Step 11 in the process.

[0112] (2) Service request allocation module (request allocation logic): Select and start multiple robot agent instances to complete the subscription task. Refer to Figure 1 Step 12 in the process.

[0113] (3) Multiple robot agent instances communicate with their respective inspection robots to complete tasks, as shown in the reference. Figure 1 Step 13 in the process.

[0114] 3. Robot service request

[0115] (1) The inspection robot sends a service request to its own agent instance, refer to Figure 1 Step 13 in the process.

[0116] (2) Robot agent instance calls microservice, refer to Figure 1 The robot agent instance calls the microservice.

[0117] (3) The microservice results are returned to the inspection robot, refer to Figure 1 Step 13 in the process.

[0118] Here, the robot agent instance calls the service analysis module to analyze service requests, uses the service refactoring module to convert services into a list of executable microservices, uses service description pores to build service monitoring stubs, and reports the current status of services in real time.

[0119] E) Issues with the virtual space location and status display of inspection robots

[0120] The method for coordinate transformation in virtual space for inspection robots can be referenced in Chinese patent application CN113535864A, and will not be elaborated upon here.

[0121] The process of creating a digital twin of a robot in virtual space is as follows:

[0122] The inspection robot transmits on-site speed, coordinates, battery level, angle, inspection point information, etc. to the robot agent instance. Figure 1 In step 13), the robot agent instance maps the coordinate information of the inspection robot at the inspection site to the virtual space after address transformation; it converts the speed of the inspection robot into the speed of the virtual space, and maps the battery level, turning angle, inspection points, and other information of the inspection robot to the digital twin robot, thereby constructing a digital twin robot in the virtual space. Figure 1 (Step 14 in the process). Then, managers only need to view various information about the digital twin robot on the terminal to understand the status of the real robot on site in real time.

[0123] In this embodiment of the invention, the information typically required for a digital twin robot includes: robot number, moving speed, coordinates, battery level, angle after reaching the inspection point, inspection point information, inspection route, current timestamp, map ID, whether the robot is online, and robot status. Optionally, the digital twin robot may also use information including, but not limited to: robot name, task type, work completion rate, current task stage, estimated completion time, and working mode.

[0124] Data interaction between the inspection robot and its agent instance can be achieved via the Message Queuing Telemetry Transport (MQTT) protocol, or through real-time HTTP or TCP protocols. Figure 5 The diagram illustrates the data interaction process using the MQTT communication mode.

[0125] Additionally, assuming the inspection robot moves from inspection point A, with coordinates (x1, y1), to the next inspection point B, with coordinates (x2, y2), then the speed of the inspection robot is:

[0126] Speed_robot = (1)

[0127] After coordinate transformation, inspection points A and B correspond to coordinates A' and B' in virtual space, respectively, where A' has coordinates (x'1, y'1) and B' has coordinates (x'2, y'2). The speed of the virtual digital twin robot is:

[0128] Speed_twin__robot= (2)

[0129] Wherein, time represents the time required for the inspection robot to travel from inspection point A to inspection point B. In this embodiment of the invention, the inspection robot moves in a straight line between two adjacent inspection points.

[0130] Based on the virtual robot's speed (Speed_twin__robot), real-time location, and coordinates of the next inspection point, the time it takes to reach the next inspection point in the virtual space can be estimated. This time is synchronized with the time it takes for the real inspection robot to reach the next inspection point. However, this calculation takes into account the latency that occurs when transmitting the real robot's location information to the robot agent via the MQTT protocol. This time delay is estimated based on the on-site measurement results and used to calculate the time to reach the next inspection point.

[0131] If the difference between the time it takes for the virtual robot to reach the next inspection point and the time it takes for the inspection robot to reach the next inspection point is greater than a certain preset threshold (e.g., 2 seconds), a weight can be added to the speed of the virtual robot to make the difference less than the threshold.

[0132] In addition, the angle at which the robot faces the device in the virtual space is based on the angle after reaching the point, and the virtual robot can adjust its posture according to this angle.

[0133] F) Problems with robot reading equipment's spatial information

[0134] Inspection robots need to check the temperature of equipment (such as pipelines and motors) on-site and transmit the results to BIM for visualization. Inspection robots cannot directly interact with the actual equipment being inspected, but they can obtain relevant data (such as spatial data, equipment information, error records, etc.) through a digital twin robot in virtual space. This requires converting the spatial coordinates of the inspected equipment into the spatial coordinates of the digital twin device in BIM. The inspection robot transmits the spatial coordinates of the inspected equipment (pipeline) and the detection data (such as temperature values) to the robot system. The spatial coordinates of the inspected equipment are converted into the coordinates of the digital twin device in BIM, and the detection data (such as pipeline temperature) detected by the inspection robot is visualized in the BIM equipment. Figure 1 Steps 13, 14, and 15 represent the process of transmitting the inspection robot's detection data to the robot agent instance, locating the inspected equipment, performing coordinate transformation, and visualizing the digital twin robot.

[0135] G) The robot reads the operation ticket instruction information.

[0136] The client can wear a voice device to send the administrator's voice commands to the cloud-based voice server for recognition. The recognized text is then sent to the robot's semantic management module in the cloud. When a string matches a predefined grammatical rule, the rule engine breaks down the string into keywords and sends these keywords to the robot agent. Based on the keywords, the robot agent initiates preset instructions, calls relevant microservices, and finally transmits the robot-recognizable instructions and related data to complete the corresponding task. Figure 6 A flowchart of a microservice invocation process described above is provided in an embodiment of the present invention.

[0137] H) Robot-to-robot interaction

[0138] Robot-to-robot interaction can be achieved through cloud-based robot agent instances acting as intermediaries. (See reference...) Figure 7 Taking communication between robots as an example, if inspection robot 1 wants to reach the location of inspection robot 2, it can be achieved through the following process:

[0139] Step 71: Inspection robot 1 first sends a request to agent instance 1 in the cloud to request the coordinates of inspection robot 2.

[0140] Step 72: Robot agent instance 1 sends a request to robot agent instance 2.

[0141] Step 73: Robot agent instance 2 sends a request to inspection robot 2 to obtain real-time location.

[0142] Step 74: Inspection robot 2 returns location information to robot agent instance 2.

[0143] Step 75: Robot agent instance 2 returns the location information to robot agent instance 1.

[0144] Step 76: Robot agent instance 1 returns the location information of inspection robot 2 to robot 1.

[0145] As can be seen from the examples above, this invention addresses the issue of displaying the location and status of the inspection robot in virtual space. The robot system collects real-time data on the inspection robot's speed, coordinates, battery level, angle, and inspection point information at the inspection site. The robot's coordinates at the inspection site are mapped to the virtual space after address transformation; the robot's speed is converted to its virtual space speed; and information such as battery level, turning angle, and inspection points are mapped to the digital twin robot, thus constructing a digital twin robot in the virtual space. Furthermore, managers can monitor the real-time status of the actual robot by simply viewing the various information of the digital twin robot on a terminal.

[0146] Regarding the issue of robots reading the spatial information of inspected equipment: The inspection robot obtains inspection data from the inspected equipment on-site and transmits this data to BIM for visualization. This requires converting the spatial coordinates of the inspected equipment into the spatial coordinates of the digital twin device within the BIM. The inspection robot transmits the spatial coordinates of the inspected equipment and the inspection data to the robot system. The robot system converts the spatial coordinates of the inspected equipment into the coordinates of the digital twin device within the BIM and visualizes the inspection data detected by the inspection robot within the digital twin device.

[0147] For robots to read operation ticket instructions: human voice can be uploaded to the robot system via a voice device. The robot system will then convert the voice into commands that the robot can recognize, enabling the robot to complete the relevant tasks.

[0148] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles described in the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A robot system based on digital twins, characterized in that, include: The service gateway, the Building Information Modeling (BIM) model, the BIM engine module, and at least one robot agent instance, wherein: The BIM model is used to construct a virtual space corresponding to the physical space; The first robot agent instance is used to receive the self-state information of the corresponding first inspection robot, convert the self-state information into the first virtual state information in the virtual space, and map the first virtual state information to the first digital twin robot in the virtual space to construct the digital twin robot of the first inspection robot in the virtual space. The BIM engine module is used to perform visualization processing on the first digital twin robot in the virtual space and generate first image data; The service gateway is used to send the first image data to the first terminal to display the first digital twin robot on the first terminal side; The robot system is characterized by further comprising: a service request allocation module, a microservice pool, and a service processing module, wherein: The service gateway is also used to receive a first service request sent by the second terminal and store the first service request in a service request message queue. The service request allocation module is used to read the first service request in the service request message queue, allocate and start the second robot agent instance to process the first service request, and generate the first subscription task of the first service request. The first subscription task is used to subscribe to the first field status information published by the second inspection robot corresponding to the second robot agent instance. The second robot agent instance is used to send the first service request to the service processing module, receive the list of executable first microservices returned by the service processing module, and send the list of first microservices to the second inspection robot corresponding to the second robot agent instance, so that the second inspection robot executes the microservices in the list of first microservices; The service processing module is used to analyze the first service request and convert it into a list of executable first microservices by calling microservices in the microservice pool.

2. The robot system according to claim 1, characterized in that, The second robot agent instance is further configured to receive the first field status information of the first inspected device published by the second inspection robot, convert the first field spatial coordinates of the first inspected device into the first virtual spatial coordinates of the first digital twin device in the virtual space, and map the first field status information to the first digital twin device. The BIM engine module is also used to perform visualization processing on the first digital twin device based on the first site status information and the first virtual space coordinates, and generate second image data. The service gateway is also used to send the second image data to the second terminal to display the first digital twin device on the second terminal side.

3. The robot system according to claim 1, characterized in that, The service gateway is also used to receive a second service request sent by a third terminal and store the second service request in a service request message queue; The service request allocation module is also used to read the second service request in the service request message queue, allocate and start multiple robot agent instances to process the second service request, and generate a second subscription task for the second service request. The second subscription task is used to subscribe to the second field status information published by multiple inspection robots corresponding to the multiple robot agent instances. Each of the plurality of robot agent instances is configured to send the second service request to the service processing module, receive the list of executable second microservices returned by the service processing module, and send the list of second microservices to the inspection robot corresponding to the robot agent instance, so that the inspection robot executes the microservices in the list of second microservices. The service processing module is further configured to analyze the second service request and convert it into a list of executable second microservices by calling microservices in the microservice pool.

4. The robot system according to claim 3, characterized in that, Each of the plurality of robot agent instances is further configured to receive the second field status information of the second inspected device published by the corresponding inspection robot, convert the second field spatial coordinates of the second inspected device into the second virtual space coordinates of the second digital twin device in the virtual space, and map the second field status information to the second digital twin device; The BIM engine module is also used to perform visualization processing on the second digital twin device based on the second site status information and the second virtual space coordinates, and generate third image data; The service gateway is also used to send the third image data to a third terminal to display the second digital twin device on the third terminal side.

5. The robot system according to claim 1, characterized in that, It also includes a speech recognition server and a semantic management module; among which, The service gateway is also used to receive a first voice command sent by a fourth terminal; The speech recognition server is used to recognize the first speech command and convert it into the first text content; The semantic management module is used to parse the first text content and obtain the keywords therein; The third robot agent instance is used to call the corresponding microservice according to the preset instruction matched by the keyword, and send the identifiable instructions and data corresponding to the microservice to the corresponding third inspection robot to complete the relevant work.

6. The robot system according to claim 1, characterized in that, Also includes: The fourth robot agent instance is used to receive the third service request sent by the corresponding fourth inspection robot, and the third service request is used to obtain relevant information about the fifth inspection robot. The third service request is sent to the fifth robot agent instance corresponding to the fifth inspection robot; The fifth robot agent instance is used to request relevant information from the fifth inspection robot according to the third service request, receive the relevant information returned by the fifth inspection robot, and forward it to the fourth robot agent instance; The fourth robot agent instance is also used to send the relevant information to the fourth inspection robot.

7. The robot system according to claim 1, characterized in that, The inspection robot's own status information includes at least one of the following: inspection robot number, moving speed, on-site empty coordinates, battery level, angle after reaching the point, inspection point information, inspection route, current timestamp, map ID, whether the inspection robot is online, inspection robot status, inspection robot name, task type, work completion rate, current task stage, estimated completion time, and working mode.

8. The robot system according to claim 1, characterized in that, When the inspection robot moves from the first inspection point to the second inspection point in a straight line and the time taken is the first time, the movement speed of the digital twin robot of the inspection robot in the virtual space is calculated based on the first virtual distance and the first time. The first virtual distance is the Euclidean distance between the first virtual inspection point and the second virtual inspection point. The first virtual inspection point and the second virtual inspection point are the coordinate points of the first inspection point and the second inspection point in the virtual space, respectively.

9. The robot system according to claim 8, characterized in that, The movement speed of the inspection robot in the virtual space, as represented by its digital twin, is calculated using the following formula: Speed_twin__robot = in, )and ) are the coordinates of the first virtual inspection point and the second virtual inspection point in the virtual space, respectively, and time represents the first time.

10. The robot system according to claim 1, characterized in that, The angle of the digital twin robot in the virtual space is the angle of the inspection robot corresponding to the digital twin robot in the physical space.

11. The robot system according to claim 1, characterized in that, The inspection robot and its corresponding robot agent instance exchange data through asynchronous message queue telemetry transmission MQTT protocol, real-time HTTP protocol or TCP protocol.

Images