Robotic workcell real-time digital twin method, apparatus, device, and medium

Abstract

This application provides a method, apparatus, device, and medium for real-time digital twin creation of a robot's working scene, relating to the field of digital twin technology. The method includes: acquiring a real-time video stream of the robot's working scene; segmenting the real-time color image stream to obtain two-dimensional boundary information and instance masks for all objects; obtaining a three-dimensional Gaussian model of the target object based on its two-dimensional boundary information; estimating the pose of the target object based on the instance mask, its three-dimensional Gaussian model, the real-time color image stream, and the real-time depth image stream to obtain a real-time six-DOF pose stream for the target object; and rendering the three-dimensional Gaussian model of the target object in real-time within the corresponding digital twin scene of the robot's working scene based on the real-time six-DOF pose stream. This application can resolve the contradiction between high-fidelity and real-time requirements in digital twin scenes, ensuring synchronization accuracy and visual consistency with the real scene.

Description

Technical Field

[0001] This invention relates to the field of digital twin technology, and more specifically, to a method, apparatus, device, and medium for real-time digital twin generation of a robot working scene. Background Technology

[0002] With the development of intelligent industrial technology and intelligent manufacturing technology, digital twin technology is playing an increasingly important role in warehousing logistics and automated production lines.

[0003] Currently, most digital twin scenarios utilize traditional mesh models or Neural Radiance Fields (NeRF) technology for rendering 3D object models. However, while traditional mesh models offer high accuracy, their complex texture mapping and difficulty handling transparent and reflective objects result in a heavy rendering load. Furthermore, the need for manual modeling or photogrammetric reconstruction in traditional mesh models makes it difficult to guarantee real-time performance. While the emerging Neural Radiance Fields (NeRF) approach offers superior visual fidelity, its implicit representation leads to long training times and slow inference speeds, failing to meet real-time requirements or achieve efficient direct rendering on web pages. Therefore, existing solutions for digital twin scenarios present a contradiction between simultaneously meeting high fidelity and real-time performance requirements in 3D object models.

[0004] Moreover, the poses of objects in digital twin scenarios are mostly estimated based on pure vision solutions, i.e., pure visual images. This makes the object pose estimation inaccurate, resulting in insufficient synchronization accuracy between the real scene and the digital twin scene, and making it difficult to guarantee visual consistency. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a method, apparatus, device, and medium for real-time digital twin creation of robot work scenarios, thereby resolving the contradiction between the high-fidelity and real-time requirements of 3D object models in digital twin scenarios, as well as the synchronization accuracy and visual consistency issues between digital twin scenarios and real-world scenarios.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, embodiments of the present invention provide a real-time digital twin method for robot working scenarios, including: Acquire a real-time video stream of the robot's working scene, the real-time video stream including: a real-time color image stream and a real-time depth image stream; The real-time color image stream is segmented to obtain the two-dimensional boundary information of all objects in the robot's working scene, as well as the instance mask; Based on the two-dimensional boundary information of the target object among all objects, the three-dimensional Gaussian model of the target object is obtained from the preset three-dimensional object model library; wherein, the preset three-dimensional object model library stores three-dimensional Gaussian models of multiple objects in advance. Based on the instance mask, the 3D Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream, the target object's six-degree-of-freedom pose is estimated to obtain the target object's real-time six-degree-of-freedom pose stream. Based on the real-time six-degree-of-freedom pose flow of the target object, the three-dimensional Gaussian model of the target object is rendered in real time in the digital twin scene corresponding to the robot's working scene.

[0007] As one possible implementation, obtaining the three-dimensional Gaussian model of the target object from a preset three-dimensional object model library based on the two-dimensional boundary information of the target object among all objects includes: Based on the two-dimensional boundary information of the target object, obtain the physical size information of the target object; Based on the physical size information of the target object, a preset object mapping table is used to determine the identifier of the target object and the corresponding model file path information; the preset object mapping table stores the identifiers, physical size information and corresponding model file path information of multiple objects; Based on the model file path information, obtain the three-dimensional Gaussian model of the target object from the preset three-dimensional object model library.

[0008] As one possible implementation, the method further includes: Acquire multi-view color images and corresponding depth images of the object; Based on the multi-view color image of the object and the corresponding depth image, obtain the sparse point cloud of the object; Based on the sparse point cloud of the object, a preset three-dimensional Gaussian sputtering algorithm is used to generate a three-dimensional Gaussian model of the object. The three-dimensional Gaussian model of the object is stored in the preset three-dimensional object model library to obtain the model file path information of the object; Based on the file path information of the object, the identifier of the object, and the corresponding physical size information, a preset object mapping relationship table is established.

[0009] As one possible implementation, the step of performing six-DOF pose estimation on the target object based on the instance mask, the three-dimensional Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream to obtain the real-time six-DOF pose stream of the target object includes: Align the real-time depth image stream and the real-time color image stream; Based on the instance mask, local point cloud data of the target object is cropped from the corresponding frame depth image in the real-time depth image stream; Based on the instance mask, the corresponding color image region is obtained from the corresponding frame color image in the real-time color image stream; Based on the local point cloud data, the color image region, and the three-dimensional Gaussian model of the target object, the six-degree-of-freedom pose of the target object is estimated to obtain the six-degree-of-freedom pose of the target object. Based on the six-degree-of-freedom pose, the real-time six-degree-of-freedom pose flow of the target object is obtained.

[0010] As one possible implementation, the method further includes: Obtain the joint status data packets sent by the physical robot in the robot's working environment; Based on the joint state data packet, the kinematics tree of the robot model corresponding to the physical robot in the digital twin scenario is used to update the posture of each corresponding joint node on the robot model in real time.

[0011] As one possible implementation, the step of rendering a 3D Gaussian model of the target object in real time within a digital twin scene corresponding to the robot's working scene, based on the real-time six-DOF pose flow of the target object, includes: Based on the real-time six-degree-of-freedom pose flow of the target object, the poses of each three-dimensional Gaussian sphere in the three-dimensional Gaussian sputtering model of the target object are updated to obtain the updated three-dimensional Gaussian model. The updated 3D Gaussian sputtering model is rendered in real time.

[0012] As one possible implementation, the method further includes: Monitor the input and output signals of the end effector of the physical robot in the robot's working environment; Based on the input and output signals, the parent-child relationship of the three-dimensional Gaussian model corresponding to the target object in the digital twin scene is dynamically reorganized.

[0013] Secondly, embodiments of this application also provide a real-time digital twin device for a robot working scene, comprising: The acquisition module is used to acquire real-time video streams of the robot's working scene, the real-time video streams including: real-time color image streams and real-time depth image streams; The segmentation module is used to segment the real-time color image stream to obtain the two-dimensional boundary information of all objects in the robot's working scene, as well as the instance mask; The pose estimation module is used to obtain the three-dimensional Gaussian model of the target object from a preset three-dimensional object model library based on the two-dimensional boundary information of the target object among all objects; wherein, the preset three-dimensional object model library stores three-dimensional Gaussian models of multiple objects in advance; and to perform six-degree-of-freedom pose estimation on the target object based on the instance mask, the three-dimensional Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream to obtain the real-time six-degree-of-freedom pose stream of the target object. The rendering module is used to render the three-dimensional Gaussian model of the target object in real time in the digital twin scene corresponding to the robot's working scene, based on the real-time six-degree-of-freedom pose flow of the target object.

[0014] Thirdly, embodiments of this application also provide a computer device, a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform any of the real-time digital twin methods for robot work scenarios provided in the first aspect above.

[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs any of the real-time digital twin methods for robot work scenarios provided in the first aspect.

[0016] The beneficial effects of this application are: The real-time digital twin method, apparatus, device, and medium for robot work scenes provided in this application can obtain the real-time video stream of the robot work scene, segment the real-time color image stream in the real-time video stream to obtain the two-dimensional boundary information of all objects in the robot work scene, as well as the instance mask, thereby realizing visual perception and real-time reasoning of objects in the robot work scene; furthermore, based on the two-dimensional boundary information of the target object among all objects, the three-dimensional Gaussian model of the target object is obtained from a preset object three-dimensional model library, thereby realizing the localization of the three-dimensional Gaussian model of the target object; and based on the instance mask, the three-dimensional Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream, the six-degree-of-freedom pose estimation of the target object is performed, thereby realizing the six-degree-of-freedom pose estimation of the target object; and the three-dimensional Gaussian model of the target object can also be rendered in real-time in the corresponding digital twin scene of the robot work scene based on the real-time six-degree-of-freedom pose stream of the target object. The robot working scene provided in this application uses a 3D Gaussian model as the target object model in the corresponding digital twin scene. This avoids the excessive rendering load caused by traditional mesh model rendering and the slow inference speed and poor real-time performance issues associated with neural radiation field technology for model rendering. The 3D Gaussian model is directly rasterized and rendered in real-time on the web page, effectively ensuring efficient rendering on the web page and guaranteeing high-definition image quality and high frame rate in the digital twin scene. This effectively resolves the contradiction between high fidelity and real-time performance of the 3D object model in the digital twin scene. Furthermore, in the process of estimating the six-degree-of-freedom pose of the target object, not only are instance masks of real-time color images used, but also the 3D Gaussian model of the target object, as well as real-time color image streams and real-time depth image streams. Therefore, it is not a purely visual solution. It effectively avoids the limitations of object pose tracking caused by purely visual image solutions, improves the accuracy of object pose, and thus improves the synchronization accuracy between the object pose in the real scene and the corresponding 3D object model in the digital twin scene, ensuring visual consistency between the real scene and the digital twin scene. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This application provides an architectural diagram illustrating an application scenario. Figure 2 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 1 ; Figure 3 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 2 ; Figure 4 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 3 ; Figure 5 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 4 ; Figure 6 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 5 ; Figure 7 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 6 ; Figure 8 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 7 ; Figure 9 A schematic diagram of a real-time digital twin device for a robot working scene provided in an embodiment of this application; Figure 10 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are within the scope of protection of the present application.

[0020] To facilitate a better understanding of the real-time digital twin method for robot work scenarios provided in the embodiments of this application, the technical terms involved in the following embodiments of this application will be explained.

[0021] 3D Gaussian Splatting (3D-GS): An explicit scene representation technique for real-time radiation field rendering. Unlike traditional mesh modeling or neural radiation fields, it represents the scene as a set of 3D Gaussian ellipsoids with a large number of discrete 3D Gaussian spheres. This technique, combined with a rasterization pipeline, enables extremely high real-time rendering frame rates while maintaining high-fidelity visual effects, making it particularly suitable for industrial scenes requiring rapid reconstruction and dynamic rendering.

[0022] Six-degree-of-freedom pose (6D Pose): Also known as 6D orientation, it refers to the absolute position and orientation of an object in three-dimensional space, encompassing six degrees of freedom. Specifically, it includes three translational degrees of freedom (displacement along the X, Y, and Z axes) and three rotational degrees of freedom (rotation angles around the X, Y, and Z axes, typically represented by Euler angles, quaternions, or rotation matrices). In this application, 6D pose is used to describe the spatial state of an object relative to the world coordinate system or the robot's base coordinate system.

[0023] The Unified Robot Description Format (URDF) is an Extensible Markup Language (XML) format used in Robot Operating System (ROS) to describe robot models, including their physical structure and kinematic characteristics. It defines the robot's links, joints, and their parent-child hierarchical relationships, and also includes the robot's geometry, collision model, and inertial parameters. This application utilizes URDF files to reconstruct the robot's kinematic tree to achieve synchronized actuation with the physical robot.

[0024] Perspective-n-Point (PnP) algorithm: This is a method for estimating camera or object pose based on the correspondence between 3D spatial points and their projection points on a 2D image. In this application, the precise 6D pose of the object relative to the camera is obtained by using known key points on the object's 3D model and visually recognized 2D image feature points.

[0025] Structure from Motion (SfM) is a technique for automatically recovering the 3D structure (sparse point cloud) and camera pose of a scene from a sequence of 2D images containing visual motion information. In this application, the SfM algorithm can be used to provide an initial sparse point cloud distribution for training a 3D Gaussian model during the object loading stage.

[0026] Instance segmentation is a task in computer vision that combines the features of object detection and semantic segmentation. Unlike ordinary object detection, which only outputs bounding boxes, instance segmentation can identify the pixel-level mask of each individual object in an image, distinguishing even objects of the same type. This application utilizes the YOLO series of algorithms for instance segmentation to accurately remove background interference and obtain clean point clouds of the target objects.

[0027] The real-time digital twin method for robot work scenarios in one embodiment of this disclosure can be applied to the field of digital twins in industrial scenarios, such as digital twins of robot work scenarios. Digital human work scenarios can be, for example, warehousing and logistics scenarios or automated production line scenarios. It should also be noted that the real-time digital twin method for robot work scenarios provided in this application can run on a local terminal device or a server. When the real-time digital twin method for robot work scenarios runs on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and client devices.

[0028] As a possible example, the real-time digital twin method for robot work scenarios provided in this application embodiment can be applied to a web architecture, which can be implemented by the interaction of various devices under the web architecture. Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 1 As shown, the application scenario structure in which the real-time digital twin method for robot working scenarios provided in this application embodiment is applied may include at least: a visual acquisition device 101, an edge computing device 102, a cloud server 103, a robot 104, and a front-end device 105.

[0029] The visual acquisition device 101 can be used to acquire color and depth images. The visual acquisition device 101 is communicatively connected to the edge computing device 102 to transmit the acquired color and depth images to the edge computing device 102, which then performs visual perception or trains a 3D model of the object based on the acquired color and depth images. The edge computing device 102 is also communicatively connected to the cloud server 103, enabling the cloud server to perform real-time rendering of the digital twin scene corresponding to the robot's working scene based on the pose of the target object obtained through visual perception, such as a six-degree-of-freedom pose, and the 3D Gaussian model of the target object, thus obtaining a real-time rendered image of the digital twin scene. The cloud server 103 is also communicatively connected to the front-end device 105, enabling the front-end device 105 to display the digital twin scene in real-time based on the real-time rendered image of the digital twin scene. The robot 104 can be any robot set up in the robot's working scene, such as an industrial robot or a handling robot. The front-end device 105 can display a digital twin scene of the robot's work scene through a web application. The digital twin scene can display the virtual environment corresponding to the robot's work scene, as well as the robot model corresponding to the robot 104. In addition, it can also display the three-dimensional models of the objects in the robot's work scene, such as the three-dimensional Gaussian model of the target object.

[0030] In addition, robot 104 can also be communicatively connected to front-end device 105 to synchronize status data packets to front-end device 105 in real time, so that the front-end device can synchronize the robot model in the digital twin scenario in real time based on the real-time synchronized status data packets, thereby realizing the twin drive of the robot model in the digital twin scenario.

[0031] Taking a warehousing and logistics scenario or an automated production line scenario as an example, during the object receiving stage, the object can enter the robot working scene through the receiving conveyor belt, and the physical robot will handle the object in the robot working scene.

[0032] In one example, the visual acquisition device 101 may include a red-green-blue (RGB) camera and a depth camera, which are used to acquire two-dimensional images and depth images, respectively. In another example, the visual acquisition device 101 may also be an RGB-D camera with the ability to acquire both color and depth images.

[0033] During the object loading stage, the visual acquisition device 101 can be a camera array arranged in a ring at the corresponding position of the loading conveyor belt. The camera array consists of multiple visual acquisition devices 101 to acquire multi-view color images and corresponding depth images.

[0034] During the object grasping and handling stage, the visual acquisition device 101 can be deployed at a preset global top-down position in the robot's working scene to acquire real-time color images and real-time depth images of the robot's working scene.

[0035] Of course, the above-mentioned warehousing and logistics scenarios or automated production line scenarios are merely examples of possible application scenarios used in the embodiments of this application. In actual applications, the application scenarios are not limited to the above examples. If the possible application scenarios used in the embodiments of this application can also be other application scenarios, then there may be no object entry stage. Accordingly, as long as the visual acquisition device 101 can acquire multi-view color images of the object and the corresponding depth images, the deployment scheme of the visual acquisition device 101 is not limited.

[0036] It should also be noted that the application architecture of this application embodiment is not limited to the above-described web page architecture, and can also be other forms of application architecture, which this application embodiment does not limit.

[0037] The following examples illustrate the real-time digital twin method for robot working scenarios provided in this application.

[0038] Figure 2 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 1 The real-time digital twin method for the robot's working scene can be implemented through software and / or hardware. Figure 2 As shown, the method may include: S201. Acquire real-time video streams of the robot's working scene. The real-time video streams include: real-time color image streams and real-time depth image streams.

[0039] This real-time video stream can be a video stream captured by a vision acquisition device deployed at a preset global overhead position in the robot's working environment. The real-time color image stream is specifically a real-time RGB image stream.

[0040] S202. Segment the real-time color image stream to obtain the two-dimensional boundary information of all objects in the robot's working scene, as well as the instance mask.

[0041] In one possible implementation, an instance segmentation algorithm can be used to perform real-time inference on the real-time color image stream, obtaining the two-dimensional boundary information of all objects in the robot's working scene and the instance mask of each object. The two-dimensional boundary information can be the two-dimensional bounding boxes of the objects. The preset instance segmentation algorithm can be an object detection algorithm, such as the YOLO (You Only LookOnce) series of algorithms. The instance mask of each object can also be a pixel-level mask of the object.

[0042] S203. Based on the two-dimensional boundary information of the target object among all objects, obtain the three-dimensional Gaussian model of the target object from the preset three-dimensional object model library.

[0043] The preset object 3D model library pre-stores multiple 3D Gaussian models of objects. Each object's 3D Gaussian model can be obtained by pre-modeling the object based on its multi-view color images and corresponding depth images. As a possible example, the 3D Gaussian models in the preset object 3D model library could be models obtained during the object entry stage, or models obtained at other stages. Each object's 3D Gaussian model is a set of 3D Gaussian spheres for that object. Each 3D Gaussian sphere in the 3D Gaussian model can have corresponding attribute parameters, which may include, for example, position parameters, rotation parameters, scaling parameters, opacity parameters, and spherical harmonic coefficients, among other multi-dimensional attribute parameters.

[0044] If all objects in the robot's working environment are identified by executing S202, and if the target object is among all the objects, then the two-dimensional boundary information of the target object can be obtained. The target object can be the physical object that the robot needs to grasp in the robot's working environment.

[0045] As one possible approach, the three-dimensional Gaussian model of the target object can be obtained by matching the three-dimensional Gaussian model of each object in a preset three-dimensional object model library based on the two-dimensional boundary information of the target object.

[0046] S204. Based on the instance mask, the 3D Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream, perform six-degree-of-freedom pose estimation on the target object to obtain the real-time six-degree-of-freedom pose stream of the target object.

[0047] One possible implementation is to use the 3D Gaussian model of the target object as prior information. Using the instance mask, the 3D Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream, six-degree-of-freedom (6DOF) pose estimation of the target object is performed to obtain the 6DOF pose for each frame. Based on the 6DOF poses of multiple frames, a real-time 6DOF pose stream of the target object is obtained. The 6DOF pose of each frame may include: 3 translational DOF poses and 3 rotational DOF poses. The 6DOF pose of each frame in the real-time 6DOF pose stream of the target object is the 6DOF pose matrix of the object relative to a preset reference coordinate system in the corresponding frame. The preset reference coordinate system can be, for example, the world coordinate system of a preset robot working scene or the robot base coordinate system.

[0048] S205. Based on the real-time six-degree-of-freedom pose flow of the target object, render the three-dimensional Gaussian model of the target object in real time in the digital twin scene corresponding to the robot's working scene.

[0049] For example, based on the real-time six-DOF pose flow of the target object, a pre-defined Gaussian Splatting Shader can be used to perform Gaussian sputtering rendering on the 3D Gaussian model of the target object in the corresponding digital twin scene of the robot's working scenario, achieving real-time rendering of the 3D Gaussian model of the target object. For instance, a pre-defined rendering engine, such as the open-source web image engine Three.js, can be used with a pre-defined Gaussian sputtering shader to perform Gaussian sputtering rendering on the 3D Gaussian model of the target object in the digital twin scene. The Gaussian sputtering shader can also form a 3D Gaussian rendering pipeline.

[0050] A digital twin scenario can also include a robot model corresponding to the physical robot, as well as a static environment of the robot's working scene. The static environment may include, for example, shelves or the floor. The robot model and the static environment can be rendered using a mesh rendering pipeline from a pre-defined Web Graphics Library (WebGL). This ensures the clarity of the geometric structure of the robot model and the static background in the digital twin scenario.

[0051] In the solution provided in this embodiment, the 3D Gaussian model of the target object in the digital twin scene, as a high-fidelity object, can be rendered in real time using a preset Gaussian sputtering shader. For the robot model and static environment in the digital twin scene, a mesh rendering pipeline from a preset web graphics library is used for rendering. Therefore, for the digital twin scene, the rendering engine is actually a hybrid rendering engine built with both a preset mesh rendering pipeline from the web graphics library and a preset Gaussian sputtering shader. This avoids the dependence on high-performance graphics workstations found in web architectures. Users can access and monitor the system simply through a standard browser, achieving low-latency, high-frame-rate, cross-platform remote digital twin visualization, improving system accessibility, and significantly reducing system deployment costs and terminal hardware requirements.

[0052] The real-time digital twin method for robot work scenes provided in this embodiment can obtain the real-time video stream of the robot work scene, segment the real-time color image stream in the real-time video stream to obtain the two-dimensional boundary information of all objects in the robot work scene, as well as the instance mask, thereby realizing visual perception and real-time reasoning of objects in the robot work scene; it can also obtain the three-dimensional Gaussian model of the target object from a preset object three-dimensional model library based on the two-dimensional boundary information of the target object, thereby realizing the localization of the three-dimensional Gaussian model of the target object; and based on the instance mask, the three-dimensional Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream, it can perform six-degree-of-freedom pose estimation of the target object to obtain the real-time six-degree-of-freedom pose stream of the target object, thereby realizing the six-degree-of-freedom pose estimation of the target object; it can also render the three-dimensional Gaussian model of the target object in real-time in the corresponding digital twin scene of the robot work scene based on the real-time six-degree-of-freedom pose stream of the target object. The robot working scene provided in this application uses a 3D Gaussian model as the target object model in the corresponding digital twin scene. This avoids the excessive rendering load caused by traditional mesh model rendering and the slow inference speed and poor real-time performance issues associated with neural radiation field technology for model rendering. The 3D Gaussian model is directly rasterized and rendered in real-time on the web page, effectively ensuring efficient rendering on the web page and guaranteeing high-definition image quality and high frame rate in the digital twin scene. This effectively resolves the contradiction between high fidelity and real-time performance of the 3D object model in the digital twin scene. Furthermore, in the process of estimating the six-degree-of-freedom pose of the target object, not only are instance masks of real-time color images used, but also the 3D Gaussian model of the target object, as well as real-time color image streams and real-time depth image streams. Therefore, it is not a purely visual solution. It effectively avoids the limitations of object pose tracking caused by purely visual image solutions, improves the accuracy of object pose, and thus improves the synchronization accuracy between the object pose in the real scene and the corresponding 3D object model in the digital twin scene, ensuring visual consistency between the real scene and the digital twin scene.

[0053] Secondly, the method provided in this application embodiment can also serve as a precise digital sandbox for robot work scenarios, applied to robot offline programming, path planning verification, remote fault diagnosis, and operator training, significantly shortening the debugging cycle of new workstations and improving the reliability and efficiency of production operations.

[0054] Based on the above embodiments, this application also provides a possible implementation method for determining a three-dimensional Gaussian model. Figure 3 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 2 ,like Figure 3As shown, in the above method, step S203, obtaining the 3D Gaussian model of the target object from the preset 3D object model library based on the 2D boundary information of the target object among all objects may include: S301. Obtain the physical size information of the target object based on the two-dimensional boundary information of the target object.

[0055] In one possible implementation, the object size can be determined based on the two-dimensional boundary information of the target object, thus obtaining the physical size information of the target object.

[0056] S302. Based on the physical size information of the target object, a preset object mapping relationship table is used to determine the identifier of the target object and the corresponding model file path information.

[0057] The preset object mapping table stores the identifiers, physical dimensions, and corresponding model file paths of multiple objects. Specifically, based on the physical dimensions of the target object, the preset object mapping table is used to match the physical dimensions of each object, obtaining the identifier of the matching object whose physical dimensions match among multiple items. This identifier is then used as the identifier of the target object, thus achieving object identifier association. Furthermore, given the identifier of the target object, the corresponding model file path can be determined from the preset object mapping table and used as the model file path for the target object.

[0058] S303. Based on the model file path information, obtain the three-dimensional Gaussian model of the target object from the preset three-dimensional object model library.

[0059] This embodiment provides a solution that can determine the physical size information of the target object based on its two-dimensional boundary information, and associate the object identifier with a preset object mapping relationship table based on the physical size information of the target object to obtain the identifier of the target object and the corresponding model file path information. Based on the model file path information, the three-dimensional Gaussian model of the target object is obtained from the preset three-dimensional object model library, thus realizing the accurate positioning of the three-dimensional Gaussian model of the target object in the preset three-dimensional object model library.

[0060] Based on any of the above embodiments, this application also provides a possible implementation method for performing three-dimensional modeling of various objects. The three-dimensional Gaussian model of each object can be obtained, for example, by performing real-time modeling during the process of importing each object into a database. Figure 4 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 3 .like Figure 4 As shown, based on the above method, it may also include: S401. Obtain multi-view color images of the object and the corresponding depth images.

[0061] For example, during the process of objects entering the warehouse, such as when objects pass through a preset conveyor belt, multi-view color images and corresponding depth images can be obtained by a visual acquisition device that performs multi-view image acquisition on the objects. Among them, the multi-view color images can include color images from multiple perspectives, and the multiple perspectives can be sparse perspectives covering 360 degrees of the object.

[0062] S402. Based on the multi-view color image of the object and the corresponding depth image, obtain the sparse point cloud of the object.

[0063] One possible approach is to perform multi-view pose calculation based on multi-view color images and corresponding depth images of the object to generate a sparse point cloud of the object. For example, a pre-defined motion recovery structure algorithm can be used to calculate the pose of the multi-view color images and corresponding depth images of the object to obtain the sparse point cloud of the object.

[0064] S403. Based on the sparse point cloud of the object, a preset three-dimensional Gaussian sputtering algorithm is used to generate a three-dimensional Gaussian model of the object.

[0065] Specifically, the sparse point cloud of the object can be used as the initial position of the initial 3D Gaussian sphere. Based on this initial position, a preset 3D Gaussian sputtering algorithm is used for real-time training to obtain the 3D Gaussian model of the object. For example, the position of the sparse point cloud can be used as the initial position to generate an initial 3D Gaussian sphere. Parameters of the initial 3D Gaussian sphere are set, and based on these parameters, a differentiable raster renderer is used to obtain rendered images of the object from various viewpoints. The rendered images and the original color image from each viewpoint are calculated, and the rendering loss is calculated. Based on the rendering loss, backpropagation optimization is performed to update the parameters of the initial 3D Gaussian sphere. After updating the parameters of the initial 3D Gaussian sphere, the updated parameters can be used to perform differentiable raster renderer again to obtain new rendered images of the object from various viewpoints, until a preset stopping iteration condition is reached. The preset stopping iteration condition can be, for example, that the rendering loss between the newly rendered image and the original color image meets a preset rendering loss condition, and / or that a preset number of training iterations is reached. If the preset stopping iteration condition is reached, the set of three-dimensional Gaussian spheres formed by all three-dimensional Gaussian spheres at the time the preset stopping iteration condition is reached can be determined as the three-dimensional Gaussian model of the object.

[0066] S404. Store the object's 3D Gaussian model in the preset object 3D model library and obtain the object's model file path information.

[0067] S405. Based on the object's file path information, object identifier, and corresponding physical size information, establish a preset object mapping relationship table.

[0068] In one possible implementation, the object's identifier and physical size information can be obtained by recognizing the identifier on the object. The identifier can be, for example, a barcode, QR code, or RFID code. With the obtained physical size information, a pre-defined object mapping table can be established based on the object's file path information, the identifier, and the corresponding physical size information, enabling the registration of the object's digital asset. It should be noted that the pre-defined object mapping table can also store the model representation of the object, i.e., the identifier of the corresponding 3D Gaussian model.

[0069] The method provided in this application embodiment can use a three-dimensional Gaussian sputtering algorithm to perform three-dimensional modeling of an object, obtaining a three-dimensional Gaussian model of the object, thus realizing rapid modeling during the object's storage process. Compared with traditional camera measurement or mesh methods, after acquiring multi-view color images and depth images, training can be completed in a very short time to generate a three-dimensional Gaussian model of the object as a high-fidelity digital model.

[0070] Based on any of the above embodiments, this application also provides a possible implementation method for real-time six-degree-of-freedom pose estimation of an object. Figure 5 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 4 .like Figure 5 As shown, in the above method, S204 performs six-degree-of-freedom pose estimation on the target object based on the instance mask, the 3D Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream, obtaining the real-time six-degree-of-freedom pose stream of the target object, which may include: S501, Align the real-time depth image stream and the real-time color image stream.

[0071] In one possible implementation, the image boundaries of each frame of depth image in the real-time depth image stream and the image boundaries of each frame of color image in the real-time color image stream can be aligned so that the regions corresponding to each frame of depth image are the same as the regions corresponding to each frame of color image.

[0072] S502. Based on the instance mask, crop out the local point cloud data of the target object from the corresponding frame depth image in the real-time depth image stream.

[0073] Specifically, in a real-time image stream, each frame of depth image has a corresponding frame of color image; similarly, each frame of color image in a real-time color image stream also has a corresponding frame of depth image. Since the instance mask is obtained by segmenting the real-time color image stream, it actually contains the instance mask of the object in each frame of color image. Thus, based on the instance mask, point cloud cropping can be performed from the corresponding frame of depth image in the real-time depth image stream to obtain the local point cloud data of the target object.

[0074] Optionally, after cropping the local point cloud data of the target object, coordinate system transformation can be performed on the local point cloud data of the target object to transform the local point cloud data of the target object from the preset camera coordinate system to the preset reference coordinate system such as the world coordinate system.

[0075] S503. Based on the instance mask, obtain the corresponding color image region from the corresponding frame color image in the real-time color image stream.

[0076] This color image region is the object image region of the target object in the corresponding frame of the color image.

[0077] S504. Based on the local point cloud data, the color image region, and the three-dimensional Gaussian model of the target object, perform six-degree-of-freedom pose estimation on the target object to obtain the six-degree-of-freedom pose of the target object.

[0078] In one possible implementation, local point cloud data and color image regions can be used as input data. Based on the 3D Gaussian model of the target object, a keypoint-based pose estimation algorithm is employed to calculate the six-DOF pose of the target object, thus obtaining the six-DOF pose of the target object. The keypoint-based pose estimation algorithm can be, for example, the PnP algorithm.

[0079] Since the local point cloud data and the color image region are obtained from the depth image and the color image of the corresponding frame, respectively, the local point cloud data and the color image region are the local point cloud data and the color region image of the corresponding frame, respectively. Accordingly, the six-degree-of-freedom pose of the target object obtained by six-degree-of-freedom pose estimation is the six-degree-of-freedom pose of the corresponding frame.

[0080] By sequentially executing S502 and S503 on multiple frames of depth images in the real-time depth image stream and multiple frames of color images in the real-time color image stream, multiple frames of local point cloud data and the corresponding color image regions can be obtained. Accordingly, based on the local point cloud data of each frame and the corresponding color image regions, combined with the three-dimensional Gaussian model of the target object, executing S504 on the above-mentioned basis, the six-degree-of-freedom pose of the target object in the corresponding frame can be obtained.

[0081] S505. Based on the six-degree-of-freedom pose of the target object, obtain the real-time six-degree-of-freedom pose flow of the target object.

[0082] By performing the above steps, the six-degree-of-freedom pose of the target object in multiple frames can be obtained. Furthermore, the six-degree-of-freedom poses in multiple frames can be filtered to obtain the real-time six-degree-of-freedom pose flow of the target object. For example, a Kalman filter algorithm can be used to filter the six-degree-of-freedom poses in multiple frames.

[0083] The method provided in this embodiment utilizes a six-degree-of-freedom (DOF) visual tracking algorithm based on instance masks, real-time color image streams, real-time depth image streams, and a 3D Gaussian model of the target object, rather than a purely visual image-based approach. This effectively avoids the limitations of object pose tracking inherent in purely visual methods, improves the accuracy of object pose tracking, and thus enhances the synchronization precision between the object pose in the real-world scene and the corresponding 3D model in the digital twin scene. Furthermore, by filtering the six-degree-of-freedom poses of multiple frames to obtain the real-time six-degree-of-freedom pose stream of the target object, visual jitter can be effectively eliminated, resulting in a smoother and more accurate real-time six-degree-of-freedom pose stream for the target object.

[0084] Furthermore, based on the methods provided in any of the above embodiments, this application also provides a possible implementation of twin-driven operation for robot models in a digital twin scenario. Figure 6 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 5 .like Figure 6 As shown, the method may also include: S601. Obtain the joint status data packet sent by the physical robot in the robot's working environment.

[0085] The robot controller of a physical robot can send joint status data packets to the front-end device at high frequency via a preset network transmission protocol. These joint status data packets may include a current timestamp and the pose information of each joint corresponding to that timestamp. The preset network transmission protocol can be, for example, TCP / UDP or MQTT, and the high frequency can be, for example, a data transmission frequency of 50Hz to 100Hz.

[0086] S602. Based on the joint state data packet, use the kinematic tree of the physical robot model corresponding to the robot model in the digital twin scene to update the posture of the corresponding joint node on the robot model in real time.

[0087] The kinematic tree of a robot model can be constructed by pre-loading the robot description file of the target robot, parsing the robot description file to obtain the link and joint hierarchy of the physical robot, and building a kinematic tree based on the link and joint hierarchy of the physical robot. After acquiring the joint state data packet, the front-end device can use the kinematic tree of the corresponding robot model to update the pose of the corresponding joint nodes in the robot model in real time according to the forward kinematics. The pose of the joint node may include the rotation matrix of the joint node. Optionally, based on the joint state data packet, the kinematic tree of the corresponding robot model can also be used to calculate and update the pose (position and orientation) of the corresponding node of the end effector in the robot model.

[0088] The method provided in this application embodiment can obtain the joint state data packet sent by the physical robot in the robot's working environment, and based on the joint state data packet, use the kinematic tree of the corresponding robot model in the digital twin scenario to update the posture of the corresponding joint node on the robot model in real time, thereby realizing the real-time reproduction of the posture of the robot model in the digital twin scenario and thus realizing the twin drive of the robot model in the digital twin scenario.

[0089] Based on the methods provided in any of the above embodiments, this application embodiment may also provide a possible implementation example for real-time rendering of a digital twin scene. Figure 7 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 6 ,like Figure 7 As shown, in the above method, step S205, which involves real-time rendering of the 3D Gaussian model of the target object in the digital twin scene corresponding to the robot's working scene based on the real-time six-DOF pose flow of the target object, may include: S701. Based on the real-time six-degree-of-freedom pose flow of the target object, update the pose of each three-dimensional Gaussian sphere in the three-dimensional Gaussian model of the target object to obtain the updated three-dimensional Gaussian model.

[0090] As a possible implementation example, the center position and rotation covariance matrix of each three-dimensional Gaussian sphere in the three-dimensional Gaussian model of the target object can be transformed in real time according to the real-time six-degree-of-freedom pose flow of the target object, thereby realizing the pose update of each three-dimensional Gaussian sphere.

[0091] Preset Gaussian sputtering shaders may include, for example, vertex shaders and fragment shaders. For instance, the vertex shader can update the poses of each 3D Gaussian sphere in the 3D Gaussian model of the target object based on the real-time six-DOF pose flow of the target object, thus obtaining an updated 3D Gaussian model.

[0092] S702. Real-time rendering of the updated 3D Gaussian model.

[0093] After updating the 3D Gaussian model, a fragment shader can be used to render the updated Gaussian model in real time.

[0094] The method provided in this application embodiment can also employ a Gaussian sputtering shader to update the poses of each 3D Gaussian sphere in the 3D Gaussian model of the target object according to the real-time six-degree-of-freedom pose flow of the target object, thereby obtaining an updated 3D Gaussian model. The updated 3D Gaussian model is then rendered in real time, achieving real-time rendering of the 3D Gaussian model. This improves the real-time performance, rendering effect, and rendering accuracy of the rendering, while avoiding dependence on high-performance graphics workstations.

[0095] Furthermore, based on the methods provided in any of the above embodiments, this application embodiment may also provide a possible implementation example of posture updating of a robot model in a digital twin scenario. Figure 8 A flowchart illustrating a real-time digital twin method for robot work scenarios provided in this application embodiment. Figure 7 ,like Figure 8 As shown, the method also includes: S801, Monitor the input and output signals of the end effector of the physical robot in the robot's working environment.

[0096] S802. Based on the input and output signals, dynamically reorganize the parent-child relationship of the three-dimensional Gaussian model corresponding to the target object in the digital twin scene.

[0097] For example, if the input / output signal is a vacuum suction cup engagement signal or a gripper closure signal, it can be determined that the end effector has grasped the target object, i.e., the end effector has successfully grasped the target object. In possible implementations, the parent-child relationship progress of the 3D Gaussian model corresponding to the target object in the digital twin scene can also be dynamically reorganized in the grasping state based on the input / output signal. For example, if the input / output signal indicates successful grasping, the parent node of the 3D Gaussian model corresponding to the target object in the digital twin scene can be attached from the world scene to the end flange on the robot model. Conversely, if the input / output signal indicates successful placement, the parent-child relationship between the parent node of the 3D Gaussian model corresponding to the target object in the digital twin scene and the end flange on the robot model can be re-attached to the world scene, and the coordinates of the target object can be updated to the latest world coordinates fed back by the vision system.

[0098] The method provided in this application can dynamically reorganize the parent-child relationships of the 3D Gaussian model corresponding to the target object in the digital twin scene based on the input and output signals of the end effector. This allows the object's pose to smoothly switch from the easily occluded visual tracking data source to the kinematic coordinate system of the robot's end effector during the grasping process. This means the object's pose no longer depends on the estimation of the visual camera, but directly follows the movement of the robot's end effector, avoiding the visual loss problem caused by the robotic arm's occlusion. This completely overcomes the visual tracking loss or instantaneous shift phenomenon caused by the robotic arm itself occluding the global camera, ensuring the continuity and accuracy of dynamic object pose synchronization in the digital twin scene.

[0099] As another possible implementation, the redrawing instructions can be updated only for the corresponding joints in the 3D Gaussian model and / or robot model of the object whose pose changes in the digital twin scene, so as to achieve twin synchronization between the digital twin scene and the real scene. For static backgrounds, updates can be performed based on a caching strategy, which ensures both the real-time performance and smoothness of scene rendering.

[0100] The following describes the apparatus, equipment, and storage medium used to implement the real-time digital twin method for robot work scenarios provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0101] Figure 9 This is a schematic diagram of a real-time digital twin device for a robot working scene provided in an embodiment of this application, as shown below. Figure 9 As shown, the real-time digital twin device 900 for robot work scenarios may include: The acquisition module 901 is used to acquire real-time video streams of the robot's working scene. The real-time video streams include real-time color image streams and real-time depth image streams.

[0102] The segmentation module 902 is used to segment based on the real-time color image stream to obtain the two-dimensional boundary information of all objects in the robot's working scene, as well as the instance mask.

[0103] The pose estimation module 903 is used to obtain the three-dimensional Gaussian model of the target object from the preset three-dimensional object model library based on the two-dimensional boundary information of the target object among all objects. The preset three-dimensional object model library stores the three-dimensional Gaussian models of multiple objects in advance. Based on the instance mask, the three-dimensional Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream, the six-degree-of-freedom pose of the target object is estimated to obtain the real-time six-degree-of-freedom pose stream of the target object.

[0104] The rendering module 904 is used to render the 3D Gaussian model of the target object in real time in the digital twin scene corresponding to the robot's working scene, based on the real-time six-degree-of-freedom pose flow of the target object.

[0105] As a possible example, the pose estimation module 903 is specifically used to: obtain the physical size information of the target object based on the two-dimensional boundary information of the target object; determine the identifier of the target object and the corresponding model file path information using a preset object mapping relationship table based on the physical size information of the target object; the preset object mapping relationship table stores the identifiers, physical size information and corresponding model file path information of multiple objects; and obtain the three-dimensional Gaussian model of the target object from the preset three-dimensional object model library based on the model file path information.

[0106] As one possible example, the real-time digital twin device 900 for robot work scenarios also includes: The object modeling module is used to acquire multi-view color images and corresponding depth images of an object; based on the multi-view color images and corresponding depth images, it acquires the sparse point cloud of the object; based on the sparse point cloud, it generates a 3D Gaussian model of the object using a preset 3D Gaussian sputtering algorithm; it stores the 3D Gaussian model of the object in a preset 3D object model library, obtaining the object's model file path information; and based on the object's file path information, object identifier, and corresponding physical size information, it establishes a preset object mapping relationship table.

[0107] As one possible example, the pose estimation module 903 is specifically used for: aligning the real-time depth image stream and the real-time color image stream; cropping local point cloud data of the target object from the corresponding frame depth image in the real-time depth image stream according to the instance mask; obtaining the corresponding color image region from the corresponding frame color image in the real-time color image stream according to the instance mask; performing six-degree-of-freedom pose estimation on the target object based on the local point cloud data, the color image region, and the three-dimensional Gaussian model of the target object to obtain the six-degree-of-freedom pose of the target object; and obtaining the real-time six-degree-of-freedom pose stream of the target object based on the six-degree-of-freedom pose.

[0108] As one possible example, the real-time digital twin device 900 for robot work scenarios also includes: The first update module is used to: acquire joint state data packets sent by the physical robot in the robot's working environment; and, based on the joint state data packets, use the kinematic tree of the robot model corresponding to the physical robot in the digital twin scenario to update the posture of each corresponding joint node on the robot model in real time.

[0109] As a possible example, the rendering module 904 is specifically used to: update the poses of each 3D Gaussian sphere in the 3D Gaussian sputtering model of the target object according to the real-time six-DOF pose flow of the target object, to obtain the updated 3D Gaussian model; and to render the updated 3D Gaussian sputtering model in real time.

[0110] As one possible example, the real-time digital twin device 900 for robot work scenarios also includes: The second update module is used to: monitor the input and output signals of the end effector of the physical robot in the robot's working scene; and dynamically reorganize the parent-child relationship of the three-dimensional Gaussian model corresponding to the target object in the digital twin scene based on the input and output signals.

[0111] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0112] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0113] Figure 10 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer can be integrated into the device or a chip of the device, and the computer device can be as described above. Figure 1 The application scenario architecture shown includes edge computing devices and cloud servers.

[0114] The computer device includes a processor 1001, a storage medium 1002, and a bus 1003. The storage medium 1002 stores machine-readable instructions executable by the processor 1001. When the computer device runs a real-time digital twin method for a robot work scene as described in the embodiment, the processor 1001 communicates with the storage medium 1002 via the bus 1003. The processor 1001 executes the machine-readable instructions to perform the aforementioned embodiment of the real-time digital twin method for a robot work scene. The specific implementation and technical effects are similar and will not be described in detail here.

[0115] Optionally, embodiments of this application also provide a computer-readable storage medium storing a computer program, which is executed by a processor to perform the above-described real-time digital twin method embodiment for robot working scenarios.

[0116] Optionally, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the real-time digital twin method for robot working scenarios described in any embodiment.

[0117] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0120] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for real-time digital twin creation of a robot's working scene, characterized in that, The method includes: Acquire a real-time video stream of the robot's working scene, the real-time video stream including: a real-time color image stream and a real-time depth image stream; The real-time color image stream is segmented to obtain the two-dimensional boundary information of all objects in the robot's working scene, as well as the instance mask; Based on the two-dimensional boundary information of the target object among all objects, the three-dimensional Gaussian model of the target object is obtained from the preset three-dimensional object model library; wherein, the preset three-dimensional object model library stores three-dimensional Gaussian models of multiple objects in advance. Based on the instance mask, the 3D Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream, the target object's six-degree-of-freedom pose is estimated to obtain the target object's real-time six-degree-of-freedom pose stream. Based on the real-time six-degree-of-freedom pose flow of the target object, the three-dimensional Gaussian model of the target object is rendered in real time in the digital twin scene corresponding to the robot's working scene.

2. The method according to claim 1, characterized in that, The step of obtaining the three-dimensional Gaussian model of the target object from a preset three-dimensional object model library based on the two-dimensional boundary information of the target object among all objects includes: Based on the two-dimensional boundary information of the target object, obtain the physical size information of the target object; Based on the physical size information of the target object, a preset object mapping table is used to determine the identifier of the target object and the corresponding model file path information; the preset object mapping table stores the identifiers, physical size information and corresponding model file path information of multiple objects; Based on the model file path information, obtain the three-dimensional Gaussian model of the target object from the preset three-dimensional object model library.

3. The method according to claim 2, characterized in that, The method further includes: Acquire multi-view color images and corresponding depth images of the object; Based on the multi-view color image of the object and the corresponding depth image, obtain the sparse point cloud of the object; Based on the sparse point cloud of the object, a preset three-dimensional Gaussian sputtering algorithm is used to generate a three-dimensional Gaussian model of the object. The three-dimensional Gaussian model of the object is stored in the preset three-dimensional object model library to obtain the model file path information of the object; Based on the file path information of the object, the identifier of the object, and the corresponding physical size information, a preset object mapping relationship table is established.

4. The method according to any one of claims 1-3, characterized in that, The step of performing six-DOF pose estimation on the target object based on the instance mask, the three-dimensional Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream to obtain the real-time six-DOF pose stream of the target object includes: Align the real-time depth image stream and the real-time color image stream; Based on the instance mask, local point cloud data of the target object is cropped from the corresponding frame depth image in the real-time depth image stream; Based on the instance mask, the corresponding color image region is obtained from the corresponding frame color image in the real-time color image stream; Based on the local point cloud data, the color image region, and the three-dimensional Gaussian model of the target object, the six-degree-of-freedom pose of the target object is estimated to obtain the six-degree-of-freedom pose of the target object. Based on the six-degree-of-freedom pose, the real-time six-degree-of-freedom pose flow of the target object is obtained.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain the joint status data packets sent by the physical robot in the robot's working environment; Based on the joint state data packet, the kinematics tree of the robot model corresponding to the physical robot in the digital twin scenario is used to update the posture of each corresponding joint node on the robot model in real time.

6. The method according to any one of claims 1-3, characterized in that, The step of rendering a 3D Gaussian model of the target object in real time within the digital twin scene corresponding to the robot's working scene, based on the real-time six-DOF pose flow of the target object, includes: Based on the real-time six-degree-of-freedom pose flow of the target object, the poses of each three-dimensional Gaussian sphere in the three-dimensional Gaussian sputtering model of the target object are updated to obtain the updated three-dimensional Gaussian model. The updated 3D Gaussian sputtering model is rendered in real time.

7. The method according to any one of claims 1-3, characterized in that, The method further includes: Monitor the input and output signals of the end effector of the physical robot in the robot's working environment; Based on the input and output signals, the parent-child relationship of the three-dimensional Gaussian model corresponding to the target object in the digital twin scene is dynamically reorganized.

8. A real-time digital twin device for a robot's work scene, characterized in that, The device includes: The acquisition module is used to acquire real-time video streams of the robot's working scene, the real-time video streams including: real-time color image streams and real-time depth image streams; The segmentation module is used to segment the real-time color image stream to obtain the two-dimensional boundary information of all objects in the robot's working scene, as well as the instance mask; The pose estimation module is used to obtain the three-dimensional Gaussian model of the target object from a preset three-dimensional object model library based on the two-dimensional boundary information of the target object among all objects; wherein, the preset three-dimensional object model library stores three-dimensional Gaussian models of multiple objects in advance; and to perform six-degree-of-freedom pose estimation on the target object based on the instance mask, the three-dimensional Gaussian model of the target object, the real-time color image stream, and the real-time depth image stream to obtain the real-time six-degree-of-freedom pose stream of the target object. The rendering module is used to render the three-dimensional Gaussian model of the target object in real time in the digital twin scene corresponding to the robot's working scene, based on the real-time six-degree-of-freedom pose flow of the target object.

9. A computer device, characterized in that, include: The computer device includes a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of a real-time digital twin method for a robot working scene as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the real-time digital twin method for a robot working scene as described in any one of claims 1 to 7.

Images