Method and system for transmitting long time series of robot operations

Abstract

The application discloses a kind of robot long time sequence operation transmission method and system, the method includes: obtaining the initial full 3D point cloud data of long time sequence operation scene, real-time acquisition long time sequence operation scene time sequence point cloud data stream;The initial full 3D point cloud data obtained is executed denoising and instance segmentation preprocessing, and the initial scene effective point cloud data set grouped according to object instance is obtained;Each point cloud data point in initial scene effective point cloud data set is encapsulated as particle state unit with state update mark;Real-time acquisition time sequence point cloud data stream is executed standardization particle encapsulation, and the complete distributed non-group framing streaming transmission of single-particle incremental binary data package generated is executed;Dynamic particle incremental data is obtained by restoring;Long time sequence non-drift 3D point cloud reconstruction is completed based on real-time global particle state set, and the planning and control of long time sequence operation are realized based on 3D point cloud data after reconstruction.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a transmission method and system for long-term operation of a robot. Background Technology

[0002] Long-sequence robotic operations enable robots to perform complex tasks in a continuous, stable, and error-correctable manner through hierarchical planning, world modeling, memory enhancement, and end-to-end long-sequence modeling, involving long sequences of hundreds to thousands of steps, dynamic disturbances, and multi-granularity tasks.

[0003] In real-world scenarios involving long-duration robotic operations, such as folding clothes, pouring liquids, and transporting items over long distances, the cloud video streams collected by the robot need to continuously transmit complete frame data. This complete frame data contains a large amount of redundant data, leading to a continuous accumulation of data transmission delays, which greatly consumes local computing power and wastes both data transmission and computing power. Summary of the Invention

[0004] In view of this, the embodiments of this application provide a transmission method and system for long-term robot operations, which solves the technical problems of large amount of redundant transmitted data and insufficient local computing power in long-term robot operations. It has the advantages of reducing data transmission latency, reducing the efficiency of local computing power, improving data transmission efficiency, improving the allocation efficiency of local computing power, and improving the real-time performance of the robot.

[0005] In a first aspect, embodiments of this application provide a transmission method for long-term operations of a robot, the method comprising: Acquire the initial full 3D point cloud data of long-term operation scenarios and collect the time-series point cloud data stream of long-term operation scenarios in real time; The initial full 3D point cloud data is preprocessed with denoising and instance segmentation to obtain an initial scene valid point cloud dataset grouped by object instances. Each point cloud data point in the initial scene valid point cloud dataset is encapsulated into a particle state unit with a state update tag to construct an initial full particle state set as a benchmark dataset for long-term state tracking. The real-time acquired time-series point cloud data stream is standardized and encapsulated into particles, and matched with the global particle state set of the previous time step to detect particle state changes. A corresponding state update tag is assigned to each particle to distinguish between static and dynamic particles. Independent incremental encoding is performed on the particle state units marked as dynamic particles to generate single-particle incremental binary data packets. Static particles are not re-encoded or transmitted. The generated single-particle incremental binary data packets are transmitted in a fully distributed, frameless streaming manner. The received single-particle incremental binary data packets are decoded in parallel to restore the dynamic particle incremental data. Based on the globally unique particle identifier, the global particle state set is incrementally updated to obtain the real-time global particle state set. Long-term drift-free 3D point cloud reconstruction is achieved based on real-time global particle state sets, and long-term operation planning and control are realized based on the reconstructed 3D point cloud data.

[0006] Secondly, a transmission system for long-term operation of a robot is provided, comprising: The acquisition unit is used to acquire the initial full 3D point cloud data of the long-term operation scenario and to collect the time-series point cloud data stream of the long-term operation scenario in real time. The processing unit performs denoising and instance segmentation preprocessing on the acquired initial full 3D point cloud data to obtain an initial scene effective point cloud dataset grouped by object instances. Each point cloud data point in the initial scene effective point cloud dataset is encapsulated into a particle state unit with a state update tag, constructing an initial full particle state set as a benchmark dataset for long-term state tracking. The unit performs standardized particle encapsulation on the real-time acquired temporal point cloud data stream, performs particle-by-particle matching with the previous global particle state set, detects particle state changes, assigns a corresponding state update tag to each particle, and distinguishes between static and dynamic particles. Independent incremental encoding is performed on particle state units marked as dynamic particles to generate single-particle incremental binary data packets, while static particles are not repeatedly encoded or transmitted. Fully distributed, frameless streaming transmission is performed on the generated single-particle incremental binary data packets. Parallel decoding is performed on the received single-particle incremental binary data packets to restore the dynamic particle incremental data. Incremental updates are performed on the global particle state set based on the globally unique particle identifier to obtain the real-time global particle state set. The reconstruction unit is used to complete long-term drift-free 3D point cloud reconstruction based on the real-time global particle state set, and to realize the planning and control of long-term operations based on the reconstructed 3D point cloud data.

[0007] Thirdly, embodiments of this application provide a robot, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in the first aspect of embodiments of this application.

[0008] Fourthly, embodiments of this application provide an electronic device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in the first aspect of embodiments of this application.

[0009] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of embodiments of this application.

[0010] Sixthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0011] It is evident that, compared to existing systems / devices that employ continuous frame-by-frame encoding and transmission, long-term operation scenarios suffer from issues such as high spatiotemporal redundancy, continuous accumulation of transmission delays, easy drift of state tracking errors, and excessive resource consumption on the edge side. The technical solution of this application first acquires the initial full 3D point cloud data and the real-time acquired temporal point cloud data stream of the robot's long-term operation scenario. Noise reduction and instance segmentation preprocessing are performed on the initial full 3D point cloud data to obtain an initial scene effective point cloud dataset grouped by object instances. Each point cloud data point in the initial scene effective point cloud dataset is encapsulated as a particle state unit with a state update tag, constructing the initial full particle state set as a benchmark dataset for long-term state tracking. Standardized particle encapsulation is performed on the real-time acquired temporal point cloud data stream, and particle-by-particle matching is performed with the global particle state set of the previous moment to detect particle state changes. A corresponding state update tag is assigned to each particle to distinguish between static and dynamic states. For static and dynamic particles, independent incremental encoding is performed on particle state units marked as dynamic particles to generate single-particle incremental binary data packets. Static particles are not repeatedly encoded or transmitted. The generated single-particle incremental binary data packets are transmitted in a fully distributed, frameless streaming manner without frame-level packet assembly, buffering, or waiting operations. The received single-particle incremental binary data packets are decoded in parallel to restore the incremental data of dynamic particles. Based on the globally unique identifier of the particles, the global particle state set is incrementally updated to obtain the real-time global particle state set. Based on the real-time global particle state set, long-term drift-free 3D point cloud reconstruction is completed. Based on the reconstructed 3D point cloud data, the planning and control of long-term operations are realized. Because the technical solution of this application abandons the traditional serial architecture of continuous full-frame transmission, and based on the state change characteristics of objects in long-term operations, it decomposes the point cloud into static particles and dynamic particles, and only performs incremental encoding and frameless streaming transmission on dynamic particles that undergo state changes, it eliminates a large amount of spatiotemporal redundant data in long-term transmission from the bottom layer. This helps to avoid the continuous accumulation of transmission delay from the root, significantly reduces bandwidth occupation and robot-side computing power overhead, and adapts to the requirements of long-term continuous operation. At the same time, based on the particle-by-particle incremental update mechanism with globally unique particle identifiers, it does not require full-frame refresh of the global object state. Combined with frameless streaming transmission, it achieves low-latency processing of "change is transmitted and arrival is updated", which helps to fundamentally solve the problem of object state tracking error accumulation and drift in long-term operations, and ensures the continuous and stable execution of long-term operation tasks. Therefore, the technical solution of this application has the advantages of reducing data transmission latency, reducing the efficiency of local computing power, improving data transmission efficiency, improving the allocation efficiency of local computing power, and improving the real-time performance of the robot. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the structure of a robot provided in an embodiment of this application; Figure 2 This is a schematic diagram of the software architecture of a robot provided in an embodiment of this application; Figure 3 A flowchart illustrating a transmission method for long-term robot operations provided in this application embodiment; Figure 4 A flowchart illustrating a transmission method for long-term robot operations provided in Embodiment 1 of this application; Figure 5 This is a schematic diagram of the structure of a transmission system for long-term operation of a robot provided in an embodiment of this application; Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0015] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0016] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0017] In this application's embodiments, "multiple" refers to two or more. In this application's embodiments, "connection" refers to various connection methods, such as direct or indirect connections, to achieve communication between devices; this application's embodiments do not impose any limitations on this.

[0018] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0019] The following describes the relevant content, concepts, meanings, technical issues, technical solutions, and beneficial effects involved in the embodiments of this application.

[0020] DiPac (Differentiable Particles) is used for robot manipulation in long-term, complex physical scenarios.

[0021] LLM-State (Large Language Model State) is a scalable state representation framework for open-world, long-term robotic tasks. It primarily addresses the issues of state loss, drift, and wasted computational resources when performing long-term planning in LLM.

[0022] An FPGA (Field-Programmable Gate Array) is a digital chip that can be repeatedly "hardware reconfigured" after leaving the factory. It uses software (hardware description language) to build the internal thousands of logic units, wiring, storage, and DSP into the circuit you want. It can be modified at any time, reconfigured an unlimited number of times, and features hardware parallelism and low latency.

[0023] PCIe Gen3 (Peripheral Component Interconnect Express Generation 3, the third generation high-speed serial computer expansion bus standard) has a single channel speed of 8GT / s.

[0024] URLLC (Ultra-Reliable Low-Latency Communication).

[0025] NVMe (Non-Volatile Memory Express) is a high-speed protocol designed specifically for solid-state drives (SSDs).

[0026] To address the aforementioned issues, this application provides a transmission method and system for long-duration robot operations. This method addresses the problem of excessive redundant data in transmitted data frames during long-duration robot operations, which affects transmission efficiency. It also resolves the issue of significant waste of local computing power during long-duration robot operations, thereby improving data transmission efficiency, increasing the allocation efficiency of local computing power, and enhancing the robot's real-time performance.

[0027] First, please refer to the details. Figure 1 , Figure 1 This is a schematic diagram of the structure of a robot provided in an embodiment of this application, such as... Figure 1 As shown, the structure of the robot described above may specifically include: an RGBD camera array 10, an FPGA parallel processing unit 11, a system-wide timing synchronization hardware unit 12, and a radio frequency transceiver module 13, wherein... The RGBD camera array 10 is physically mounted at the end of the robot arm and fixed points on the robot body, supporting continuous and stable operation 24 / 7. The minimum global shutter exposure time is 10μs, with no motion blur. It adopts a pixel-level parallel output design without frame buffer latch. After the single pixel point cloud data is acquired, it can be directly output in real time through the MIPICSI-2 interface without waiting for the whole frame exposure to be completed, and it is compatible with frameless transmission. It has a built-in hardware-level automatic exposure and white balance correction circuit, and there is no image brightness and depth drift during long-term operation, making it suitable for long-term operation scenarios such as clothing folding and liquid pouring.

[0028] The FPGA parallel processing unit 11 is directly connected to the RGBD camera array 10 via a PCIeGen3×4 physical interface. It adopts a multi-core peer-to-peer parallel hardware logic architecture, with each core mapped one-to-one to the data channel of a single camera, eliminating the need for master-slave cores and centralized data scheduling. It features built-in hardware-level particle state comparison logic circuitry, enabling spatial distance matching of adjacent timestamp particle sets and hardware marking of static / dynamic particles, eliminating the need for frame-level buffering. It also incorporates incremental encoding hardware logic, encoding only the changes in dynamic particles and encoding static particles only once in the initial stage, eliminating software processing latency. It supports wide temperature operation from -40℃ to 85℃, adapting to long-term continuous operation scenarios.

[0029] The system-wide timing synchronization hardware unit 12 includes a high-precision temperature-controlled crystal oscillator (OCXO), a robot joint encoder synchronization interface, and a camera trigger synchronization interface. It is physically connected to the FPGA parallel processing unit 11, the RGBD camera array 10, and the robot controller via a coaxial cable, outputting a 10MHz synchronization clock and a 1PPS pulse per second. It achieves sub-microsecond synchronization accuracy, eliminates clock drift during long-term operation, and ensures that the particle timestamps are perfectly aligned with the time reference of the robot joint pose.

[0030] The RF transceiver module 13 adopts a three-physical-link design of 5G URLLC plus WiFi 6 wired Ethernet. Each link has an independent hardware transceiver front end and a built-in link quality hardware monitoring circuit, which can detect the link packet loss rate, latency, and out-of-order rate in real time and dynamically switch the transmission link at the hardware level. There is no frame-level retransmission logic, and selective retransmission is only performed on high-priority dynamic incremental particle data packets, which is suitable for long-term network fluctuation scenarios.

[0031] For the robot described in this application, a corresponding hardware network also needs to be configured. The aforementioned hardware network may specifically include: An industrial-grade edge forwarding gateway with local non-volatile storage, deployed in a distributed manner. Each gateway has built-in NVMe solid-state storage, providing hardware-level caching of static particle sets, eliminating the need for repeated front-end transmissions. It features built-in stateless hardware forwarding logic that performs pass-through forwarding on incremental particle data packets, with a single packet forwarding latency of less than 1ms and no frame-level buffering. It supports the IEEE 802.1ASTSN time-sensitive networking standard, ensuring no forwarding jitter accumulation during long-term operation.

[0032] The multi-link redundant transmission hardware switching matrix, industrial-grade gigabit TSN switch, and built-in multi-link load balancing hardware logic can dynamically distribute incremental particle data packets to multiple physical links, providing hardware-level resistance to network fluctuations; it also reserves dedicated bandwidth and time slots for high-priority dynamic particle data packets, ensuring low-latency transmission during critical stages of long-sequence operations.

[0033] The distributed link status hardware monitoring unit is deployed at each node of the transmission network. It has built-in real-time detection circuits for packet loss rate, latency, and out-of-order rate. It can feed back link status data to the front-end FPGA encoding unit in real time, adjust the encoding strategy at the hardware level, and adapt to changes in long-term network environments.

[0034] Please refer to the details. Figure 2 , Figure 2 This is a schematic diagram of the software architecture of a robot provided in an embodiment of this application, such as... Figure 2 As shown, the software architecture of the robot described above may specifically include: The long-time-series scene point cloud preprocessing and object instance state representation layer 20 is the input layer of the software system. Its core function is to construct the initial object instance state representation for long-time-series operations and provide a benchmark for subsequent particle state separation. It includes three core software modules: initial scene point cloud denoising and registration module 201, long-time-series object instance tracking module 202, and object particle state representation module 203.

[0035] The initial scene point cloud denoising and registration module 201 is used to input the initial scene multi-view RGBD point cloud data and camera intrinsic and extrinsic parameter matrices, perform statistical filtering denoising and multi-view point cloud registration, and output the scene global initial point cloud dataset. The object instance long-term tracking module 202 takes an initial point cloud dataset and an RGB image as input, performs instance segmentation, assigns a globally unique instance ID to each manipulated object, constructs the initial bounding box and state tracker for the object instance, and outputs an object point cloud dataset grouped by instance and an instance tracking handle. The object particle state representation module 203 takes as input a point cloud dataset of objects in sub-instances, constructs a standardized particle state representation for each point cloud data point, defines the spatial coordinates, feature attributes, timestamp, instance ID, globally unique identifier, and state update flag of the particle, and outputs an initial full set of particle states without frame-level data aggregation logic.

[0036] The particle state incremental coding layer 21 has the core function of separating static and dynamic particles, performing incremental coding on dynamic particles, and eliminating the spatiotemporal redundancy of long-term transmission. It includes three core software modules: particle state change detection module 210, dynamic particle incremental coding module 211, and particle coding verification module 212.

[0037] The particle state change detection module 210 is used to take into account the particle set collected at the current time and the global particle state set at the previous time. Based on K-nearest neighbor spatial distance matching, it detects the spatial position and attribute changes of particles, assigns a state update flag updateflag to each particle, marks static particles and dynamic particles, and outputs a particle set with state flags. The dynamic particle increment encoding module 211 is used to input a dynamic particle set with state markers. It only performs adaptive arithmetic entropy encoding on the change of particles. Static particles only perform full encoding once in the initial stage. It outputs encoded single-particle increment binary data packets. Each data packet has an independent parsing identifier and check bit, and can be decoded without relying on other data packets. The particle encoding verification module 212 is used to input incremental particle data packets, add independent CRC32 check bits, instance ID, timestamp, status flag and other packet header information, and output standardized incremental particle data packets that can be directly used for transmission.

[0038] The long-time adaptive transmission scheduling layer 22 has the core function of realizing frameless incremental particle streaming transmission, adapting to network fluctuations and task requirements of long-time operations. It includes three core software modules: frameless incremental transmission scheduling module 220, link adaptive scheduling module 221, and link status real-time monitoring module 222.

[0039] The frameless incremental transmission scheduling module 220 is used to input standardized incremental particle data packets. For dynamic particle data packets with flag=1, it performs frameless, real-time transmission of each particle. After the data packet is encoded, it is immediately pushed to the transmission link without waiting for other data packets in the same time slice. There is no frame-level packet assembly, buffering, or waiting logic. For static particle data packets, it is transmitted only once in the initial stage and is not transmitted repeatedly thereafter. It outputs a distributed transmission stream of incremental particle data packets. The link adaptive scheduling module 221 is used to input incremental particle data packets and real-time link status data. Based on the link packet loss rate and latency, it dynamically adjusts the transmission link and redundancy distribution strategy of the data packets, performs multi-link redundant transmission for high-priority dynamic particles, and adapts to long-term network fluctuation scenarios. The link status real-time monitoring module 222 is used to input real-time packet loss rate, delay, and out-of-order rate data of each transmission link, calculate the link quality score, and output to the incremental coding module and transmission scheduling module to dynamically adjust the coding compression ratio and transmission strategy.

[0040] The distributed decoding and particle state incremental update layer 23 has the core function of independently decoding incremental particle data packets and performing incremental updates on the particle state set of objects without full refresh, thus reducing the computational overhead of long-term operation. It includes three core software modules: multi-channel parallel decoding module 230, particle state incremental update module 231, and out-of-order data fusion module 232.

[0041] The multi-channel parallel decoding module 230 is used to input incremental particle data packets received from each link, perform independent parallel decoding, without waiting for other data packets to arrive, without processing according to timestamp order, supports decoding of out-of-order data packets, and outputs decoded dynamic particle incremental data. The particle state incremental update module 231 is used to input the decoded dynamic particle incremental data and the current global object particle state set. Based on the instance ID and the global unique identifier of the particle, it performs incremental updates on the corresponding dynamic particles in the global state set. Newly added particles are added to the state set, while static particles that have not changed remain unchanged. There is no need to refresh the state set completely. The updated real-time global particle state set is output. The out-of-order data fusion module 232 is used to input the incremental particle data and the global particle state set that arrive out of order, and to complete the fusion and update of the out-of-order data based on the particle timestamp and spatial matching relationship, and output the global particle state set with consistent time sequence.

[0042] The long-term state error correction and complete reconstruction layer 24 has the core function of eliminating accumulated errors in long-term transmission and realizing drift-free tracking and complete reconstruction of the object state. It includes three core software modules: correction window reference data management module 240, Chamfer-α distance global error correction module 241, and object point cloud complete reconstruction module 242.

[0043] The calibration window reference data management module 240 is used to input the global particle state set and the initial reference particle set, generate the calibration reference key frame particle set according to a fixed time window, cache it to local storage, and provide reference data for error correction. The Chamfer-α distance global error correction module 241 is used to input the current global particle state set and the correction window reference particle set, calculate the bidirectional matching error based on the Chamfer-α distance, perform global optimization correction on long-term accumulated position drift and particle loss, and output the corrected drift-free particle state set. The object point cloud complete reconstruction module 242 is used to input the corrected particle state set, perform spatial topology stitching, deduplication, and outlier filtering, and output a complete object reconstructed particle point cloud without cumulative drift.

[0044] The robot long-term task planning linkage layer 25 is the output layer of the software system. Its core function is to realize the real-time linkage between the reconstruction of particle point cloud and the robot long-term task planning, dynamically adjust the transmission and encoding strategies to form a closed loop, and includes three core software modules: particle data-task planning interface module 251, task stage identification module 252, and task feedback scheduling module 253.

[0045] The particle data-task planning interface module 251 is used to input real-time reconstructed particle point clouds, perform standardized data format conversion, and output object state representation data adapted to the robot's long-term operation planning algorithm. The task stage identification module 252 is used to input the task execution status and operation instructions output by the robot task planner, identify the current execution stage of a long-sequence task, and output the task stage identifier and the corresponding perception accuracy requirements. The task feedback scheduling module 253 is used to input task stage identifiers and perception accuracy requirements, generate update threshold adjustment instructions and priority adjustment instructions, and feed them back to the particle state change detection module and transmission scheduling module to dynamically adjust the particle change detection threshold, transmission priority and encoding strategy to adapt to the needs of different task stages.

[0046] See Figure 3 , Figure 3 This is a flowchart illustrating a transmission method for long-term operations of a robot, as provided in an embodiment of this application. The method described above... Figure 1 The structure of the robot shown and as Figure 2 The robot operates under the software architecture shown above, and the method described above is as follows: Figure 3 As shown, it includes the following steps: Step S301: Obtain the initial full 3D point cloud data of the long-term operation scenario and collect the time-series point cloud data stream of the long-term operation scenario in real time. For example, the acquisition of initial full 3D point cloud data for long-term operation scenarios may specifically include: By combining static fixed-point scanning and low-speed global cruise scanning with LiDAR, global 3D coordinate data of the work site ground, walls, large racks, and peripheral equipment is collected, generating a basic sparse global point cloud of the scene. Following a preset traversal path, the robotic arm equipped with an RGBD camera array completes 360° all-view surround acquisition, sequentially traversing radar blind spots such as the inside of the workstation, equipment gaps, and material stacking areas, acquiring local dense depth images and RGB texture information frame by frame. All acquisition processes are synchronously connected to pose data, and the spatial position of the acquisition viewpoint of each frame is recorded in real time, and pose trajectory files are stored.

[0047] Temporal alignment, spatial registration, and coordinate unification operations are performed on the pose trajectory file, basic sparse global point cloud, and RGB texture information. After performing layered fusion, blind spot completion, and dynamic noise removal operations, the initial scene structure data (i.e., the initial full 3D point cloud data) of the long-term task is retained.

[0048] For example, the time-series point cloud data stream for the above-mentioned real-time acquisition of long-term operation scenarios may specifically include: The control robot arm is equipped with an RGBD camera array to follow the target object and collect dense temporal point clouds of the target and contact surface in real time, capturing subtle posture and shape changes. By storing the pose trajectory file, the control robot arm corrects the dense temporal point cloud of the target and contact surface to obtain temporal point cloud data.

[0049] Step S302: Perform denoising and instance segmentation preprocessing on the acquired initial full 3D point cloud data to obtain an initial scene effective point cloud dataset grouped by object instances; encapsulate each point cloud data point in the initial scene effective point cloud dataset into a particle state unit with a state update tag to construct an initial full particle state set as a benchmark dataset for long-term state tracking; perform standardized particle encapsulation on the real-time acquired time-series point cloud data stream, match it with the global particle state set of the previous time step, detect particle state changes, assign a corresponding state update tag to each particle, and distinguish between static and dynamic particles; perform independent incremental encoding on the particle state units marked as dynamic particles to generate single-particle incremental binary data packets, and do not perform repeated encoding and transmission on static particles; perform fully distributed frameless streaming transmission on the generated single-particle incremental binary data packets; perform parallel decoding on the received single-particle incremental binary data packets to restore the dynamic particle incremental data, and perform incremental updates on the global particle state set based on the globally unique particle identifier to obtain the real-time global particle state set.

[0050] For example, the above encapsulation of each point cloud data point in the initial scene valid point cloud dataset into a particle state unit with a state update tag, and the construction of the initial full particle state set can specifically include: Assign a globally unique and time-series-unchanging particle identification identifier, a unique identifier of the object instance to which it belongs, and a globally synchronized timestamp to each point cloud data point; match the corresponding spatial coordinates and local geometric feature attributes for each point cloud data point to construct full state metadata for a single particle; assign an initial state update marker to all particles to generate particle state units with independent parsing capabilities; and summarize all particle state units to construct an initial full set of particle states.

[0051] For example, an implementation scheme for assigning a globally unique and long-term time-series unchanged particle identification identifier to each point cloud data point can be: For each particle identification identifier, traverse each point and generate an identifier (ID) according to the rules, such as B—TABLE_01—timestamp identifier—0001, B—CUP_01—timestamp identifier—0002, B—OPE_01—timestamp identifier—XXXX; where “TABLE_01, CUP_01, OPE_01” can be unique identifiers for object instances, and the timestamp identifier can include: year, month, day, hour, second, etc. The above 0001 is the particle's serial number.

[0052] The unique identifiers for the above-mentioned objects, such as TABLE_01, CUP_01, and OPE_01, can be composed of the object's English abbreviation plus a number. Of course, in other application scenarios, other methods can also be used to generate unique identifiers for object instances.

[0053] For example, the above method of matching corresponding spatial coordinates and local geometric feature attributes for each point cloud data point to construct single-particle full state metadata can specifically include: Extract the native 3D coordinates of each point cloud, retain the measured accuracy values, and use them as the core parameters for particle spatial positioning. Record the actual position of the particle in the global scene. Select neighborhood sampling points centered on the 3D coordinates of a single particle, and generate multi-dimensional geometric features to form the unique geometric attributes of the particle. Summarize the spatial coordinates and various local geometric features and combine them into an indivisible single-particle state metadata. The above single-particle state metadata is used to completely record the inherent spatial and morphological attributes of the particle.

[0054] For example, a specific implementation method for selecting neighborhood sampling points centered on the three-dimensional coordinates of a single particle, and calculating and generating multi-dimensional geometric features to constitute the unique geometric properties of the particle may include: Let the coordinates of the central particle be... , the relative offset vector of the neighborhood point, the relative offset is the Euclidean distance between the coordinates of the i-th example and the coordinates of the center particle; Perform an outer product operation on the k groups of relative offset vectors and the transpose of the k groups of relative offset vectors, sum the results of the outer products of all groups and take the arithmetic mean to obtain a 3rd order matrix C; the elements of the 3rd order symmetric matrix are pairwise equal on both sides of the main diagonal from the upper left to the lower right, and the transpose is completely identical to itself. k The number of effective neighborhood sampling points of the central particle.

[0055] Eigenvalue decomposition of a 3rd order symmetric matrix C yields three sets of eigenvalues ​​and three eigenvectors. The minimum eigenvector corresponding to the minimum value of the three sets of eigenvalues ​​is extracted, and the local normal vector is obtained by normalizing the minimum eigenvector. Calculate the surface curvature of the central particle, where the surface curvature is equal to the ratio of the minimum value to the sum of the three eigenvalues.

[0056] Calculate using the normal vector to fit the local optimal plane equation k Calculate the perpendicular distance from each neighboring point to the local normal vector. k The mean square error of each perpendicular distance is used as the roughness of the central particle. .

[0057] Get k Given a set of offset vectors from neighboring points, calculate the spatial angle between any two vectors in the set of offset vectors. The method for calculating the spatial angle between the above vectors is not limited in this application; for example, existing methods for calculating the vector angle can be used.

[0058] Based on all spatial angles corresponding to all vectors, a complete set of spatial angles is obtained. The mean angle, the standard deviation of the angle, and the local maximum angle are statistically analyzed. Averaging is performed over all effective regions to obtain the mean angle, the mean of the standard deviation of the angle, and the mean of the local maximum angle. A weighted sum of the mean angle, the mean of the standard deviation of the angle, and the mean of the local maximum angle is then calculated to obtain the edge feature coefficient. e ,in .

[0059] For example, the above-mentioned summarization and combination of spatial coordinates and various local geometric features into an indivisible single-particle state metadata may specifically include: combining the identifier, three-dimensional coordinates, local normal vector, surface index, roughness, and edge feature coefficients to obtain single-particle state metadata.

[0060] For example, the above assigns initial state update tags to all particles, generating particle state units with independent resolution capabilities; summarizing all particle state units to construct the initial full particle state set can specifically include: In the initial stage, all particles have no changes in position, shape, or affiliation, and are uniformly assigned the value Flag=0. All particles in the scene are traversed, and a flag field is bound to each one. The flag only represents the initial unupdated state. When the data is modified in the long time series, it is switched to Flag=1 to distinguish the particle data iteration state. The flag is bound to the corresponding particle's unique identifier to obtain a particle state unit. The particle state unit is a collection of single particle state metadata and state update flags. All particle state units are summarized to construct the initial full particle state set.

[0061] For example, the above process involves matching the global particle state set with the previous time step, detecting changes in particle state, assigning a corresponding state update flag to each particle, and distinguishing between static and dynamic particles. Specifically, this may include: Based on the globally unique identification identifier of the particle, the real-time encapsulated particles are aligned and matched one by one with the corresponding particles in the global particle state set of the previous moment. Calculate the spatial position changes and geometric feature changes between matched particles; when the change of a particle exceeds a preset change threshold, the particle is determined to be a dynamic particle and a first state update tag is assigned to it; when the change of a particle does not exceed the preset change threshold, the particle is determined to be a static particle and a second state update tag is assigned to it. Specific implementation methods may include: The change in spatial position is the change in the spatial position of the particle between the current moment and the previous moment. The aforementioned change in spatial position is the vector magnitude formed by the spatial coordinates of the particle at the current moment and the spatial coordinates at the previous moment. The geometric feature variation is a weighted sum of the curvature variation, roughness variation, edge feature variation, and the angle between the normal vectors in space. The changes in curvature, roughness, edge features, and the angle between the normal vectors are all changes between the particle at the current moment and the previous moment. For example, the change in curvature is the difference between the surface curvature of the particle at the current moment and the surface curvature at the previous moment, and the change in roughness is the difference between the roughness of the particle at the current moment and the roughness at the previous moment, and so on.

[0062] If the change in spatial position is greater than the position threshold or the change in geometric features is greater than the change in geometric features, the particle is determined to be a dynamic particle, and a first state update flag Flag=1 is assigned. If the change in spatial position is less than the position threshold and the change in geometric features is less than the change in geometric features, the particle is determined to be a static particle, and a second state update flag Flag=0 is assigned.

[0063] For example, the above-mentioned independent incremental encoding of particle state units labeled as dynamic particles to generate single-particle incremental binary data packets may specifically include: For dynamic particles marked with a first-state update flag, only the change in their state relative to the previous time step is extracted; context-independent adaptive arithmetic entropy encoding is performed on the extracted change data; independent check bits and header parsing identifiers are added to the encoded single-particle data to generate independently decodeable single-particle incremental binary data packets. Specifically, this may include: Extract the dynamic particles of the first state update label and obtain the set of single-particle changes. The set of single-particle changes can be: spatial position change, curvature change, roughness change, edge feature change, normal vector space angle change, and geometric feature change.

[0064] Initialize the encoding interval, read the regularized incremental data stream character by character, count the frequency of occurrence of each numerical symbol in real time, dynamically calculate the symbol probability distribution, divide the encoding interval according to the current symbol probability, and update the initial encoding interval; repeat this process until all incremental data is processed. After the iteration ends, select a binary value from the final encoding interval as the compressed binary code stream of the particle incremental data, configure the packet header parsing identifier and independent check bit for the binary code stream, and encapsulate it to obtain a single particle incremental binary data packet.

[0065] For example, the above-described fully distributed, frameless streaming of the generated single-particle incremental binary data packets may specifically include: Each single-particle incremental binary data packet is assigned a corresponding transmission link and transmission priority; after the data packet completes the priority and link allocation, it is immediately pushed to the corresponding transmission link for transmission. The above transmission process does not require waiting for other data packets in the same time slice; no frame-level packet assembly, frame-level buffering, or frame-level waiting operations are performed throughout the process, realizing packet-by-packet real-time streaming transmission of single particles and improving the real-time performance of the transmission.

[0066] For example, before detecting particle state changes and assigning a corresponding state update tag to each particle, the above method further includes: Based on the execution state data of the robot's long-term operation scenario, the current task execution stage is identified; based on the perception accuracy requirements of the task execution stage, the preset change threshold for particle state change detection is dynamically adjusted; based on the operation objectives of the task execution stage, the particle transmission priority weight of each object instance is dynamically adjusted.

[0067] For example, identifying the current task's execution stage based on the execution state data of a robot's long-term operation scenario can specifically include: identifying the current task's execution stage based on the mapping relationship between state data and execution stages. Of course, in other application scenarios, the current task's execution stage can also be determined by the interval in which the set of single-particle changes in the execution state data falls.

[0068] For example, the aforementioned method of dynamically adjusting the preset threshold for detecting particle state changes based on the perception accuracy requirements of the task execution phase includes: lowering the preset threshold when the current task is identified as being in a high-precision operation phase; and raising the preset threshold when the current task is identified as being in a low-precision movement phase. By dynamically adjusting the threshold, for example, lowering the preset threshold, the sensitivity of particle state change detection is improved; by raising the preset threshold, the number of dynamic particles identified is reduced, and transmission redundancy is decreased.

[0069] There are several ways to lower or raise the preset change threshold. For example, the preset change threshold can be lowered or raised by arithmetic progression, or by geometric progression (the same percentage is adjusted each time, for example, 5%).

[0070] For example, the above-mentioned dynamic adjustment of particle transmission priority weights for each object instance based on the operational objectives of the task execution phase includes: calculating the corresponding task relevance coefficient, motion relevance coefficient, and precision requirement coefficient based on the object's relevance to the current operational task, the object's motion state, and the operational precision requirements; calculating the transmission priority weights for each object instance based on the preset weights of each coefficient; and allocating corresponding transmission links and redundancy distribution strategies to the dynamic particle data packets of the corresponding object based on the transmission priority weights.

[0071] For example, the calculation of the corresponding task relevance coefficient, motion relevance coefficient, and accuracy requirement coefficient based on the relevance of the object to the current operation task, the object's motion state, and the operation accuracy requirements can specifically include: The task relevance coefficient can be calculated by weighting the category matching factor and the spatial overlap factor.

[0072] In one application scenario, the above category matching factor can be determined based on whether the object is in the whitelist associated with the current stage. For example, if the object belongs to the whitelist, the category matching factor = 1; if it does not belong to the whitelist, the category matching factor = 0.1 or 0.2. The above spatial overlap factor can be based on the area overlap ratio between the object's spatial region and the current operation interaction area.

[0073] The motion correlation coefficient can be calculated by: the proportion of dynamic particles within the object, the average value of the spatial positional changes of dynamic particles within the object, and the weighted average value of the characteristic changes of the dynamic particle set within the object.

[0074] The accuracy requirement coefficient can be calculated as a weighted sum of the task-preset accuracy factor, the stage correction factor, and the object attribute correction factor. The task-preset accuracy factor can be obtained based on the mapping relationship between the task and the accuracy factor, the stage correction factor can be obtained based on the mapping relationship between the stage and the correction factor, and the object attribute correction factor can be obtained based on the mapping relationship between the object and the correction factor.

[0075] For example, the transmission priority weight of each object instance can be a weighted sum of the task relevance coefficient, motion relevance coefficient, and accuracy requirement coefficient.

[0076] For example, the above-mentioned allocation of corresponding transmission links and redundancy distribution strategies for dynamic particle data packets of corresponding objects based on transmission priority weights may specifically include: Based on transmission priority weights, multi-level data priorities are defined, and dedicated transmission links are dynamically matched for dynamic particle incremental data packets of different priorities. Simultaneously, differentiated redundant distribution and retransmission strategies are configured. Specifically, high-speed links and multi-path redundant distribution are used for high-weight critical particle data to ensure transmission reliability and real-time performance, while conventional and simplified transmission strategies are used for medium- and low-weight particle data to conserve system bandwidth resources. Through dynamic iterative updates of transmission strategies linked to weights, adaptive, differentiated, and highly reliable scheduling and transmission of particle data streams in long-term robot operation scenarios are achieved.

[0077] For example, after obtaining the real-time global particle state set, the above method further includes: According to the preset long-term correction time window, the global state error correction process is triggered, which includes: loading the correction window reference particle set, calculating the bidirectional spatial matching error between the current real-time global particle state set and the reference particle set; performing global optimization correction on the current global particle state set based on the bidirectional spatial matching error; and updating the correction window reference particle set to the corrected global particle state set for error correction in the next correction window.

[0078] For example, the global state error correction process triggered according to the preset long-term correction time window may specifically include: Start the global timing calculation formula and continuously accumulate the job duration as the sum of the cumulative job duration of the current correction window and the time interval of a single (state) iteration.

[0079] When the duration of the continuously accumulated job exceeds the duration threshold, the global state error correction process is triggered, and the duration of the continuously accumulated job is reset to 0.

[0080] For example, the above-mentioned loading of the calibration window reference particle set and calculation of the bidirectional spatial matching error between the current real-time global particle state set and the reference particle set can specifically include: The forward spatial error is calculated as the norm of the vector between the real-time particle spatial coordinates and the reference particle spatial coordinates; the reverse spatial error is calculated as the norm of the vector between the reference particle spatial coordinates and the real-time particle spatial coordinates. The global feature vector is the arithmetic mean of all particle features, where: the feature of a single particle is equal to the average of the forward spatial error and the backward spatial error of a single particle.

[0081] For example, the above-mentioned global optimization correction is performed on the current global particle state set based on the bidirectional spatial matching error, eliminating the long-term accumulated position drift and particle loss errors; updating the reference particle set of the correction window to the corrected global particle state set for error correction in the next correction window may specifically include: Based on the bidirectional spatial matching error, global compensation is performed on the coordinates of all particles. The compensated example coordinates can be the difference between the real-time particle spatial coordinates and the global feature vector.

[0082] Based on the corrected particle neighborhood point cloud distribution, the local tangent plane of the particle is refitted, and the normal vector n is updated. Based on the updated spatial coordinates and normal vector, the curvature of the local surface of the particle is calculated a second time, and the curvature parameter is refreshed. Based on the discreteness of the spatial coordinates of the particle neighborhood after correction, the local surface concavity and convexity fluctuation characteristics are re-statistically analyzed, and the roughness parameter is updated. Combining the continuity of the particle spatial position and the gradient change characteristics of the neighborhood, the edge feature coefficient e is iteratively updated. After the global correction and accuracy verification are completed, the window reference particle set is iteratively replaced, and the optimized and corrected new particle state set is set as the error correction for the next correction window.

[0083] The specific calculation method for the above parameter correction can be found in step S302, and will not be repeated here.

[0084] Step S303: Complete long-term drift-free 3D point cloud reconstruction based on real-time global particle state set, and realize the planning and control of long-term operations based on the reconstructed 3D point cloud data.

[0085] For example, the above-mentioned long-term drift-free 3D point cloud reconstruction based on the real-time global particle state set may specifically include: constructing a three-dimensional point cloud mapping relationship using parameters within the real-time global particle state set; within a single window period, using the iteratively updated reference particle set as the fixed base of the scene, performing local addition updates to the fixed base of the scene for the real motion and deformation dynamic particles during the task process, and outputting a long-term drift-free 3D point cloud scene model to achieve 3D point cloud reconstruction.

[0086] Compared to existing systems / devices that use continuous frame-by-frame encoding and transmission, long-term operation scenarios suffer from problems such as high spatiotemporal redundancy, continuous accumulation of transmission delay, easy drift of state tracking errors, and excessive consumption of edge resources. The technical solution of this application first acquires the initial full 3D point cloud data and the real-time acquired temporal point cloud data stream of the robot's long-term operation scenario. Noise reduction and instance segmentation preprocessing are performed on the initial full 3D point cloud data to obtain an initial scene effective point cloud dataset grouped by object instances. Each point cloud data point in the initial scene effective point cloud dataset is encapsulated as a particle state unit with a state update tag, constructing the initial full particle state set as the benchmark dataset for long-term state tracking. Standardized particle encapsulation is performed on the real-time acquired temporal point cloud data stream, and particle-by-particle matching is performed with the global particle state set of the previous moment to detect particle state changes. A corresponding state update tag is assigned to each particle to distinguish between static and dynamic states. For static and dynamic particles, independent incremental encoding is performed on particle state units marked as dynamic particles to generate single-particle incremental binary data packets. Static particles are not repeatedly encoded or transmitted. The generated single-particle incremental binary data packets are transmitted in a fully distributed, frameless streaming manner without frame-level packet assembly, buffering, or waiting operations. The received single-particle incremental binary data packets are decoded in parallel to restore the incremental data of dynamic particles. Based on the globally unique identifier of the particles, the global particle state set is incrementally updated to obtain the real-time global particle state set. Based on the real-time global particle state set, long-term drift-free 3D point cloud reconstruction is completed. Based on the reconstructed 3D point cloud data, the planning and control of long-term operations are realized. Because the technical solution of this application abandons the traditional serial architecture of continuous full-frame transmission, and based on the state change characteristics of objects in long-term operations, it decomposes the point cloud into static particles and dynamic particles, and only performs incremental encoding and frameless streaming transmission on dynamic particles that undergo state changes, it eliminates a large amount of spatiotemporal redundant data in long-term transmission from the bottom layer. This helps to avoid the continuous accumulation of transmission delay from the root, significantly reduces bandwidth occupation and robot-side computing power overhead, and adapts to the requirements of long-term continuous operation. At the same time, based on the particle-by-particle incremental update mechanism with globally unique particle identifiers, it does not require full-frame refresh of the global object state. Combined with frameless streaming transmission, it achieves low-latency processing of "change is transmitted and arrival is updated", which helps to fundamentally solve the problem of object state tracking error accumulation and drift in long-term operations, and ensures the continuous and stable execution of long-term operation tasks. Therefore, the technical solution of this application has the advantages of reducing data transmission latency, reducing the efficiency of local computing power, improving data transmission efficiency, improving the allocation efficiency of local computing power, and improving the real-time performance of the robot.

[0087] See Figure 4 , Figure 4 This application provides a method for transmitting data during long-term operations of a robot, as described in Embodiment 1. Figure 1 The structure of the robot shown and as Figure 2 The robot operates under the software architecture shown above, and the method described above is as follows: Figure 4 As shown, it includes the following steps: Step S401: The initial scene's original depth image, RGB image, camera intrinsic parameter matrix, and extrinsic parameter matrix output by the robot's multi-view RGBD camera array are used to obtain a denoised cloud dataset based on these data. Specifically, this includes: For the original depth image of each viewpoint, the 2D pixel coordinates are converted into 3D spatial coordinates in the robot base coordinate system by combining the camera intrinsic and extrinsic parameter matrices. The above conversion can be carried out by using the back projection transformation formula of camera pixel coordinates to the world (robot base) coordinate system 3D coordinates. Of course, other conversion methods can also be used, which will not be elaborated here.

[0088] Multi-view registration is performed on the 3D point clouds after all viewpoint transformations. The Iterative Closest Point (ICP) algorithm is used to complete coarse and fine registration, and duplicate point clouds are removed to obtain the initial full point cloud dataset of the scene. Statistical filtering is performed on the cloud dataset to remove outliers and noise to obtain the denoised cloud dataset.

[0089] Step S402: Obtain the initial full set of particle states based on the denoised cloud dataset, global synchronization timestamp, and robot long-sequence operation task instructions; specifically including: Instance segmentation is performed on the denoised point cloud dataset. Based on the Mask R-CNN algorithm, the manipulated objects and background objects in the scene are identified. A globally unique instance ID is assigned to each object instance, resulting in a point cloud subset grouped by instance.

[0090] An initial state tracker is built for each object instance to record the object's initial bounding box, centroid position, and particle distribution characteristics, serving as a benchmark for long-term state tracking. For each point cloud data point of each object instance, a particle-based state representation encapsulation is performed, assigning a unique particle identifier to each particle and defining a normalized state unit for each particle. The structure consists of: the 3D spatial state of the example, the local geometric feature vector of the particle, the timestamp of the particle, the unique identifier of the particle's object instance, the unique identifier of the particle, and the set of state update markers of the particle. The initial full set of particle states is obtained by collecting all the state units of the particles. The local geometric feature vector of the particle may include: normal vector, curvature, roughness, etc.

[0091] Step S403: Preprocess the real-time point cloud dataset acquired at the current moment to obtain the current initial full particle state set. Based on the current initial full particle state set, the initial full particle state set, and the previous initial full particle state set, obtain the incremental data packet set with state tags, the current static particle set, and the dynamic particle set. Specifically, this includes: Based on the globally unique identifier of particles and object instances, a particle-by-particle matching is performed on the current real-time full set of particles and the full set of particle states at the previous time step. The change in the spatial position of the particles is calculated, and the change in spatial position is equal to the Euclidean norm of the particle spatial coordinate vectors at the previous time step and the particle spatial coordinate vectors at the previous time step.

[0092] Based on the relationship between spatial position change and change threshold, a state update label is assigned to each particle, specifically including: If the change in spatial position is greater than the change threshold (initial value is 0.5mm, but other values ​​are also possible), or if the particle's geometric characteristics change, it is determined to be a dynamic particle, and Flag=1; If the change in spatial position is less than the threshold for change and the particle's geometric characteristics remain unchanged, it is determined to be a static particle, and Flag=0.

[0093] For dynamic particles, incremental encoding is performed, using adaptive arithmetic entropy encoding only on the particle's change, generating a single-particle incremental binary data packet; for static particles, no encoding or transmission is performed; each incremental data packet is supplemented with an independent CRC32 checksum, a unique particle identifier, a unique object instance identifier, and a timestamp. The header information is used to ensure that each data packet can be decoded independently. Dynamic particles are grouped together to form a dynamic particle set, static particles are grouped together to form a static particle set, and incremental binary data packets are grouped together to form an incremental data packet set.

[0094] Step S404: Based on the incremental data packet set, the robot's long-term task execution status, and the real-time status data of each transmission link, obtain the incremental particle data packet stream for distributed multi-link transmission, and dynamically adjust the change threshold. Specifically, this includes: Based on the robot's long-term task execution state, the current task stage is identified, and the transmission priority weight of each object instance is calculated. Based on the priority weight, a transmission priority and link are assigned to each incremental data packet: dynamic particle data packets of high-weight objects are assigned to high-priority low-latency transmission links, and data packets of low-weight objects are assigned to ordinary links; for high-priority data packets with a priority weight greater than 0.8, multi-link redundancy distribution is performed, and the packets are simultaneously pushed to two independent physical links.

[0095] Once each incremental data packet is assigned a priority, it is immediately pushed to the corresponding transmission link without waiting for other data packets in the same time slice. There is no frame-level packet assembly, caching, or waiting logic. The packet loss rate, latency, and out-of-order rate data of each transmission link are collected in real time, and the link quality score is calculated. When the link packet loss rate exceeds the threshold of 3%, the number of redundant distribution links for high-priority data packets is dynamically increased, the change threshold is adjusted, and the transmission bandwidth usage is reduced.

[0096] Step S405: Based on the incremental particle data packet stream and the current initial full particle state set, obtain the updated real-time global particle state set. Specifically, this includes: Each received incremental data packet undergoes an independent CRC32 check. Data packets that pass the check proceed to the decoding stage. Data packets that fail the check are discarded. Selective retransmission is triggered only for high-priority data packets to avoid the accumulation of retransmission delays.

[0097] Independent parallel entropy decoding is performed on the verified data packets. The decoding process relies only on the data within a single data packet, without waiting for other data packets to arrive or processing them in timestamp order. Parallel decoding of out-of-order data packets is supported. After decoding, the incremental change data of dynamic particles can be restored. .

[0098] Based on the unique identifier of the particle and the unique identifier of the particle's object instance, incremental updates are performed on the global particle state set: for newly added dynamic particles, they are directly added to the global particle state set; for existing dynamic particles, the corresponding particle state in the global state set is updated with incrementally changed data; for static particles, the original state in the global state set remains unchanged, requiring no processing and no full refresh of the global state set.

[0099] For incremental data packets arriving out of order, the out-of-order data is fused and updated based on the particle timestamp and spatial matching relationship to obtain the updated real-time global particle state set, ensuring the temporal consistency of the global state set.

[0100] Step S406: Based on the real-time global particle state set and the calibration window reference particle set, obtain the calibrated global particle state set.

[0101] Step S407: Based on the corrected global particle state set and the instructions of the task stage, obtain the task feedback scheduling instruction, complete the long-term drift-free 3D point cloud reconstruction according to the corrected global particle state set and the task feedback scheduling instruction, and realize the planning and control of long-term operations based on the reconstructed 3D point cloud data.

[0102] The technical solution provided in this application, compared to the existing system / device's mechanism of continuous frame-by-frame encoding and transmission, suffers from problems such as high spatiotemporal redundancy, continuous accumulation of transmission delay, easy drift of state tracking errors, and excessive consumption of terminal resources in long-term operation scenarios. The technical solution of this application first acquires the initial full 3D point cloud data and the real-time acquired temporal point cloud data stream of the robot's long-term operation scenario. It performs denoising and instance segmentation preprocessing on the initial full 3D point cloud data to obtain an initial scene effective point cloud dataset grouped by object instances. Each point cloud data point in the initial scene effective point cloud dataset is encapsulated as a particle state unit with a state update tag, constructing an initial full particle state set as a benchmark dataset for long-term state tracking. The real-time acquired temporal point cloud data stream is subjected to standardized particle encapsulation and matched particle-by-particle with the global particle state set of the previous moment to detect particle state changes. A corresponding state update tag is assigned to each particle to distinguish between static and dynamic states. For static and dynamic particles, independent incremental encoding is performed on particle state units marked as dynamic particles to generate single-particle incremental binary data packets. Static particles are not repeatedly encoded or transmitted. The generated single-particle incremental binary data packets are transmitted in a fully distributed, frameless streaming manner without frame-level packet assembly, buffering, or waiting operations. The received single-particle incremental binary data packets are decoded in parallel to restore the incremental data of dynamic particles. Based on the globally unique identifier of the particles, the global particle state set is incrementally updated to obtain the real-time global particle state set. Based on the real-time global particle state set, long-term drift-free 3D point cloud reconstruction is completed. Based on the reconstructed 3D point cloud data, the planning and control of long-term operations are realized. Because the technical solution of this application abandons the traditional serial architecture of continuous full-frame transmission, and based on the state change characteristics of objects in long-term operations, it decomposes the point cloud into static particles and dynamic particles, and only performs incremental encoding and frameless streaming transmission on dynamic particles that undergo state changes, it eliminates a large amount of spatiotemporal redundant data in long-term transmission from the bottom layer. This helps to avoid the continuous accumulation of transmission delay from the root, significantly reduces bandwidth occupation and robot-side computing power overhead, and adapts to the requirements of long-term continuous operation. At the same time, based on the particle-by-particle incremental update mechanism with globally unique particle identifiers, it does not require full-frame refresh of the global object state. Combined with frameless streaming transmission, it achieves low-latency processing of "change is transmitted and arrival is updated", which helps to fundamentally solve the problem of object state tracking error accumulation and drift in long-term operations, and ensures the continuous and stable execution of long-term operation tasks. Therefore, the technical solution of this application has the advantages of reducing data transmission latency, reducing the efficiency of local computing power, improving data transmission efficiency, improving the allocation efficiency of local computing power, and improving the real-time performance of the robot.

[0103] Figure 5 This is a schematic diagram of the structure of a transmission system for long-term operation of a robot provided in an embodiment of this application; as shown... Figure 5 As shown, the above system includes: The acquisition unit 501 is used to acquire the initial full 3D point cloud data of the long time-series operation scenario and to collect the time-series point cloud data stream of the long time-series operation scenario in real time. Processing unit 502 is used to perform denoising and instance segmentation preprocessing on the acquired initial full 3D point cloud data to obtain an initial scene effective point cloud dataset grouped by object instances; encapsulate each point cloud data point in the initial scene effective point cloud dataset into a particle state unit with a state update tag to construct an initial full particle state set as a benchmark dataset for long-term state tracking; perform standardized particle encapsulation on the real-time acquired time-series point cloud data stream, match it with the global particle state set of the previous time step, detect particle state changes, assign a corresponding state update tag to each particle, and distinguish between static and dynamic particles; perform independent incremental encoding on the particle state units marked as dynamic particles to generate single-particle incremental binary data packets, and do not perform repeated encoding and transmission on static particles; perform fully distributed frameless streaming transmission on the generated single-particle incremental binary data packets; perform parallel decoding on the received single-particle incremental binary data packets to restore the dynamic particle incremental data, and perform incremental updates on the global particle state set based on the globally unique particle identifier to obtain the real-time global particle state set. The reconstruction unit 503 is used to complete long-term drift-free 3D point cloud reconstruction based on the real-time global particle state set, and to realize the planning and control of long-term operations based on the reconstructed 3D point cloud data.

[0104] For example, the processing unit 502 described above is specifically used to assign a globally unique and long-term time-series unchanged particle identification identifier, a unique identifier of the object instance to which it belongs, and a globally synchronized timestamp to each point cloud data point; match the corresponding spatial coordinates and local geometric feature attributes for each point cloud data point to construct single-particle full-scale state metadata; assign initial state update markers to all particles to generate particle state units with independent parsing capabilities; and summarize all particle state units to construct an initial full-scale particle state set.

[0105] For example, the processing unit 502 described above is specifically used to perform one-by-one alignment and matching between the real-time encapsulated particles and the corresponding particles in the global particle state set at the previous moment, based on the globally unique identification identifier of the particles; calculate the spatial position change and geometric feature change between the matched particles; when the change of a particle exceeds a preset change threshold, the particle is determined to be a dynamic particle and a first state update tag is assigned to the particle; when the change of a particle does not exceed the preset change threshold, the particle is determined to be a static particle and a second state update tag is assigned to the particle.

[0106] For example, the processing unit 502 described above is specifically used to extract only the change data of the dynamic particle marked as the first state update mark relative to the previous state; perform context-independent adaptive arithmetic entropy encoding on the extracted change data; add independent check bits and packet header parsing identifiers to the encoded single particle data, and generate a single particle incremental binary data packet that can be independently decoded.

[0107] For example, the processing unit 502 described above is specifically used to allocate a corresponding transmission link and transmission priority to each single-particle incremental binary data packet; after the data packet completes the priority and link allocation, it is immediately pushed to the corresponding transmission link for transmission.

[0108] For example, the processing unit 502 described above is also used to identify the current task execution stage based on the execution state data of the robot's long-term operation scenario; dynamically adjust the preset change threshold for particle state change detection based on the perception accuracy requirements of the task execution stage; and dynamically adjust the particle transmission priority weight of each object instance based on the operation target of the task execution stage.

[0109] For example, the processing unit 502 described above is specifically used to reduce the preset change threshold when the current task is identified as being in a high-precision operation stage, and to increase the preset change threshold when the current task is identified as being in a low-precision movement stage.

[0110] For example, the processing unit 502 described above is specifically used to calculate the corresponding task correlation coefficient, motion correlation coefficient, and accuracy requirement coefficient based on the correlation between the object and the current operation task, the object's motion state, and the operation accuracy requirement; to calculate the transmission priority weight of each object instance based on the preset weight of each coefficient; and to allocate the corresponding transmission link and redundancy distribution strategy to the dynamic particle data packet of the corresponding object based on the transmission priority weight.

[0111] For example, the processing unit 502 described above is also used to trigger a global state error correction process according to a preset long-term correction time window, specifically including: loading the correction window reference particle set, calculating the bidirectional spatial matching error between the current real-time global particle state set and the reference particle set; performing global optimization correction on the current global particle state set based on the bidirectional spatial matching error; and updating the correction window reference particle set to the corrected global particle state set for error correction in the next correction window.

[0112] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application. For example... Figure 6As shown, the electronic device 600 may include one or more of the following components: a processor 601, and a memory 602 coupled to the processor 601, wherein the memory 602 may store one or more computer programs, which may be configured to implement the methods described in the examples above when executed by one or more processors 601. Figure 6 The electronic device 600 shown can be a robot.

[0113] Processor 601 may include one or more processing cores. Processor 601 connects to various parts within the electronic device 600 using various interfaces and lines, and performs various functions and processes data of the electronic device 600 by running or executing instructions, programs, code sets, or instruction sets stored in memory 602, and by calling data stored in memory 602. Optionally, processor 601 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 601 may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. It is understood that the aforementioned modem may also not be integrated into processor 601, but may be implemented separately through a communication chip.

[0114] The memory 602 may include random access memory (RAM) or read-only memory (ROM). The memory 602 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 602 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method examples described above. The data storage area may also store data created during the use of the electronic device 600.

[0115] It is understood that the electronic device 600 may include more or fewer structural elements than those shown in the above block diagram, such as a power module, physical buttons, WiFi (Wireless Fidelity) module, speaker, Bluetooth module, sensor, etc., without limitation.

[0116] This application also provides a computer storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements some or all of the steps of any of the methods described in the above method embodiments.

[0117] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0118] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0122] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus RAM (DRRAM), etc., which are various media capable of storing program code.

[0123] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.

Claims

1. A transmission method for long-term operations of a robot, characterized in that, The method includes: Acquire the initial full 3D point cloud data of long-term operation scenarios and collect the time-series point cloud data stream of long-term operation scenarios in real time; The initial full 3D point cloud data is preprocessed with denoising and instance segmentation to obtain an initial scene valid point cloud dataset grouped by object instances. Each point cloud data point in the initial scene valid point cloud dataset is encapsulated into a particle state unit with a state update tag to construct an initial full particle state set as a benchmark dataset for long-term state tracking. The real-time acquired time-series point cloud data stream is standardized and encapsulated into particles, and matched with the global particle state set of the previous time step to detect particle state changes. A corresponding state update tag is assigned to each particle to distinguish between static and dynamic particles. Independent incremental encoding is performed on the particle state units marked as dynamic particles to generate single-particle incremental binary data packets. Static particles are not re-encoded or transmitted. The generated single-particle incremental binary data packets are transmitted in a fully distributed, frameless streaming manner. The received single-particle incremental binary data packets are decoded in parallel to restore the dynamic particle incremental data. Based on the globally unique particle identifier, the global particle state set is incrementally updated to obtain the real-time global particle state set. Long-term drift-free 3D point cloud reconstruction is achieved based on real-time global particle state sets, and long-term operation planning and control are realized based on the reconstructed 3D point cloud data.

2. The transmission method for long-term robot operations according to claim 1, characterized in that, The step of encapsulating each point cloud data point in the initial scene valid point cloud dataset into a particle state unit with a state update tag, and constructing the initial full particle state set specifically includes: Assign a globally unique and time-series-unchanging particle identification identifier, a unique identifier of the object instance to which it belongs, and a globally synchronized timestamp to each point cloud data point; match the corresponding spatial coordinates and local geometric feature attributes for each point cloud data point to construct full state metadata for a single particle; assign an initial state update marker to all particles to generate particle state units with independent parsing capabilities; and summarize all particle state units to construct an initial full set of particle states.

3. The transmission method for long-term robot operations according to claim 2, characterized in that, The process of performing a particle-by-particle matching with the global particle state set from the previous moment, detecting changes in particle states, assigning a corresponding state update flag to each particle, and distinguishing between static and dynamic particles specifically includes: Based on the globally unique identifier of the particles, the real-time encapsulated particles are aligned and matched one by one with the corresponding particles in the global particle state set of the previous moment; the spatial position change and geometric feature change between the matched particles are calculated; when the change of a particle exceeds a preset change threshold, the particle is determined to be a dynamic particle and a first state update tag is assigned to the particle; when the change of a particle does not exceed the preset change threshold, the particle is determined to be a static particle and a second state update tag is assigned to the particle.

4. The transmission method for long-term robot operations according to claim 3, characterized in that, The step of performing independent incremental encoding on the particle state units marked as dynamic particles to generate single-particle incremental binary data packets specifically includes: For dynamic particles marked as first-state update markers, only the change data relative to the previous state is extracted; context-independent adaptive arithmetic entropy encoding is performed on the extracted change data; independent check bits and header parsing identifiers are added to the encoded single-particle data to generate independently decodeable single-particle incremental binary data packets.

5. The transmission method for long-term robot operations according to claim 4, characterized in that, The process of performing fully distributed, frameless streaming on the generated single-particle incremental binary data packets specifically includes: Each single-particle incremental binary data packet is assigned a corresponding transmission link and transmission priority; after the data packet completes the priority and link allocation, it is immediately pushed to the corresponding transmission link for transmission.

6. The transmission method for long-term robot operations according to any one of claims 1-5, characterized in that, Before detecting particle state changes and assigning a corresponding state update tag to each particle, the method further includes: Based on the execution state data of the robot's long-term operation scenario, the current task execution stage is identified; based on the perception accuracy requirements of the task execution stage, the preset change threshold for particle state change detection is dynamically adjusted; based on the operation objectives of the task execution stage, the particle transmission priority weight of each object instance is dynamically adjusted.

7. The transmission method for long-term robot operations according to claim 6, characterized in that, The preset threshold for dynamically adjusting the particle state change detection based on the perception accuracy requirements of the task execution phase specifically includes: When the current task is identified as being in a high-precision operation phase, the preset change threshold is lowered; when the current task is identified as being in a low-precision movement phase, the preset change threshold is raised.

8. The transmission method for long-term robot operations according to claim 6, characterized in that, The dynamic adjustment of particle transport priority weights for each object instance based on the operational objectives of the task execution phase includes: Based on the relevance of the object to the current operation task, the object's motion state, and the operation accuracy requirements, the corresponding task relevance coefficient, motion relevance coefficient, and accuracy requirement coefficient are calculated respectively. Based on the preset weights of each coefficient, the transmission priority weight of each object instance is calculated. Based on the transmission priority weight, the corresponding transmission link and redundancy distribution strategy are assigned to the dynamic particle data packets of the corresponding object.

9. The transmission method for long-term robot operations according to any one of claims 1-5, characterized in that, After obtaining the real-time global particle state set, the method further includes: According to the preset long-term correction time window, the global state error correction process is triggered, which includes: loading the correction window reference particle set, calculating the bidirectional spatial matching error between the current real-time global particle state set and the reference particle set; performing global optimization correction on the current global particle state set based on the bidirectional spatial matching error; and updating the correction window reference particle set to the corrected global particle state set for error correction in the next correction window.

10. A transmission system for long-term operation of a robot, characterized in that, include: The acquisition unit is used to acquire the initial full 3D point cloud data of the long-term operation scenario and to collect the time-series point cloud data stream of the long-term operation scenario in real time. The processing unit performs denoising and instance segmentation preprocessing on the acquired initial full 3D point cloud data to obtain an initial scene effective point cloud dataset grouped by object instances. Each point cloud data point in the initial scene effective point cloud dataset is encapsulated into a particle state unit with a state update tag, constructing an initial full particle state set as a benchmark dataset for long-term state tracking. The unit performs standardized particle encapsulation on the real-time acquired temporal point cloud data stream, performs particle-by-particle matching with the previous global particle state set, detects particle state changes, assigns a corresponding state update tag to each particle, and distinguishes between static and dynamic particles. Independent incremental encoding is performed on particle state units marked as dynamic particles to generate single-particle incremental binary data packets, while static particles are not repeatedly encoded or transmitted. Fully distributed, frameless streaming transmission is performed on the generated single-particle incremental binary data packets. Parallel decoding is performed on the received single-particle incremental binary data packets to restore the dynamic particle incremental data. Incremental updates are performed on the global particle state set based on the globally unique particle identifier to obtain the real-time global particle state set. The reconstruction unit is used to complete long-term drift-free 3D point cloud reconstruction based on the real-time global particle state set, and to realize the planning and control of long-term operations based on the reconstructed 3D point cloud data.

Images