A robotic large-scale autonomous operating system and method

Abstract

A robot large-range autonomous operation system and method based on space-time memory graph and end-to-end action library, constructs a space-time memory graph based on visual SLAM, associates multi-modal feature retrieval and data, updates state timeline record and time sequence, discovers space-time memory query and operation opportunity, cascades end-to-end action library execution and control, online perception alignment, local re-planning and memory write-back, continuous learning of map and action library and cloud-edge collaboration, the application can simultaneously maintain geometric passability, semantic operability and time sequence auditability in unified map representation, and directly use the above information for online task generation and execution closed loop.

Description

Technical Field

[0001] This invention belongs to the field of robot autonomous movement and autonomous operation technology. Specifically, it relates to a large-scale autonomous operating system and method for robots based on spatiotemporal memory graphs and end-to-end motion libraries. Background Technology

[0002] Industrial robotic systems are typically divided into a navigation subsystem and an operation subsystem. The navigation side widely employs laser SLAM or visual SLAM to obtain 2D / 3D occupancy grids or sparse point cloud maps; the operation side relies on engineers writing finite state machines or scripts for each device and each workstation. This separation leads to the following specific problems: (1) Maps "show geometry, but not the task." Traditional SLAM outputs are mainly based on pose maps, landmarks, or grids, lacking stable encoding of operable geometry and semantics such as "knob center, handle axis, button pressable area, and dial reading window." As a result, the robot knows where it is in the corridor, but it is difficult to answer "which devices around it need to be covered in this inspection, what operations are allowed for each, and what is their historical status" in a unified data structure. After task switching, production line modification, or equipment relocation, it is often necessary to re-teach and modify the script, resulting in high maintenance costs.

[0003] (2) Semantic annotation is sparse and disconnected from the execution strategy. Some studies overlay category labels on point clouds or rasters, but fail to firmly bind multimodal retrieveable features (local appearance, key points, operational constraint parameters) with SLAM keyframes / subgraphs; the re-identification stage is easily affected by viewpoint, lighting, and local occlusion. More seriously, the interface between the semantic layer and the action layer is mostly a one-time "target ID + pose", lacking a mechanism to structurally write back the observation residuals, contact force features, and failure modes to the map during execution, making it difficult for the system to form an accumulable memory.

[0004] (3) Lack of systematic temporal state memory. Industrial operation and maintenance is not only concerned with "where the equipment is", but also with timeline information such as "last inspection reading, the most recent closing / opening time, and whether the indicator light is flashing abnormally". If only external MES / SCADA databases are relied upon without maintaining state factors bound to spatial location in the robot-side map, the robot will find it difficult to complete on-site verification and anomaly closure in scenarios with weak network, offline, or inconsistent cross-system data.

[0005] (4) Layered pipelines lead to error accumulation and response lag. If the interfaces for sensing output pose, planning output trajectory, and controlling tracking trajectory are not aligned with the uncertainty description, small calibration errors will be amplified in contact tasks. Purely heavy planning strategies frequently halt planning when the target or environment changes abruptly, resulting in insufficient real-time performance and smoothness.

[0006] (5) If the end-to-end learning strategy lacks a safety shell, the risk of industrial implementation is high. When the neural network directly outputs joint torque or speed, it may violate joint limits, speed limits, and contact force thresholds. If it is not combined with inverse kinematics feasibility checks, impedance / force control, and emergency backoff logic, it will be difficult to pass the on-site safety assessment.

[0007] For example, Chinese patent document CN120949665A discloses a multimodal embodied intelligent robot control method. This method simultaneously collects visual, auditory, tactile, force, and proprioceptive information, establishes a cross-modal spatiotemporal stamp alignment mechanism, and unifies data collected by different sensors to the same spatiotemporal reference through timestamp calibration and spatial coordinate system transformation. It performs hierarchical feature extraction on multimodal information and generates a unified multimodal scene state representation through a cross-modal attention fusion module. This unified multimodal scene state representation includes vector encoding of environmental geometric topology, semantic entities, physical attributes, and task intent. Based on this unified multimodal scene state representation, it utilizes an embodied intelligent agent framework for task reasoning and decision-making. A task semantic parser decomposes high-order task objectives into executable atomic action sequences, and a multimodal policy network is invoked to output control commands in real time. Based on these control commands, it executes multimodal motion planning and control. In the non-contact phase, visual servo trajectory tracking is used; in the contact phase, a tactile-visual fusion closed-loop controller is activated to dynamically correct motion parameters. Based on the robot's interactive data stream, an incremental policy distillation mechanism is used to optimize the multimodal policy network online. This scheme solves the spatiotemporal asynchrony problem of multimodal sensing information through a cross-modal spatiotemporal stamp alignment mechanism, avoiding the difficulties in cross-modal information association caused by time delay and spatial offset. However, it also suffers from the aforementioned technical problems.

[0008] For example, Chinese patent document CN121453057A discloses a method for creating and using a semantic map. This method involves a mobile robot collecting multimodal data from the environment and attaching corresponding spatial location information to the multimodal data, which includes at least image and sound data. The multimodal data is processed to extract corresponding multimodal features. An unsupervised clustering algorithm is used to cluster the multimodal features to form at least one data cluster. A pre-trained multimodal large model is used to generate natural language semantic labels for the data clusters. Based on the spatial location information of the data clusters and the natural language semantic labels, a multidimensional semantic map is constructed. In response to natural language queries, a search is performed within the multidimensional semantic map to provide feedback. While this approach improves the depth and naturalness of human-computer interaction and enhances environmental monitoring and analysis capabilities, it still cannot achieve an end-to-end closed loop within the same spatiotemporal memory map.

[0009] Therefore, there is a need for a large-scale autonomous operating system and method that integrates geometric, semantic, and temporal information within the same spatiotemporal memory graph framework, completes local closed loops with an embedded end-to-end action library, and writes back execution results in an auditable manner. Summary of the Invention

[0010] In view of the technical problems existing in the prior art, the present invention aims to provide a robot large-scale autonomous operating system and method based on spatiotemporal memory graph and end-to-end motion library.

[0011] The core idea of ​​this invention is as follows: on the pose graph or subgraph structure established by visual SLAM, "semantic entity nodes" carry multimodal descriptions and constraints of operable targets; a state timeline is maintained for each node; the robot retrieves, associates, and updates these nodes online; operable opportunities are discovered in the local spatial neighborhood by combining task rules; motion intentions are output from the action library template, and then implemented through classical control and safe projection; during execution, continuous alignment perception and local replanning are performed, and deviations and events are written back to the map, thereby forming a closed loop that can evolve over a long period of time.

[0012] To achieve the above objectives, the present invention adopts the following technical solution: According to one aspect of the present invention, a method for large-scale autonomous operation of a robot based on a spatiotemporal memory map and an end-to-end motion library is provided, characterized by comprising the following steps: S1: Construct a spatiotemporal memory map based on visual SLAM; S2: Retrieval of multimodal features and data; S3: Update the status timeline records and sequence; S4: Discover opportunities for querying and manipulating spatiotemporal memories; S5: End-to-end action library execution and control cascading; S6: Online sensing alignment, local replanning, and memory write-back; S7: Continuous learning of maps and action libraries and cloud-edge collaboration.

[0013] Furthermore, as described above, a method for large-scale autonomous robot operation based on a spatiotemporal memory graph and an end-to-end motion library is characterized in that the robot carries at least one visual sensor to move in the environment and estimate its own pose sequence {x}. t Construct a pose graph containing keyframe nodes, subgraph nodes, or scan nodes; perform one or more of instance segmentation, semantic segmentation, and object detection on the keyframe image to obtain operable entity categories, pixel masks, 2D bounding boxes, and optional 3D center / size estimates.

[0014] Preferably, the method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end action libraries, as described above, is characterized in that each operable entity is abstracted into a semantic entity node v.i It is then bound to the nearest keyframe, common keyframe set, or local subgraph. Node attributes include at least: entity category label, operable type, attitude cone or workspace subset that the end effector can approach, required tool or end fixture model, and safety-related parameters.

[0015] Furthermore, as described above, a method for large-scale autonomous robot operation based on spatiotemporal memory graphs and an end-to-end motion library is characterized by, for each v i Compute or learn multimodal embeddings, the embedding representation of the i-th semantic entity node is: Equation (1), In equation (1), The depth visual feature vector extracted from the keyframe cropping region or mask region can be obtained by a convolutional network, Vision Transformer or contrastive learning encoder; These are geometric and topological statistics; Φ(·) is a parameterizable vector that operates on semantics and constraints; Φ(·) can be a linear projection, a multilayer perceptron, or a small-scale fusion network used to map heterogeneous vectors to a unified metric space.

[0016] Preferably, the above-described method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end motion libraries is characterized in that, in step S2, when the robot re-enters a similar region, query features are extracted from the current frame. Obtained from the current depth / stereo measurement or SLAM map query. And according to the task context, the strategy vector is concatenated, and the query vector q is obtained through the same normalization and mapping. In the memory graph node set {v i Calculate and sort similarities using the following methods. Cosine similarity is used: Equation (2) Where ε is a small positive number to prevent division by zero. For Top-K candidate nodes, geometric consistency, temporal consistency and co-visibility relationship are used for scoring and fusion to obtain stable data association results. For targets that reappear after long-term occlusion, multiple hypothesis tracking is adopted, which converges to a single entity ID after sufficient observation.

[0017] Preferably, the method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end action libraries described above is characterized in that, in step S3, a time-ordered sequence of state factors is maintained for each semantic entity node, and the k-th record is written as: Equation (3), Where t kFor timestamps; y k It can be a discrete label or a continuous quantity; c k The confidence level is based on the detection score, multi-frame consistency, or sensor noise model. The operation task identifier that triggers this state change is empty if it is only a passive observation; the background maintenance thread performs sliding window fusion, abnormal jump detection, and reconciliation with the ledger issued by the main brain on the sequence.

[0018] Preferably, the above-described method for large-scale autonomous robot operation based on a spatiotemporal memory graph and an end-to-end motion library is characterized in that, in step S4, when the robot navigates to a certain work area or its neighborhood, it uses its current pose x t Using the query center, retrieve the set of candidate entities N within a radius R. R (x t Retrieval is accelerated through pose graph adjacency expansion, 3D voxel indexing, or KD-tree for e∈N. R (x t Read its latest status. Tasks in the task library and strategy context Whether an opportunity constitutes an actionable opportunity is determined by a rule engine or a learning-based scoring function: Equation (4) The Rule is a deterministic rule or the output of an anomaly detection model based on historical data; for O t Elements are sorted by urgency, path cost, and resource consumption to generate an executable queue; if O t If empty, the robot can continue patrolling or request the mainframe to issue a new task package.

[0019] The above-described method for large-scale autonomous robot operation based on a spatiotemporal memory graph and an end-to-end action library is characterized in that, in step S5, each type of operation in the action library corresponds to an action template, which can employ deep reinforcement learning strategies, behavior cloning, diffusion strategies, or multimodal Transformers, with the input observation O... t At least including: the current RGB or RGB-D image, robot joint angles q t End-effector pose estimation, target entity embedding, and optional force / torque. and short-term historical h {t-1} The strategy network outputs the reference terminal trajectory, target impedance parameter, or normalized action, denoted as intermediate quantity. Subsequently, through inverse kinematics, trajectory smoothing, model predictive control, or QP acceleration constraint solutions, the... Mapped to executable joint instructions, and mapped to set C via secure projection: Equation (5) Proj C (·) refers to: box constraint truncation of joint position / velocity / torque, force redistribution when the contact force estimate exceeds the threshold, and geometric projection of the restricted space; if there is no solution for IK (Inverse Kinematics) or the change in motion after projection exceeds the threshold, a degradation strategy is triggered.

[0020] Preferably, the above-described method for large-scale autonomous robot operation based on a spatiotemporal memory map and an end-to-end motion library is characterized in that, in step S6, during execution, perception alignment is performed at a frequency higher than the control cycle: the target key points and nodes in the current image are aligned... Alignment is performed, and residuals Δp and ΔR are estimated. If the residuals exceed a threshold, a short-term replanning of the end trajectory is performed instead of interrupting the entire task. Force anomalies trigger impedance parameter adjustments or trajectory regression. After each significant adjustment, Δ, contact statistics, and failure / success labels are compressed into an event summary and written back to the corresponding node or z. k The sequence is used for subsequent offline training and auditing.

[0021] Preferably, the above-described method for large-scale autonomous robot operation based on a spatiotemporal memory map and an end-to-end action library is characterized in that, in step S7, when a structural change in the environment is detected, a local SLAM update and node addition / deletion are triggered; nodes of dismantled equipment are marked as decommissioned but their historical timelines are retained in read-only mode; the action library supports offline incremental training and controlled online fine-tuning, provided that regulatory and safety assessments are met; the monitoring module statistically analyzes the success rate, time consumption, and anomaly rate of each template, and triggers data collection or rollback for low-performance templates. The robot periodically reports map summaries and key z-coordinates to the backend master computer via a standard message protocol. k The mainframe adjusts task priorities, inserts temporary tasks, or updates based on the global state of multiple machines. The local scheduler performs minimum perturbation rearrangement under hard real-time constraints.

[0022] According to another aspect of the present invention, a large-scale autonomous operating system for robots based on spatiotemporal memory graphs and end-to-end motion libraries is also provided, characterized in that it includes the following functional modules: (1) Spatiotemporal memory graph construction and maintenance module: responsible for driving visual SLAM, semantic parsing and node binding, maintaining consistent indexes of pose graph and semantic entity nodes, and performing node coordinate correction after map loop closure detection; (2) Multimodal retrieval and data association module: realizes similarity calculation, Top-K re-ranking and multiple hypothesis tracking; (3) Status timeline management module: implements writing, merging, trimming and exception marking; (4) Task opportunity discovery and scheduling module: implements judgment, priority sorting and queue management; (5) End-to-end action library and security control cascade module: Load each action template and execute IK and security projection links; (6) Online alignment and local replanning module: high-frequency residual estimation, short-time trajectory repair and force control parameter adjustment; (7) Brain communication and continuous learning module: message encoding and decoding, task reordering, model and map version management.

[0023] According to another aspect of the present invention, a large-scale autonomous operating system for robots based on spatiotemporal memory graphs and end-to-end motion libraries is provided for use in the large-scale autonomous operation method for robots based on spatiotemporal memory graphs and end-to-end motion libraries as described above.

[0024] According to the present invention, at least the following beneficial effects are achieved: (1) Unified representation: The same graph structure simultaneously carries geometric, semantic, operational constraints and temporal states, enabling the robot to make decisions on-site such as "knowing what it can do and what it has done" and reducing its dependence on fixed scripts.

[0025] (2) Robust retrieval and association: Multimodal embedding and geometric / temporal joint verification reduce the probability of false associations, which is beneficial for long-term operation in complex industrial scenarios.

[0026] (3) Execution safety: End-to-end intention is cascaded with traditional control and force control and explicit safety projection, taking into account both learning ability and industrial safety boundaries.

[0027] (4) Evolvable: Execution of write-back and incremental learning enables the map and strategy to improve with data; cloud-edge collaboration supports global optimization of multiple robots.

[0028] (5) Auditable: The status timeline provides a structured basis for fault tracing and compliance checks.

[0029] According to this invention, autonomous inspection and maintenance can be performed in a wide range of scenarios, including industrial plants, power distribution facilities, and warehousing and logistics. The technical scope encompasses: Visual Simultaneous Localization and Mapping (VisualSLAM) based on cameras and optional depth information; semantic parsing and map fusion for operable entities; timestamped device state memory; task-level opportunity discovery and queue scheduling; and cascaded execution of end-to-end operation strategies based on deep learning or reinforcement learning with traditional motion control and force control. Unlike navigation patents that only address "from point A to point B," this invention can simultaneously maintain geometrical drivability, semantic operability, and temporal auditability in a unified map representation, and directly use this information in the online task generation and execution closed loop. Attached Figure Description

[0030] Figure 1This is a schematic diagram illustrating the overall architecture and data closed loop of a robot's large-scale autonomous operating system according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the multimodal fields of semantic entity nodes in the spatiotemporal memory graph of a robot's large-scale autonomous operating system and method according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the spatiotemporal memory query and operation opportunity discovery process in a robot large-scale autonomous operating system and method according to an embodiment of the present invention; Figure 4 This diagram illustrates the end-to-end action templates, traditional control cascades, and safety constraints in a robot-wide autonomous operating system and method according to an embodiment of the present invention. Detailed Implementation

[0031] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Those skilled in the art should understand that the following embodiments are used to illustrate the technical concept of the present invention and do not constitute an undue limitation on the scope of protection of the present invention; equivalent substitutions can be made to the module division, parameter values, and algorithm implementation without departing from the spirit of the present invention.

[0032] Reference Figure 2-4 In one embodiment of the present invention, the method for large-scale autonomous operation of a robot based on a spatiotemporal memory graph and an end-to-end motion library provided by the present invention includes the following steps: S1: Spatiotemporal memory map construction based on visual SLAM. Specifically, the robot carries at least one visual sensor (monocular, binocular, or RGB-D), moves in the environment, and estimates its own pose sequence {x}. t Construct a pose graph containing keyframe nodes, subgraph nodes, or scan nodes; optionally, fuse IMU, wheel odometry, or GNSS to improve robustness. Perform one or more of instance segmentation, semantic segmentation, and object detection on the keyframe image to obtain operable entity categories, pixel masks, 2D bounding boxes, and optional 3D center / size estimates. Abstract each operable entity into a semantic entity node v. i This data is then bound to the nearest keyframe, a set of shared keyframes, or a local subgraph, thus forming an association edge of "geometric vertex - semantic node" in the graph structure. Node attributes include at least: entity category label, operable type (rotation, pressing, insertion, reading, etc.), attitude cone or workspace subset that allows the end effector to approach, required tool or end effector model, and safety-related parameters (maximum contact force, no-collision zone, etc.).

[0033] Preferably, to improve retrieval stability, for each v i Compute or learn multimodal embeddings. The embedding of the i-th semantic entity node can be represented as: (Equation 1) In equation (1), The depth visual feature vector extracted from the keyframe cropping region or mask region can be obtained by a convolutional network, Vision Transformer or contrastive learning encoder; These are geometric and topological statistics, such as the size of the 3D bounding box, the coordinates of key points (knob axis, handle grip point), the main direction of the local point cloud normal distribution, and the minimum distance to the nearest obstacle. Φ(·) is a parameterizable vector that operates on semantics and constraints (such as concatenating discrete categories with one-hot encoding and continuous constraints); Φ(·) can be a linear projection, a multilayer perceptron, or a small-scale fusion network, used to map heterogeneous vectors to a unified metric space, facilitating nearest neighbor retrieval and online updates.

[0034] S2: Multimodal feature retrieval and data association. When the robot re-enters a similar region, the query features are extracted from the current frame. Obtained from the current depth / stereo measurement or SLAM map query. (Scene geometric features), and concatenate the strategy vector (if any) according to the task context, then perform the same normalization and mapping to obtain the query vector q. In the memory graph node set {v i Calculate and sort similarities using the algorithm. Cosine similarity is preferred. (Equation 2) Here, ε is a small positive number to prevent division by zero. For Top-K candidate nodes, geometric consistency (reprojection error, 3D distance threshold), temporal consistency (smoothing constraint with the ID associated with the previous time step), and co-view relationship are further used for scoring and fusion to obtain stable data association results. For targets that reappear after long-term occlusion, multiple hypothesis tracking can be used, which converges to a single entity ID after sufficient observation, avoiding erroneous writing to the state timeline.

[0035] S3: State Timeline Recording and Time-Series Updates. For each semantic entity node, maintain a time-ordered sequence of state factors to express "when was what state observed, what is the confidence level, and what operation caused it." The k-th record is written as: (Equation 3) Where t k For timestamps (UTC or monotonic clock can be used); y k It can be a discrete label (such as open / close, light color category) or a continuous quantity (instrument reading, torque peak); c kThe confidence level is based on the detection score, multi-frame consistency, or sensor noise model. This is the identifier for the operation task that triggers the state change; it is empty if it is only a passive observation. The background maintenance thread performs sliding window fusion, abnormal jump detection (e.g., short-term sharp fluctuations in readings), and reconciliation with the ledger issued by the mainframe. The fused summary can improve upper-level scheduling efficiency and reduce storage redundancy.

[0036] S4: Spatiotemporal memory query and opportunity discovery. When the robot navigates to a work area or its neighborhood, it uses its current pose x t Using the query center, retrieve the set of candidate entities N within a radius R. R (x t Retrieval can be accelerated via pose graph adjacency expansion, 3D voxel indexing, or KD-tree. For e∈N R (x t Read its latest status. Tasks in the task library and strategy context (Including task priority, time limit, current security mode, team permissions, etc.), the rule engine or learning-based scoring function determines whether an actionable opportunity exists: (Equation 4) The rule can be a deterministic rule (e.g., "If the indicator light is red and the last inspection was more than T hours ago, a re-inspection is required"), or it can be the output of an anomaly detection model based on historical data. For O... t Elements are sorted by urgency, path cost, and resource consumption to generate an executable queue; if O t If empty, the robot can continue patrolling or request the mainframe to issue a new task package.

[0037] S5: End-to-end action library execution and control cascade. Each type of operation in the action library corresponds to an action template, which can internally employ deep reinforcement learning strategies, behavior cloning, diffusion strategies, or multimodal Transformers. Input observation O t At least including: the current RGB or RGB-D image, robot joint angles q t End-effector pose estimation, target entity embedding, and optional force / torque. and short-term historical h {t-1} The strategy network outputs the reference terminal trajectory, target impedance parameter, or normalized action, denoted as intermediate quantity. Subsequently, through inverse kinematics, trajectory smoothing, model predictive control, or QP acceleration constraint solutions, the... Mapped to executable joint instructions, and mapped to set C via secure projection: (Equation 5) Proj C(·) can be implemented as: box constraint truncation of joint position / velocity / torque, force redistribution when the contact force estimate exceeds the threshold, and geometric projection of the restricted space. If there is no solution for IK or the motion change after projection exceeds the threshold, a degradation strategy is triggered (reducing the workspace, changing the gripping point, or requesting manual intervention).

[0038] S6: Online perceptual alignment, local replanning, and memory write-back. During execution, perceptual alignment is performed at a frequency higher than the control cycle: matching target keypoints and nodes in the current image. Alignment is performed, and residuals Δp and ΔR are estimated. If the residuals exceed a threshold, short-term replanning is performed on the end trajectory (time window configurable from 50ms to 500ms), instead of interrupting the entire task. Force anomalies (impact, saturation, slippage) trigger impedance parameter adjustments or trajectory regression. After each significant adjustment, Δ, contact statistics, and failure / success labels are compressed into an event summary and written back to the corresponding node or z. k The sequence is used for subsequent offline training and auditing.

[0039] S7: Continuous learning and cloud-edge collaboration of the map and action library. When structural changes in the environment are detected (equipment relocation, installation of new equipment, removal of old equipment), local SLAM updates and node additions / deletions are triggered; removed equipment nodes can be marked as decommissioned but retain read-only historical timelines. The action library supports offline incremental training and controlled online fine-tuning, provided that regulatory and safety assessments are met; the monitoring module statistically analyzes the success rate, time consumption, and anomaly rate of each template, triggering data collection or rollback for low-performance templates. The robot periodically reports map summaries and key z-coefficients to the backend master computer via standard message protocols. k The mainframe adjusts task priorities, inserts temporary tasks, or updates based on the global state of multiple machines. The local scheduler performs minimum perturbation rearrangement under hard real-time constraints.

[0040] like Figure 1 As shown, in another embodiment of the present invention, a robot large-scale autonomous operating system based on a spatiotemporal memory graph and an end-to-end motion library is provided. This operating system can be used to implement the method described above and includes the following functional modules: (1) Spatiotemporal memory graph construction and maintenance module: responsible for driving visual SLAM, semantic parsing and node binding, maintaining consistent indexes of pose graph and semantic entity nodes, and performing node coordinate correction after map loop closure detection. (2) Multimodal retrieval and data association module: implements similarity calculation, Top-K reordering and multiple hypothesis tracking of equation (2). (3) State timeline management module: implements writing, fusion, pruning and anomaly marking of equation (3). (4) Task opportunity discovery and scheduling module: implements rule determination, priority sorting and queue management of equation (4). (5) End-to-end action library and security control cascade module: loads each action template and executes the IK and security projection link shown in equation (5). (6) Online alignment and local replanning module: high-frequency residual estimation, short-term trajectory repair and force control parameter adjustment. (7) Brain communication and continuous learning module: message encoding and decoding, task reordering, model and map version management. The above modules can be distributed across airborne computing units, edge industrial control computers, and cloud training services. This invention does not limit the specific hardware form.

[0041] The invention will now be described in detail through specific application scenarios.

[0042] Example 1 (Power Distribution Room Scene Mapping and Node Binding): An RGB-D camera and a wheeled mobile chassis were used. The robot travels along the inspection route, running feature-based visual SLAM to obtain keyframe sequences and pose maps. For each keyframe, an instance segmentation network is run to extract instance masks such as "knobs, buttons, indicator lights, and meter windows"; combined with depth map back projection, the 3D position and bounding box size of each instance's center point in the map coordinate system are obtained. Each instance is written into a semantic entity node v. i , bound to the common-view keyframe set, and calculated according to equation (1) (The mask area is pooled by the backbone network) (Statistics on center, dimensions, and normals) (Category and allowed one-hot operation and maximum contact force scalar). For the detection of the same physical device appearing repeatedly in multiple keyframes, instance merging is performed by fusing 3D distance and feature similarity to avoid duplicate nodes.

[0043] Example 2 (Online Retrieval and Data Association): The robot returns to a similar workstation, and the target region is extracted in the current frame. The candidate nodes in the map are projected onto the image using SLAM pose, and a candidate subset within the reprojection error threshold is calculated. Formula (2) is used to calculate this subset, with K=5. Candidates are sorted by weighted order of reprojection error in ascending order and similarity in descending order. If the entity ID has been associated in the previous cycle, a temporal smoothing term is introduced to penalize ID jumps; if the new ID is supported by M consecutive frames, the switching is completed. For the target re-emergence after occlusion, two hypotheses are retained until the observation support rates significantly diverge.

[0044] Example 3 (State Timeline Writing and Fusion): For the indicator light entity, the color classification result is collected every 2 seconds and written into equation (3), where y k c is one of {red, green, extinguish}. k This is the maximum probability for softmax classification. When three consecutive frames are of the same type and c k When the confidence level is >0.85, the data is merged into a single high-confidence record; when the color changes and is accompanied by button operation... When not empty, the event is marked as "Post-Operation State Change" for auditing by the master computer. For analog meters, y k The reading range is encoded, and an image hash is added for subsequent dispute verification.

[0045] Example 4 (Operation Opportunity Discovery): Let x t Given the robot's current pose, R = 8m. The retrieved value is N. R (x t It contains 12 entities. The task library stipulates that: the highest priority review task is generated for entities with "red indicator lights and a duration of more than 10 minutes"; and the meter reading task is generated for entities with "the difference between the meter reading and the previous reading exceeds the threshold". The Rule in Equation (4) is compiled from the above rules. The sorting π adopts a weighted summation: urgency weight 0.5, path cost weight 0.3, and resource conflict penalty 0.2. The output task queue is used for sequential execution by the action library.

[0046] Example 5 (Cascaded Execution of Knob Operations): Select the "Rotate Knob" template. Output the desired end-effector angular velocity and approach direction; the IK module solves for the joint velocity, and the QP layer limits the joint acceleration; the force control outer loop limits the peak tangential force. If the alignment residual exceeds 3mm, initiate a 200ms local replanning and realignment of the knob axis; if freewheeling is detected (torque is below the threshold but the angle remains unchanged), write back the failure label and request the mainframe to decide whether to continue.

[0047] Example 6 (Map Update and New Equipment Access): After the new electrical cabinet is in place, a manually controlled robot completes local SLAM by circling the cabinet once; semantic nodes are added and configured through CAD / BIM import or on-site tablet annotation. For the new operation type, 50 demonstration trajectories were first collected using teleoperation for initial training. After simulation and security verification, it is deployed as a grayscale template, and after the success rate meets the standard, it is officially approved.

[0048] Example 7 (Master Brain Collaboration and Weak Network Operation): Under normal circumstances, the latest z-values ​​of each entity are reported every 30 seconds. k Summary and queue length; local cache of summaries during weak network conditions, batch resending upon recovery. The mainframe issues a "typhoon warning mode". The rule automatically increases the frequency of outdoor cabinet inspections and reduces unnecessary operations.

[0049] The above embodiments can be combined to reproduce the technical effects of this invention in industrial settings. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for large-scale autonomous operation of a robot based on a spatiotemporal memory graph and an end-to-end motion library, characterized in that, Includes the following steps: S1: Construct a spatiotemporal memory map based on visual SLAM; S2: Retrieval of multimodal features and data; S3: Update the status timeline records and sequence; S4: Discover opportunities for querying and manipulating spatiotemporal memories; S5: End-to-end action library execution and control cascading; S6: Online sensing alignment, local replanning, and memory write-back; S7: Continuous learning of maps and action libraries and cloud-edge collaboration.

2. The method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end motion libraries as described in claim 1, characterized in that, The robot carries at least one visual sensor, moves in its environment, and estimates its own pose sequence {x}. t Construct a pose graph containing keyframe nodes, subgraph nodes, or scan nodes; perform one or more of instance segmentation, semantic segmentation, and object detection on the keyframe image to obtain operable entity categories, pixel masks, 2D bounding boxes, and optional 3D center / size estimates; abstract each operable entity into a semantic entity node v. i It is then bound to the nearest keyframe, common keyframe set, or local subgraph. Node attributes include at least: entity category label, operable type, attitude cone or workspace subset that the end effector can approach, required tool or end fixture model, and safety-related parameters.

3. The method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end motion libraries as described in claim 2, characterized in that, For each v i Compute or learn multimodal embeddings, the embedding representation of the i-th semantic entity node is: Equation (1), In equation (1), The depth visual feature vector extracted from the keyframe cropping region or mask region can be obtained by a convolutional network, Vision Transformer or contrastive learning encoder; These are geometric and topological statistics; A parameterizable vector for operational semantics and constraints; It can be a linear projection, a multilayer perceptron, or a small-scale fusion network, used to map heterogeneous vectors to a unified metric space.

4. A method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end motion libraries as described in any one of claims 1-3, characterized in that, In step S2, when the robot re-enters the similar region, the query features are extracted from the current frame. Obtained from the current depth / stereo measurement or SLAM map query. And according to the task context, the strategy vector is concatenated, and after the same normalization and mapping, the query vector q is obtained, which is then placed in the memory graph node set {v i The similarity scores are calculated and sorted using cosine similarity. Equation (2) Where ε is a small positive number to prevent division by zero. For Top-K candidate nodes, geometric consistency, temporal consistency and co-visibility relationship are used for scoring and fusion to obtain stable data association results. For targets that reappear after long-term occlusion, multiple hypothesis tracking is adopted, which converges to a single entity ID after sufficient observation.

5. A method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end motion libraries as described in any one of claims 1-3, characterized in that, In step S3, a time-ordered sequence of state factors is maintained for each semantic entity node, and the k-th record is written as: Equation (3), Where t k For timestamps; y k It can be a discrete label or a continuous quantity; c k The confidence level is based on the detection score, multi-frame consistency, or sensor noise model. The operation task identifier that triggers this state change is empty if it is only a passive observation; the background maintenance thread performs sliding window fusion, abnormal jump detection, and reconciliation with the ledger issued by the main brain on the sequence.

6. A method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end motion libraries as described in any one of claims 1-3, characterized in that, In step S4, when the robot navigates to a work area or its neighborhood, it uses its current pose x t Using the query center, retrieve the set of candidate entities N within a radius R. R (x t Retrieval is accelerated through pose graph adjacency expansion, 3D voxel indexing, or KD-tree. Read its latest status Tasks in the task library and strategy context Whether an opportunity constitutes an actionable opportunity is determined by a rule engine or a learning-based scoring function: Equation (4) The Rule is a deterministic rule or the output of an anomaly detection model based on historical data; for O t Elements are sorted by urgency, path cost, and resource consumption to generate an executable queue; if O t If empty, the robot can continue patrolling or request the mainframe to issue a new task package.

7. A method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end motion libraries as described in any one of claims 1-3, characterized in that, In step S5, each type of operation in the action library corresponds to an action template, which can employ deep reinforcement learning strategies, behavior cloning, diffusion strategies, or multimodal Transformers. The input observation O t At least including: the current RGB or RGB-D image, robot joint angles q t End-effector pose estimation, target entity embedding, and optional force / torque. and short-term historical h {t-1} The strategy network outputs the reference terminal trajectory, target impedance parameter, or normalized action, denoted as intermediate quantity. Subsequently, through inverse kinematics, trajectory smoothing, model predictive control, or QP acceleration constraint solutions, the... Mapped to executable joint instructions, and mapped to set C via secure projection: Equation (5) Proj C (·) refers to: box constraint truncation of joint position / velocity / torque, force redistribution when the contact force estimate exceeds the threshold, and geometric projection of the restricted space; if there is no solution for IK or the motion change after projection exceeds the threshold, a degradation strategy is triggered.

8. A method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end motion libraries as described in any one of claims 1-3, characterized in that, In step S6, perceptual alignment is performed at a frequency higher than the control cycle during execution: the target key points and nodes in the current image are aligned. Alignment, estimation of residuals Δp and ΔR; If the residual exceeds the threshold, a short-term replanning of the end trajectory is performed instead of interrupting the entire task. Force anomalies trigger impedance parameter adjustments or trajectory regression. After each significant adjustment, Δ, contact statistics, and failure / success labels are compressed into an event summary and written back to the corresponding node or z. k The sequence is used for subsequent offline training and auditing.

9. A method for large-scale autonomous robot operation based on spatiotemporal memory graphs and end-to-end motion libraries as described in any one of claims 1-3, characterized in that, In step S7, when a structural change in the environment is detected, local SLAM updates and node additions / deletions are triggered; nodes of dismantled equipment are marked as decommissioned but their historical timelines are retained in read-only mode; the action library supports offline incremental training and controlled online fine-tuning, provided that regulatory and safety assessments are met; the monitoring module calculates the success rate, time consumption, and anomaly rate of each template, triggers data collection or rollback for low-performance templates, and the robot periodically reports map summaries and key z-coefficients to the backend master computer via standard message protocols. k The mainframe adjusts task priorities, inserts temporary tasks, or updates based on the global state of multiple machines. The local scheduler performs minimum perturbation rearrangement under hard real-time constraints.

10. A large-scale autonomous operating system for robots based on spatiotemporal memory graphs and an end-to-end motion library, characterized in that, Includes the following functional modules: (1) Spatiotemporal memory graph construction and maintenance module: responsible for driving visual SLAM, semantic parsing and node binding, maintaining consistent indexes of pose graph and semantic entity nodes, and performing node coordinate correction after map loop closure detection; (2) Multimodal retrieval and data association module: realizes similarity calculation, Top-K re-ranking and multiple hypothesis tracking; (3) Status timeline management module: implements writing, merging, trimming and exception marking; (4) Task opportunity discovery and scheduling module: implements judgment, priority sorting and queue management; (5) End-to-end action library and security control cascade module: Load each action template and execute IK and security projection links; (6) Online alignment and local replanning module: high-frequency residual estimation, short-time trajectory repair and force control parameter adjustment; (7) Brain communication and continuous learning module: message encoding and decoding, task reordering, model and map version management.

Images