A full life cycle offline embodied intelligent control system and method and related device

Abstract

This invention discloses a full lifecycle offline embodied intelligent control system, method, and related equipment, and also discloses an embodied intelligent robot or embodied intelligent device, electronic device, medium, and product. Upon power-up, the system first establishes an external communication isolation boundary. It then performs signature, integrity, version rollback prevention, dependency, compatibility, validity period, space, and security policy verification on the offline imported capability package. Trusted offline updates are achieved through inactive slot installation, local self-testing, offline playback testing, regression testing, atomic switching of the control cycle boundary, and observation window rollback. During operation, the fusion state, task objective, world representation, security constraints, and ontology constraints are uniformly encoded into condition vectors. Skill graph planning, action generation, security constraint projection, ontology adaptation, and resource adaptive scheduling are performed sequentially. Combined with unified anomaly handling and local chained auditing, secure execution, trusted updates, and traceable operation and maintenance in an offline environment are achieved.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence, robot control, edge computing and information security, and in particular to an embodied intelligence control system, method and related equipment that do not rely on external networks and external inference services throughout their entire life cycle. It further relates to embodied intelligence robots or embodied intelligence devices, electronic devices, computer-readable storage media and computer program products implemented using the said system or method. Background Technology

[0002] An embodied intelligent robot is an AI robot that integrates artificial intelligence into a physical entity, enabling it to autonomously perceive and learn. When in operation, the embodied intelligent robot's control system perceives, understands, and makes decisions based on information from its surrounding environment, then drives the robot's own movements to perform corresponding tasks, such as motion tracking, sweeping, or object transport.

[0003] Currently, the inventors have discovered that the control schemes for embodied intelligent robots mainly fall into three categories: The first type is the cloud-edge-device collaborative solution. This type of solution typically places task understanding, strategy reasoning, and complex planning on external servers or remote inference nodes, while the robot's local devices mainly handle perception and data acquisition and low-level action execution. This type of solution can complete complex reasoning with high computing power under stable network conditions, but it is susceptible to a sharp drop in capability after link interruption, unstable closed-loop latency, and data leakage risks in isolated networks, no network, weak network, strong electromagnetic interference, or high-security scenarios.

[0004] The second category is the partial offline solution during operation. This type of solution allows the robot to complete perception, localization, navigation or control functions locally during the operation phase. However, during the deployment, update, data synchronization or anomaly recovery of capability packages, it still needs to rely on external servers or remote download mechanisms, making it difficult to achieve a true offline closed loop covering the entire process of capability deployment, switching, updating, operation and auditing.

[0005] The third type is the functional splicing solution. This type of solution integrates multiple external subsystems such as positioning, task planning, motion generation, execution control, and audit management into the same robot platform. Each link depends on different external systems to complete, and there is a lack of unified condition driving and safety boundaries between different subsystems. This results in problems such as inconsistent state expression between modules, fragmented resource scheduling, incomplete fault recovery chain, and high cost of cross-body migration.

[0006] In summary, current control systems for embodied intelligent robots rely on external computing devices to perform corresponding computational reasoning and / or capability update processes. They cannot operate offline without external devices and are susceptible to network interruptions and latency fluctuations in complex scenarios. There is a risk of a sudden drop in capabilities and data leakage after the task execution link is interrupted, which affects the continuous stability and reliability of robot performance. Summary of the Invention

[0007] This invention provides a full lifecycle offline embodied intelligent control system, method, and related equipment, and further provides robots, electronic devices, and computer-readable storage media implemented using the said system or method. Throughout the entire process of capability deployment, updates, task execution, and audit retention, it does not rely on external networks or inference devices, thereby achieving full lifecycle offline closed-loop control. This addresses the problems of current embodied intelligent robots being unable to provide full lifecycle offline operation capabilities and failing to meet the continuous stability and reliability requirements in scenarios without network access, with strong electromagnetic interference, or with high security.

[0008] Compared to existing technologies, this invention does not simply list offline deployment, offline control, offline updates, and offline auditing side by side. Instead, it establishes an integrated closed-loop link: "Establishment of external communication isolation boundary—Local trust root verification—Inactive slot installation—Dependency topology loading—Local self-test and offline playback testing—Atomic switching of control cycle boundary—Observation window rollback—Unified condition vector-driven skill graph planning and action generation—Rigid safety projection—Ontology abstraction adaptation—Runtime resource adaptive scheduling—Unified exception handling—Local chained auditing." This closed-loop link enables offline boundaries, trusted updates, task execution, security constraints, cross-platform reuse, and fault recovery to cooperate within the same control system, thereby solving the problems of incomplete offline boundaries, fragmented control links, unrecoverable updates, lack of rigid constraints on execution security, and untraceable anomalies in existing solutions.

[0009] In a first aspect, embodiments of the present invention provide a full lifecycle offline embodied intelligent control system, applied to embodied intelligent agents, including but not limited to robots, mobile platforms, robotic arms, composite robots, or other embodied intelligent devices with sensing, decision-making, and execution capabilities. The control system includes an external communication isolation control module, a local root of trust and monotonic counting module, an offline capability package governance module, an offline control execution module, and a security constraint and ontology adaptation module; in some embodiments, it further includes at least one of a runtime resource adaptive scheduling module, a unified exception handling module, and a local auditing module.

[0010] The external communication isolation control module establishes offline boundaries during startup or before loading runtime resources. The local trust root and monotonic counting module provides the foundation for local trusted verification, version rollback prevention, slot switching, and audit rollback prevention for the offline capability package governance module. The offline capability package governance module performs trusted verification, inactive slot installation, local verification, active slot switching, and rollback on offline capability packages imported via offline media. The offline control execution module loads capability resources from active slots or local storage and generates candidate control actions based on observation data, task objectives, world representation, and constraint information. The security constraint and ontology adaptation module performs security processing on the candidate control actions and maps them to underlying control instructions. The runtime resource adaptive scheduling module, unified exception handling module, and local auditing module are used to perform runtime configuration adjustments, exception hierarchical handling, and local chained auditing as needed.

[0011] In one embodiment, the core control layer of the control system includes: The multimodal perception and preprocessing module is used to acquire multimodal observation data of the robot body and perform preprocessing to obtain multimodal perception data; the preprocessing includes time synchronization, coordinate registration, missing data repair, noise estimation and outlier removal; The Fusion State Estimation and World Representation module is used to fuse multimodal perception data to obtain the robot's fused state, and to construct the environment based on the multimodal perception data to obtain a world representation that includes occupied space, semantic objects, and topological relationships. The skill graph planning module is used to search for the optimal path in a preset skill graph based on the task objective, fusion state, and world representation, and obtain the target skill path. The uncertainty assessment and action sequence generation module is used to adaptively adjust the sampling noise and denoising step size in the robot action generation process based on the uncertainty of the fused state and world representation, so as to generate candidate action sequences corresponding to the target skill path. The ontology abstraction and adaptation module is used to map the control commands of the robot ontology based on the candidate action sequence, so as to obtain the underlying control commands of the robot ontology. The underlying control commands are used to drive the robot to perform corresponding actions to complete the task objectives.

[0012] In one embodiment, the core control layer of the control system further includes: The task understanding and condition vector construction module is used to determine the task objective. It integrates the fusion state, world representation, task objective, safety constraint representation and robot body constraint representation into a unified condition vector to eliminate control deviations caused by inconsistencies in input representations between modules. The skill graph planning module is used to search for the optimal path in a preset skill graph based on condition vectors to obtain the target skill path.

[0013] In one embodiment, the core control layer of the control system further includes: The safety constraint execution module is used to perform projection solving on the candidate actions in the candidate action sequence to satisfy the preset safety constraints in order to obtain safe executable actions, and to trigger the execution of the preset safety handling strategy when no safe executable action is found. The preset safety handling strategy includes at least one of emergency stop, safety action rollback or safety degradation. The body abstraction and adaptation module is used to map safe and executable actions to low-level control instructions of the robot body, and to perform position control, speed control, force control or force-position hybrid control according to the task type during the instruction generation process.

[0014] In one embodiment, the offline capability package governance module is further configured to: Before the offline capability package is put into use, offline cross-platform consistency verification and cross-ontology adaptability verification are performed on the offline capability package. After both cross-platform consistency verification and cross-ontology adaptability verification are passed, the relevant modules in the core control layer are triggered to perform the robot's offline control tasks based on the offline capability package in the active slot.

[0015] In one embodiment, the unified exception handling module is specifically used for: It receives abnormal information reported by other functional modules, and performs global state switching and abnormal level handling based on the received abnormal information, preset abnormal classification and state transition rules.

[0016] Secondly, a full lifecycle offline embodied intelligent control method is provided, including: After the control system is powered on, the external communication isolation control module is activated to close, shield, or logically isolate the external communication path of the control system, and triggers the subsequent control chain to start when the offline boundary is verified; reads the local trust root and slot information; receives candidate offline capability packages through the offline medium, and performs signature verification, integrity verification, version rollback prevention verification, dependency verification, compatibility verification, validity period verification, space verification, and security policy verification on the offline capability packages based on the local trust root; after the verification is passed, the offline capability package is installed in the inactive slot, the component is loaded according to the dependency topology sorting, and local self-test and offline replay test are performed on the inactive slot. Trial and regression testing; when the offline capability package meets the preset activation conditions, perform atomic switching at the control cycle boundary or safe stop point to switch the inactive slot to the active slot, and monitor the system operation parameters in the observation window to automatically roll back to the previous stable active slot when the system operation parameters trigger the rollback condition; load the model file, map resources, strategy file, configuration file and body adaptation parameters required for the current operation from the active slot to complete the runtime warm-up; during operation, acquire the multimodal observation data of the robot body, perform time synchronization, coordinate registration, missing data repair, noise estimation, anomaly gating and weight calculation, obtain the fusion state and update. World representation; construct a unified condition vector based on the fusion state, task objective, world representation, safety constraint representation, and robot body constraint representation; perform path search in the preset skill graph based on the condition vector to obtain the target skill path; determine the comprehensive uncertainty index based on the uncertainty of the fusion state, the uncertainty of the world representation, the intensity of dynamic environment changes, and the confidence of localization and recognition, and generate candidate action sequences corresponding to the target skill path accordingly; perform projection solving on the current candidate action in the candidate action sequence to satisfy the preset safety constraints, obtain safe executable actions, and trigger emergency stop, safe action rollback, or safety degradation when no feasible solution exists. One less type of handling; mapping safe and executable actions to the robot's underlying control commands, and performing position control, speed control, force control, or force-position hybrid control according to the task type; dynamically adjusting the operating configuration of the control system based on time delay, power consumption, temperature rise, memory usage, and minimum safe control frequency constraints; receiving abnormal information reported by each functional module in the control system and performing global state switching and abnormal level handling according to preset abnormal classification and state transition rules, while generating chain audit records for deployment, import, verification, installation, loading, activation, operation, alarm, degradation, emergency stop, manual takeover, rollback, and decommissioning events and storing them locally.

[0017] Thirdly, a full-lifecycle offline embodied intelligent robot is provided, characterized by including a robot body and the aforementioned full-lifecycle offline embodied intelligent control system.

[0018] Fourthly, embodiments of the present invention provide an electronic device, including a processor, a memory, a sensor interface, an actuator interface, and an offline media interface. The memory stores program instructions executable by the processor, and when the program instructions are executed, the above-mentioned full lifecycle offline embodied intelligent control method is implemented.

[0019] Fifthly, embodiments of the present invention provide a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described full lifecycle offline embodied intelligent control method.

[0020] The aforementioned systems, methods, and related devices establish an external communication isolation boundary during the startup phase and complete at least one of the following within the offline boundary: trusted verification of capability resources, slot management, local control execution, security constraint processing, ontology adaptation, anomaly handling, and audit retention. This enables embodied intelligent agents to maintain local closed-loop control in environments without external networks or with limited networks, and provides rollback and traceability capabilities in the event of capability update failures or operational anomalies. This improves operational continuity, security, and operational traceability in offline scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is an architecture diagram of a full lifecycle offline embodied intelligent control system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an embodied intelligent robot according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating a full lifecycle offline embodied intelligent control method according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the functional implementation process of the offline capability package management module in one embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the specific process of obtaining the target skill path through skill graph search in one embodiment of the present invention; Figure 6 This is a schematic diagram of the generation process of a candidate action sequence in one embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0027] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0028] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0029] For ease of understanding, the relevant terms in the embodiments of this application are defined as follows: Full lifecycle offline refers to the entire process from initial system deployment, capability loading, task execution, version updates, anomaly rollback, audit retention, to decommissioning, during which the control system does not rely on external servers, external inference services, external knowledge bases, or external application interfaces to obtain task decision results, model inference results, or execution permissions. Capabilities obtained through offline media, locally pre-built resources, or historically installed resources can enter the controlled area within the system after passing local trusted verification. The system can retain the internal communication paths of the embodied intelligent agent, including sensor bus, actuator bus, internal message bus, and offline media interface. In full lifecycle offline mode, communication paths to external networks or external online services are closed, shielded, physically isolated, logically isolated, or unreachable.

[0030] A capability package is a set of installable objects that contain at least model files, policy files, configuration files, map resources, dependency lists, security metadata, version identifiers, and signature information, used to provide or extend a certain function / capability to an embodied intelligent robot system.

[0031] Offline capability packages are capability packages imported via offline media, used to update or add certain functions / capabilities of the robot.

[0032] Capability resources can come from offline capability packages imported from offline media, or from local resources in the control system's factory presets, historical installations, or activated active slots. Unless otherwise specified, capability resources in this manual include model files, policy files, configuration files, map resources, interface description files, ontology adaptation parameters, security policy parameters, test vectors, and combinations thereof.

[0033] A local root of trust refers to at least one of the following stored in a local protected area: root public key, integrity verification benchmark, version count, audit count, slot identifier, resource space identifier, and equivalent rollback prevention status. It is used to support capability resource verification, version comparison, enable / disable switching, rollback determination, and audit rollback prevention. Local protected areas include, but are not limited to, protected hardware areas, trusted execution environments, independent secure elements, one-time programmable storage, fuse-protected or access-controlled non-volatile storage areas.

[0034] An active slot refers to a capability package slot in a robot that is currently loaded and running, providing corresponding capabilities (such as navigation, obstacle avoidance, voice interaction, image recognition, etc.). Each active slot corresponds to an activated capability package instance, which consumes the robot's computing, storage, and communication resources and directly participates in the robot's real-time task processing.

[0035] Inactive slots refer to candidate slots in a control system used for installing, verifying, or authenticating offline capability packages or new versions of capability packages. Capability packages installed in inactive slots do not participate in the current closed-loop control and can occupy storage or temporary computing resources under controlled conditions to complete verification, installation, loading, testing, authentication, or configuration without affecting the stable operation of the currently active slots.

[0036] The offline boundary is the logical or physical isolation line between the system's internal trusted operating environment and all external entities (including the cloud, servers, other network devices, wireless communication signals, etc.). Within this offline boundary, the system independently completes the entire lifecycle tasks from deployment, operation, update, auditing to rollback, relying on its own hardware, local root of trust, local storage, and offline capability packages. Any information, instructions, or code outside the offline boundary is not actively received or responded to by the system unless it is transmitted through offline media and verified by the local root of trust in multiple dimensions, and is only allowed to enter the inactive slots within the boundary under strictly controlled conditions.

[0037] The high-level control chain is the top-level logical control link in the decision-making system of an embodied intelligent agent, serving as the brain and decision-making center of the agent. Its core function is to complete overall planning, directional decisions, and task breakdown and scheduling around the task objectives, without directly interfering with the robot's specific joint movements, real-time obstacle avoidance, and other low-level execution details.

[0038] Offline media refers to physical storage media that do not rely on a network environment and can independently store / transmit data, including but not limited to USB flash drives, portable hard drives, memory cards, optical discs, as well as magnetic tapes / tape libraries, bare disks / offline hard drives, and dedicated archiving media (such as M-DISC).

[0039] Atomic switching refers to the slot switching of an offline capability package or its capability components at the control cycle boundary or safe stop point without intermediate visible state. The switching process does not change the control commands already issued in the current control cycle and is used to reduce the impact of control jitter, state inconsistency or switching failure on the current closed-loop control.

[0040] Unified control representation refers to a unified data representation used to transfer data between task understanding, skill planning, action generation, safety handling, and ontology adaptation. Condition vectors are one implementation of unified control representation. Unless otherwise specified, the unified control representation in this specification can be implemented by condition vectors, structured state representation, rule-encoded representation, or a combination thereof.

[0041] Chained auditing refers to a form of audit record where each audit record contains a summary of the previous record, and the audit record is bound by a hash chain to prevent tampering and rollback.

[0042] An embodied intelligent agent refers to an intelligent entity that possesses sensing, decision-making, and execution capabilities and can interact with its physical environment through its own body. This includes, but is not limited to, industrial robots, service robots, mobile robots, robotic arms, composite robots, special-purpose robots, and other embodied intelligent devices with sensors, actuators, and local controllers. An embodied intelligent robot is a typical implementation of an embodied intelligent agent. Unless otherwise specified, any references to robots or robot bodies in this specification shall be understood as specific implementations of embodied intelligent agents or embodied intelligent agent bodies.

[0043] It is important to understand that existing control schemes for embodied intelligent robots mainly fall into three categories: cloud-edge-device collaborative control schemes, partial offline control schemes during operation, and function splicing schemes. The inventors have discovered the following shortcomings in existing control schemes: (1) During the initial deployment, capability loading, policy update, anomaly recovery, log auditing and retirement retention stages, the system still relies on external servers, remote inference services or external knowledge bases, resulting in the offline boundary not being closed, making it difficult to meet the requirements for continuous and stable operation in isolated networks, no network, weak network, strong electromagnetic interference and high confidentiality scenarios. (2) Existing control systems typically lack local trusted roots for offline updates, version rollback prevention, dependency resolution, inactive slot installation, atomic switching, and automatic rollback mechanisms. If the model package, map package, or strategy package is damaged, has incomplete dependencies, or fails to switch during the import process, it can easily cause the system to fail to start, the control chain to be interrupted, or the version to be unrecoverable. (3) The state expression, update timing and triggering conditions among perception, state estimation, task understanding, planning, action generation, execution control and safety constraints in the existing control system are not consistent, which leads to information fragmentation, timing drift and control link inconsistency between different modules, thus affecting the stability and real-time performance of robot-side closed-loop control. (4) There is a general lack of rigid safety projection mechanism between the action output of the existing control system and the underlying executable control. Although the high-level planning is feasible at the task level, it may still violate the collision distance, contact force threshold, speed limit, driving torque limit, joint limit or restricted area constraints at the execution level. There is a problem that the planning is feasible but the execution is dangerous. (5) Existing control systems typically do not incorporate the robot’s computing power, temperature rise, power consumption, memory usage, execution latency and minimum safe control frequency into a unified schedule, making it difficult to maintain reliable local closed-loop operation for a long time under limited hardware conditions; (6) When the existing control system is updated, the high-level skill logic of its capability package is highly coupled with different robot bodies. Skill reuse usually relies on a lot of manual reconfiguration. There is a lack of a unified body abstraction layer, interface description and cross-platform consistency verification mechanism, which makes it difficult to migrate across computing platforms, operating systems, chip architectures and robot bodies. (7) Existing control systems lack a unified anomaly classification, state machine handling logic and traceable audit mechanism for special situations such as sensor failure, time synchronization anomaly, state divergence, planning failure, action generation timeout, actuator failure, thermal runaway, audit chain anomaly and update failure, making it difficult to achieve self-recovery and responsibility tracing in offline scenarios.

[0044] To address the aforementioned issues, this invention provides a full lifecycle offline embodied intelligent control system, method, and related equipment. The system constructs an offline boundary and trusted governance layer by including an external communication isolation control module, a local root of trust and monotonic counting module, and an offline capability package governance module. It then constructs a core control layer by including a multimodal perception and preprocessing module, a fusion state estimation and world representation module, a task understanding and condition vector construction module, a skill graph planning module, an uncertainty assessment and action sequence generation module, a safety constraint execution module, and an ontology abstraction and adaptation module. Finally, it constructs a runtime resource adaptive scheduling module, a unified exception handling module, and a local auditing module to build a runtime assurance layer. The core control layer's sensor interface and ontology control interface connect to sensors on the embodied intelligent entity to receive sensor signals and drive actuators on the embodied intelligent entity. The control system architecture is as follows: Figure 1As shown. Furthermore, the above modules do not exist independently and in parallel, but rather form a closed-loop collaborative link: the external communication isolation control module first establishes the offline boundary, the local trust root and monotonic counting module provide a reliable verification and version rollback prevention foundation for the offline capability package governance module, the offline capability package governance module completes inactive slot installation, dependency topology loading, local self-check, offline playback test, regression test, atomic switching and observation window rollback; the offline capability package in the active slot, during task execution, sequentially passes through multimodal perception and preprocessing, fusion state estimation and world representation, task understanding and condition vector construction, skill graph planning, uncertainty assessment and action sequence generation, safety constraint execution and ontology abstraction adaptation, and then drives the robot body to complete the offline control task; the runtime resource adaptive scheduling module dynamically adjusts the running configuration during the above process; the unified exception handling module performs global state switching and exception hierarchical handling on the exception information reported by each module, and the local audit module forms a chain audit record for the entire process events to prevent rollback. By employing a three-layer architecture—security and governance layer, core control layer, and scheduling and execution layer—to work collaboratively, a control system is provided that does not rely on external networks throughout the entire process of initial system deployment, task execution, capability switching, version updates, exception rollback, and audit tracing. This enables offline control of the robot's entire lifecycle, thereby solving the aforementioned technical problems and achieving the following technical effects: 1. By implementing external communication path isolation and offline boundary determination mechanisms through an external communication isolation control module, the system achieves complete closure of the offline boundary. This extends the offline requirements from the operational phase to cover the entire process of system deployment, loading, updating, rollback, auditing, and decommissioning. Combined with the local root of trust, a verifiable full lifecycle offline closed loop is formed, reducing dependence on external networks, external servers, and remote inference services. This provides a secure offline operating environment for the offline capability package governance process and task execution process, enabling the robot to meet continuous and stable operation requirements in isolated networks, no network, weak network, strong electromagnetic interference, and high-security scenarios.

[0045] 2. The offline governance link, which executes the offline capability package list parsing, hash tree verification, signature verification, dependency topology loading, inactive slot installation, atomic switching, and observation window rollback through the offline capability package governance module, enables the control system to still have the ability to upgrade safely, recover from failures, and trace versions even without network conditions. This reduces or even avoids system malfunctions caused by data corruption, incomplete dependencies, or switching failures during the import process.

[0046] 3. By implementing a unified technology chain through various modules, including multimodal perception, fusion state estimation, unified condition vector, skill graph planning, uncertainty-driven action generation, safe projection, and ontology adaptation, execution deviations caused by inconsistencies in state representation, update timing, and triggering conditions among multiple modules are reduced. This ensures the consistency of control link execution and improves the real-time performance, stability, and reliability of offline closed-loop control on the robot side.

[0047] 4. A comprehensive uncertainty assessment is conducted based on the fusion state representing the robot's state and the world representation representing the characteristics of the surrounding environment. This comprehensive uncertainty index is introduced into the robot's motion generation process, which can automatically expand the motion search range when the environment is unstable, the recognition confidence is low, or the state estimation variance is large, and reduce sampling noise when the environment is stable, thereby improving the accuracy of the generated motion. This allows the robot's motion to take into account exploration ability, execution smoothness, and control real-time performance.

[0048] 5. By performing a forced safety projection problem-solving and force-position hybrid control on the generated candidate action sequence, the output candidate action must meet the constraints of collision, force, speed, torque, joint limit and no-entry area before execution. This reduces the risk of correct skill path planning but dangerous underlying execution and improves the accuracy and safety of robot task execution.

[0049] 6. Through the runtime resource adaptive scheduling module, the robot's computing power, temperature rise, power consumption, memory usage, execution latency and minimum safety control frequency are incorporated into the unified resource scheduling, so that the task execution process can dynamically adjust resources according to the actual state of the robot, and the robot can maintain reliable local offline closed-loop operation for a long time even under limited hardware conditions.

[0050] 7. The ontology abstraction and adaptation module maps high-level abstract action semantics to low-level control instructions that adapt to the robot ontology. The offline capability package governance module performs cross-platform and / or cross-ontology consistency verification on the capability package, enabling the same high-level skill logic to be reused across different chip architectures, operating systems, driver environments and robot ontologs. This reduces the difficulty of capability migration caused by the lack of a unified ontology abstraction layer, interface description and cross-platform consistency verification mechanism, reduces migration costs and improves capability package reusability.

[0051] 8. Through a unified abnormal state handling and chain auditing mechanism, the system achieves hierarchical handling and local traceability of sensor failures, planning failures, generation timeouts, actuator abnormalities, update failures, and audit abnormalities, thereby enhancing the system's continuous operation capability and accountability traceability in high-security offline scenarios.

[0052] The full lifecycle offline embodied intelligent control method provided in this invention can be applied to, for example... Figure 2The embodied intelligent robot shown includes an embodied intelligent body (i.e., the robot body) and a full-lifecycle offline embodied intelligent control system (hereinafter referred to as the control system). The control system communicates with the embodied intelligent body via a bus.

[0053] Among them, such as Figure 1 As shown, the control system includes an external communication isolation control module, a local root of trust and monotonic counting module, a multimodal perception and preprocessing module, a fusion state estimation and world representation module, a task understanding and condition vector construction module, a skill graph planning module, an uncertainty assessment and action sequence generation module, a safety constraint execution module, an ontology abstraction and adaptation module, a runtime resource adaptive scheduling module, a unified anomaly handling module, and a local auditing module. These modules together constitute the core control layer of the control system. The runtime resource adaptive scheduling module provides operational configuration adjustments for the core control layer. The unified anomaly handling module is responsible for performing global state switching and hierarchical handling of anomaly information reported by each module. The local auditing module is responsible for generating chained audit records for key events throughout the entire process. The robot body is equipped with multiple sensors of different types, as well as motion actuators or drive systems that drive the robot body's movement, and corresponding interfaces, such as sensor interfaces and body control interfaces. The various sensors, actuators, or drive systems on the robot body communicate with the control system via various buses, such as sensor buses, actuator buses, and internal message buses. The functional modules within the control system communicate with each other via these buses to implement the control method provided in this embodiment. The control system establishes connections with sensors, offline media, actuators, or drive systems through sensor interfaces, offline media interfaces, and actuator interfaces (i.e., the robot body control interface), thereby enabling the transmission and reception of different types of data.

[0054] Among them, such as Figure 3 As shown, the full lifecycle offline embodied intelligent control method is applied in Figure 1 Taking the control system in the example, the functional modules of the control system are configured to perform the following steps: S01: After the control system is powered on, close, shield, or logically isolate the communication paths outside the control system to establish an offline boundary.

[0055] Upon power-up of the control system, before loading the model, strategy, map, and task resources, the external communication isolation control module is activated. This module shuts down, masks, or logically isolates the control system's external communication paths to establish an offline boundary. Subsequent control chains are triggered upon verification of the offline boundary's establishment. In other words, by shutting down, masking, or logically isolating the control system's external communication paths, an offline operating environment is established, triggering the operation of other functional modules.

[0056] S02: Read local trust root and slot information.

[0057] The offline capability package governance module reads the local trust root and slot information stored in the local trust root and monotonic counting module. The local trust root includes the stored root public key, version count, integrity verification benchmark, and audit chain head information, etc.; the slot information includes inactive slot identifiers and active slot identifiers.

[0058] S03: Is an offline capability package detected? If yes, proceed to step S04; otherwise, proceed to step S07.

[0059] The offline capability package management module detects whether an offline medium is inserted. If an offline medium is detected and the offline medium contains a candidate offline capability package, then it is determined that an offline capability package has been detected, and step S04 is executed. If no offline medium is detected, or if an offline medium is detected but the offline medium does not contain a candidate offline capability package, then it is determined that no offline capability package has been detected, and step S07 is executed, that is, the active slot is directly loaded to run the resource.

[0060] S04: Receive candidate offline capability packets through offline media, import and verify the offline capability packets.

[0061] After receiving candidate offline capability packages via offline media, the offline capability package governance module imports the offline capability packages into the isolation area and performs multi-dimensional verification on the offline capability packages based on the local root of trust, including signature verification, integrity verification, version rollback prevention verification, dependency verification, compatibility verification, validity period verification, space verification, and security policy verification.

[0062] S05: After verification, install the offline capability package into the inactive slot, load the components according to the dependency topology, and complete the test.

[0063] The offline capability package management module installs the offline capability package into the inactive slot corresponding to the inactive slot identifier, performs component loading according to the dependency topology sorting, and completes local self-check, offline playback test and regression test to test whether the offline capability package meets the preset activation conditions.

[0064] S06: When the offline capability package is found to meet the preset activation conditions, perform atomic switching and observation window monitoring; when the system operation parameters monitored in the observation window do not trigger the rollback conditions, proceed to step S07.

[0065] When an offline capability package in an inactive slot meets the preset activation conditions, the offline capability package governance module performs an atomic switch at the control cycle boundary or safe shutdown point to switch the inactive slot to an active slot. After performing the atomic switch, the offline capability package governance module monitors system operating parameters within a preset observation window to determine whether the system operating parameters trigger rollback conditions. If the system operating parameters in the observation window trigger rollback conditions, the system automatically rolls back to the previous stable active slot. If the system operating parameters monitored in the observation window do not trigger rollback conditions, step S07 is executed.

[0066] S07: Load the active slot's operating resources and complete the preheating process during operation.

[0067] When an atomic switch is performed and the system operating parameters monitored in the observation window do not trigger the rollback condition, or no offline capability package is detected, the offline capability package governance module loads the operating resources required for the current operation, such as model files, map resources, policy files, configuration files, and ontology adaptation parameters, into the current active slot.

[0068] S08: Collect multimodal data to obtain multimodal observation data of the robot body, and update the fusion state based on the multimodal observation data.

[0069] During operation, the multimodal perception and preprocessing module acquires multimodal observation data of the robot body and performs preprocessing operations such as time synchronization, coordinate registration, missing data repair, noise estimation, and anomaly gating to obtain multimodal perception data.

[0070] The Fusion State Estimation and World Representation module calculates weights for multimodal sensing data, obtains the fusion state, and constructs and updates a world representation that includes occupied space, semantic objects, and topological relationships based on the multimodal sensing data.

[0071] S09: Construct a unified condition vector.

[0072] The task understanding and condition vector construction module constructs a unified condition vector based on the fused state, task objective, world representation, safety constraint representation, and robot body constraint representation.

[0073] S10: Path planning is performed based on condition vectors to obtain the target skill path.

[0074] The skill graph planning module performs optimal path search in the preset skill graph based on condition vectors to obtain the target skill path.

[0075] S11: Generate a sequence of candidate actions corresponding to the target skill path.

[0076] The uncertainty assessment and action sequence generation module determines a comprehensive uncertainty index based on the uncertainty of the fusion state, the uncertainty of the world representation, the intensity of dynamic environmental changes, and the confidence level of localization and recognition, and generates candidate action sequences corresponding to the target skill path accordingly. S12: Perform safe projection and ontology adaptation on the current candidate action.

[0077] The safety constraint execution module performs a projection solution that satisfies the preset safety constraints on the current candidate action in the candidate action sequence, obtains a safe and executable action, and triggers at least one of the following actions when there is no feasible solution: emergency stop, safety action rollback, or safety degradation.

[0078] The body abstraction and adaptation module maps safe and executable actions to low-level control commands of the robot body, and performs position control, speed control, force control or force-position hybrid control according to the task type.

[0079] S13: Issue low-level control commands.

[0080] After receiving the underlying control instructions, the body abstraction and adaptation module sends the underlying control instructions to the actuator or drive system through the actuator interface, so as to drive the robot body to perform corresponding actions based on the underlying control instructions.

[0081] S14: Runtime resource adaptive scheduling.

[0082] During operation, the runtime resource adaptive scheduling module detects the system's latency, power consumption, temperature rise, memory usage, and safety control frequency, and dynamically adjusts the system's operating configuration based on these factors and minimum safety control frequency constraints.

[0083] Among them, the offline capability package in the activity slot sequentially drives the robot body to complete the offline control task through multimodal perception and preprocessing, fusion state estimation and world representation, task understanding and condition vector construction, skill graph planning, uncertainty assessment and action sequence generation, safety constraint execution and ontology abstraction and adaptation during task operation. The runtime resource adaptive scheduling module dynamically adjusts the operation configuration during the above process.

[0084] S15: Exception handling and local auditing.

[0085] During the operation of each functional module, the unified exception handling module receives exception information reported by each functional module and performs global state switching and exception handling according to preset exception classification and state transition rules. At the same time, the local audit module generates chained audit records for key events in the operation of each functional module, including deployment, import, verification, installation, loading, activation, operation, alarm, degradation, emergency stop, manual takeover, rollback and decommissioning events, and stores them locally.

[0086] In summary, this control system constructs an offline closed-loop control link encompassing communication isolation, trusted verification deployment, offline updates, task execution, unified anomaly handling, and local auditing. This closed-loop link enables offline boundaries, trusted updates, task execution, safety constraints, cross-platform reuse, and fault recovery to coordinate within the same control system. This solves the problems of incomplete offline boundaries, fragmented control links, unrecoverable updates, lack of rigid constraints on execution safety, and untraceable anomalies found in existing solutions. It achieves full lifecycle offline operation, ensuring safe execution, trusted updates, and traceable maintenance of the robot in an offline environment, thus improving the robot's stability and reliability.

[0087] Specifically, for ease of description, the following describes each functional module of the control system separately, in order to explain the specific process by which each functional module in the control system implements the above-mentioned full life cycle offline embodied intelligent control method in combination with the function of each functional module and its implementation path.

[0088] The external communication isolation control module is used to start before the model, strategy, map and task resources are loaded after the control system is powered on, to close, shield or logically isolate the communication path outside the control system in order to establish an offline boundary, and to trigger the start of the subsequent control chain when the offline boundary is verified.

[0089] Specifically, during the startup phase after the control system is powered on or before the operating resources are loaded, the external communication isolation control module performs shutdown, unloading, shielding, physical isolation, logical isolation, or default rejection operations on external communication paths such as wired network ports, wireless network cards, cellular communication modules, Bluetooth external connection modules, remote login services, external API proxy processes, and domain name resolution services. Only internal or offline controlled communication functions such as sensor buses, actuator buses, internal message buses, and offline media interfaces are retained. Verifiable offline boundaries are established at the hardware and software levels, reducing the risk of the system accidentally accessing external networks or receiving remote commands during operation.

[0090] Specifically, after closing, shielding, or logically isolating the communication paths outside the control system, the external communication isolation control module determines whether the offline boundary of the control system is established based on the number of external routing entries, the set of external listening interfaces, and the number of external application interfaces. When the offline boundary of the control system is established, it indicates that an offline operating environment has been successfully established for the robot, and subsequent control chain startup can be triggered. This means triggering the operation of other functional modules, such as sending work information to other functional modules to remind them to enter working status. When the offline boundary of the control system is not established, it indicates that an offline operating environment has not been successfully established for the robot, the offline boundary is determined to be invalid, and the local audit module is prompted to record the offline boundary failure event and refuse to trigger subsequent control chain startup.

[0091] The offline boundary indicator is determined based on the number of external routing entries, the set of external listening interfaces, and the number of external application interfaces. When the offline boundary indicator equals 1 (i.e., only when the number of external routing entries is zero, the set of external listening interfaces is empty, and the number of external application interfaces is zero), the offline boundary of the control system is determined to be established. In this case, the control system is in an offline operating environment, triggering the subsequent control chain to run other functional modules, allowing the control system to enter the stages including capability package governance, runtime resource loading, and control loop startup. When the offline boundary indicator equals 0 (i.e., the number of external routing entries is not zero, the set of external listening interfaces is not empty, or the number of external application interfaces is not zero), the offline boundary of the control system is determined to be not established. The control system blocks the subsequent startup of the higher-level control chain, and this is written as an offline boundary failure event into the local audit chain record of the control system. This mechanism ensures that the establishment of the offline boundary is no longer a vague configuration state, but a verifiable and repeatable deterministic condition, avoiding the risk of offline failure due to the omission of closing any external communication path.

[0092] The offline boundary index is represented by the following formula: ; in, This is an offline boundary indicator; Indicates the current number of external routing entries. This represents the set of external listening interfaces that are currently in a listening state. This indicates the number of external application interfaces currently enabled. Indicates an indicator function; This represents the empty set. When... When the offline boundary is established, the control system is allowed to enter the startup phase of the execution process, including capability package selection, runtime resource loading, and control loop execution; if If the offline boundary is not established, the control system will block the subsequent activation of the higher-level control chain and write it as an offline boundary failure event into the local audit chain record of the control system.

[0093] The startup sequence of the control system during the startup phase includes the following actions executed sequentially: offline boundary establishment, local trust root reading, active slot confirmation, critical capability component loading, runtime warm-up, self-test completion, and closed-loop control entry. Critical capability components include at least security policies, execution control kernels, ontology abstraction adaptation parameters, driver interface descriptions, and minimum security control configurations. Only after all critical capability components have been successfully loaded with trusted credentials is loading of higher-level skill graphs, map resources, and action generation models permitted. This strictly phased and explicitly dependent startup sequence ensures the system's secure establishment from the bottom boundary to higher-level capabilities in an offline environment, preventing control loop anomalies due to improper resource loading order or the incomplete readiness of critical components.

[0094] The local root of trust and monotonic counting module is used to store the local root of trust and slot information of the control system. The local root of trust includes the local root public key, anti-rollback information and capability package verification benchmark.

[0095] The local trust root and monotonic counting module stores the local trust root and slot information of the control system. This local trust root is used to verify subsequent offline capability packages before they are loaded, ensuring the offline security of the control system. The local trust root includes a root public key, rollback prevention information, and capability package verification benchmarks. The rollback prevention information includes a version count (i.e., the monotonic count value of the control system version) and audit chain head information. Slot information includes active and inactive slot identifiers for the robot. The active slot identifier indicates the slot where the capability package is located during normal robot operation; the inactive slot identifier indicates candidate slots in the robot where no running capability package is installed.

[0096] By storing the root public key, rollback prevention information (including version count and audit chain head information), slot identifier, and capability package verification benchmark in the local trust root and monotonic counting module, the system can verify the source trust and integrity of candidate offline capability packages in a completely offline environment. It uses the one-way non-rollback characteristic of version counting to prevent malicious downgrade attacks, and combines active and inactive slot identifiers to achieve secure capability package switching. Based on the audit chain head information, it ensures the continuous anti-tampering of audit records. Thus, a complete offline security chain from verification and rollback prevention to slot management is established before capability packages are loaded, ensuring that the control system can safely and reliably complete the deployment, update, and operation of capability packages without relying on external networks throughout the entire lifecycle.

[0097] The offline capability package governance module is used to perform multi-dimensional verification on the offline capability package based on the local root of trust after receiving the candidate offline capability package through the offline medium. After the verification is passed, the offline capability package is installed into the robot's inactive slot, the component loading is performed according to the dependency topology sorting, and when the offline capability package in the inactive slot meets the preset activation conditions, an atomic switch is performed at the control cycle boundary or safe stop point to switch the inactive slot into an active slot.

[0098] The offline capability package governance module manages the import, isolation, verification, installation, loading, activation, rollback, and deregistration of offline capability packages.

[0099] The offline capability package management module detects whether the robot has been inserted with offline media via the offline media interface. Upon detection, it imports the offline capability packages from the offline media into the robot's memory isolation area or temporary cache. Within this secure area, capability package verification, installation, loading, activation, rollback, and deregistration operations are performed. Direct operation within the active slot is strictly prohibited to ensure the continuity of the robot's current control logic.

[0100] Among them, a single offline capability package The specific information is as follows: ; in, List of capability packages; A collection of model files, map resources, strategy files (such as safety policy files), configuration files, and interface description files required for robot operation; A list of dependencies required for the robot to run. This is the set of test vectors for performing offline self-tests, playback tests, and regression tests on the robot. This is digital signature information.

[0101] Among them, the list of capability packages for offline capability packages The information included is as follows: ; in, For capability package identification; This is the version number of the offline capability package, i.e., the version number of the control system to be updated. This version number is recorded in the capability package list and is protected by digital signature along with the capability package list. It is preferred to use a monotonically increasing sequence. A rollback prevention index for the version number; For the set of supported computing platforms; For the set of supported robot body types; For file hash tables; The validity period of the offline capability package; The set of dependencies required to install this offline capability package; For a set of security policy parameters; This represents the minimum set of resource requirements needed for the robot.

[0102] After importing the offline capability package into the robot's memory isolation area, multi-dimensional verification is performed on the offline capability package. Only when the verification results of the candidate offline capability package meet the capability package installation conditions is the offline capability package allowed to be installed, registered, or loaded into at least one of the following: an inactive slot, a candidate resource space, or a non-current running area in the robot. In other words, the offline capability package is copied, decompressed, registered, or loaded into the corresponding candidate running environment. The inactive slot is identified by an inactive slot identifier. The multi-dimensional verification includes signature verification, integrity verification, version verification, dependency verification, compatibility verification, validity period verification, space verification, and security policy verification.

[0103] Specifically, multi-dimensional verification is performed on the offline capability package, including the following verification process: 1. The digital signature information in the offline capability package is verified using the local root of trust and the root public key stored in the monotonic counting module. Specifically, a hash tree is constructed from the contents of all files within the offline capability package using a hash function to obtain the hash tree root value Hpkg; the signature object information and digital signature information in the capability package list are read, and the key metadata in the capability package list and the hash tree root value are used as verification objects. The digital signature information is verified using the root public key to obtain the signature verification result. This method verifies the credibility of the capability package content, list structure, and source.

[0104] The signature verification can be represented by the following formula: ; in, For signature verification results; This is a function for verifying digital signatures. The root public key in the local trust root; List of capability packages; This indicates the digital signature information in the capability package list; This is a splicing symbol; This represents the overall summary of the contents of all files within the offline capability package, specifically the root value of the hash tree obtained by constructing a hash tree from all files within the offline capability package using a hash function. In other words, if the signature verification result is 1, the signature verification passes; if the signature verification result is 0, the signature verification fails.

[0105] The overall summary can be represented as follows: ; in, This is the overall summary of the offline capability package, i.e., the root value of the hash tree; The first one in the offline capability package One file; This represents the total number of files within the offline capability package. This is a hash function.

[0106] Therefore, the digital signature of the offline capability package is generated by concatenating the hash value of the overall content of the capability package and the digital signature information of the capability package list. Based on this, the content and structure of the capability package can be jointly verified, realizing the dual anti-tampering function of content and structure, thereby improving security.

[0107] 2. Integrity Verification: Read the baseline hash value of each file in the offline capability package from the file hash table, and calculate the actual hash value of each file. The integrity verification passing can be represented by the following formula: ; in, The result of the integrity verification; For the first The actual hash value of each file; For the first The base hash value of the first file is the first one pre-generated and recorded when the capability package is released. The standard hash value of a file; Indicates an indicator function; This is the multiplication symbol.

[0108] Based on the baseline hash value of each file recorded in the capability package list, the actual hash value is calculated for each file within the capability package, and then compared with the corresponding expected hash value. The capability package integrity verification is considered successful when all file comparisons match. By comparing the actual hash value of each file in the capability package with the expected hash value one by one, the capability package integrity verification is considered successful only when all files match. This forms a double verification with the overall hash, which can pinpoint the specific file that has been tampered with, improving the accuracy of data integrity verification.

[0109] 3. Version validity verification, i.e., version rollback prevention verification: The version number in the offline capability package is verified against the latest local version number determined by the local version counter. A successful version rollback verification can be expressed by the following formula: ; in, This is the result of the version rollback verification; Cmp is the version comparison function; This refers to the version number of the offline capability package in the capability package list; This is the local minimum license version, which is the latest local version number determined by version counting; For offline capability packages, a rollback prevention index is provided. This is the minimum fallback index stored in the local root of trust and the monotonic counting module; This indicates an indicator function.

[0110] By simultaneously verifying that the capability package version is not lower than the minimum version requirement and that there is no version rollback, the legitimacy of the capability package version can be determined. This can prevent the installation of older versions (i.e., control system function degradation) and prevent malicious rollback attacks, thereby improving the security of the control system.

[0111] 4. Dependency Verification: Based on the dependency set already installed in the control system, the integrity of the dependency relationships in the offline capability package is verified. A successful dependency verification can be expressed by the following formula: ; in, This depends on the verification results; This refers to the set of dependencies required to install the offline capability package, which is listed in the capability package inventory. This is the collection of dependencies that are already installed locally. Indicates an indicator function; A dependency closure indicates that it includes not only direct dependencies but also recursive dependencies (i.e., indirect dependencies) of those dependencies.

[0112] Dependency validation passes when all dependencies required by the capability package are included in the set of dependencies already installed on the control system. Compared to checking only direct dependencies, indirect dependencies are also considered here to avoid missing components at runtime and improve the stability and executability of the control system.

[0113] 5. Compatibility Verification: Based on the hardware and software information of the control system running locally, the operating environment of the offline capability package is verified for compatibility. This hardware and software information is represented by two dimensions: the locally running computing platform (i.e., software) and the robot body type (i.e., hardware).

[0114] The compatibility check passing can be represented by the following formula: ; in, This is the result of the compatibility check. A collection of computing platforms supported by the offline capability package; A computing platform for the robot to run locally; The robot body type of the robot in which the control system is located is used to characterize the differences in robot structure; it can be the robot body type. This refers to the set of robot body types supported by the offline capability package, such as the supported robot types.

[0115] By determining whether the current operating platform and device type are both within the support range of the capability package, it is possible to determine whether the capability package is compatible with the control system. The compatibility check will only pass if both the current control system platform and the device type are within the support range of the capability package. This prevents the installation of a control system version that is not supported by the platform from running the capability package on the wrong device, thereby improving the stability and compatibility security of the control system.

[0116] 6. Validity Verification: The validity period of the offline capability package is verified based on the local trusted clock of the control system. A successful validity verification can be expressed by the following formula: ; in, This indicates the validity period verification result; Indicates a local trusted clock or a local time identifier obtained by a protected monotonic count mapping; The expiration date is specified in the capability package list; This indicates an indicator function.

[0117] By comparing the current time with the expiration date of the capability package, it is determined whether the capability package is still valid. If the current control system time has not exceeded the expiration date of the capability package, the validity period verification is deemed to have passed, preventing expired capability packages from being loaded or using invalid or insecure versions, thereby improving the security and timeliness control capabilities of the control system.

[0118] 7. Space Constraint Verification: Based on the remaining available storage space and reserved protection space of the control system, a space constraint verification is performed on the offline capability package. The passing of the space constraint verification can be expressed by the following formula: ; in, This is the result of spatial constraint verification; The remaining available storage space for the control system; Space required for installing the capability package; Reserved protection space for the control system; This indicates an indicator function.

[0119] The space constraint check is passed by determining whether the remaining space in the control system meets the installation requirements of the capacity package and the safety reservation requirements. When the currently available space of the control system is greater than or equal to the sum of the space required for capacity package installation and the reserved space, the space constraint check is considered passed. This can prevent failure due to insufficient space during installation or abnormalities in the control system due to space exhaustion, thereby improving the stability and operational safety of the control system.

[0120] 8. Security Policy Verification: The successful verification of the security policy can be represented by the following formula: ; in, For security policy verification results; This refers to the set of security policy parameters in the capability package list; The minimum security policy requirements for controlling the system locally; Indicates an indicator function; The formula indicates that the policy satisfies the relationship, does not exceed the relationship, or is allowed. This formula means that the security policy required for the offline capability package is not lower than the local minimum security policy requirement.

[0121] By comparing the security policies of capability packages with the local security policies of the control system, it is determined whether the capability package meets the security requirements of the control system. When the security policies of capability packages meet the minimum local security policy requirements and do not violate the permissions, resources, and security constraints of the local control system, the security policy verification is deemed successful. This can prevent high-risk or uncontrolled capability packages from being loaded, reduce the risk of permission violations or security policy conflicts, and improve the overall security and policy consistency of the control system.

[0122] After performing multi-dimensional verification on the offline capability package, the results of each verification are used to determine whether the offline capability package meets the aforementioned capability package installation conditions. The capability package installation conditions are expressed by the following formula: ; in, For signature verification results; The result of the integrity verification; This is the result of the version rollback verification. This depends on the integrity verification results; This is the result of the compatibility check. This is the result of the validity period verification; This is the result of spatial constraint verification; This is the result of the security policy verification.

[0123] when When the multi-dimensional verification is successful, it indicates that the verification results meet the installation conditions of the capability package. That is, the offline capability package is allowed to be installed in the inactive slot corresponding to the inactive slot identifier only when the signature verification, integrity verification, version verification, dependency verification, compatibility verification, validity period verification, space verification, and security policy verification are all successful. When the multi-dimensional verification fails, it indicates that the verification results do not meet the installation conditions of the capability package. That is, if any of the signature verification, integrity verification, version verification, dependency verification, compatibility verification, validity period verification, space verification, and security policy verification fails, the offline capability package will be refused to be installed, and the installation refusal event and verification results will be sent to the local audit module so that the local audit module can record it as a chain audit record.

[0124] After the multi-dimensional verification results meet the capability package installation conditions, i.e., after the multi-dimensional verification passes, the control system determines the active slot in the robot based on the active slot identifier and the inactive slot based on the inactive slot identifier. The offline capability package that has passed the multi-dimensional verification is then installed into the inactive slot. Next, component loading is performed according to the dependency topology order of each component in the offline capability package to load the offline capability package into the corresponding runtime environment. Through the above multi-dimensional verification, it is ensured that only capability packages that simultaneously meet the following criteria can be installed into inactive slots: trustworthy source, untampered, version anti-rollback, complete dependencies, hardware and software compatibility, not expired, sufficient storage space, and compliance with security policies. This effectively blocks malicious, damaged, expired, incompatible, or resource-insufficient capability packages from entering the control system in a completely offline environment, significantly improving the security and reliability of the capability package installation process and providing a stable and reliable candidate basis for subsequent uninterrupted atomic switching.

[0125] In one embodiment, after the offline capability package is installed into an inactive slot, component loading is performed on the candidate capability according to dependency topology sorting. Specifically, a dependency graph is constructed based on the dependencies of each component in the candidate capability, and topology sorting is performed. The components in the offline capability package are then loaded sequentially according to the topology sorting result. This includes the following steps: parsing the dependencies of different components in the offline capability package, constructing a dependency graph based on the dependencies of each component, performing topology sorting on the dependency graph to obtain a component loading sequence in the offline capability package; this component loading sequence includes multiple components arranged in topological order. According to this component loading sequence, the multiple components in the offline capability package are sequentially loaded into the runtime environment (i.e., the execution environment of the memory isolation area) corresponding to the inactive slot.

[0126] The process of constructing the dependency graph can be represented by the following formula: ; in, Represents the dependency graph of the build process; This represents the collection of components to be loaded. Edges representing component dependencies; This represents the component loading sequence after topological sorting; TopoSort represents the function or algorithm that implements topological sorting.

[0127] Loading components using dependency graphs and topology sorting can reduce the risk of installation failures caused by incorrect component loading order.

[0128] In one embodiment, the offline capability package includes multiple components, including critical components and ordinary components (also known as non-critical components). During component loading, critical components are loaded first, and non-critical components are loaded on demand. Specifically, when loading multiple components sequentially into inactive slots, the pre-defined critical components can be loaded first into the runtime environment corresponding to the inactive slot, according to the component loading sequence. Then, based on actual needs, multiple ordinary components (i.e., non-critical components) are loaded into the runtime environment corresponding to the inactive slots, thus achieving priority loading of critical components and on-demand loading of non-critical components. Critical components include security policies, execution control kernels, ontology abstraction adaptation parameters, driver interface descriptions (i.e., driver protocols), and security constraint libraries (including minimum security control configurations), among other related components.

[0129] In one embodiment, during the component loading process, for each component, its signature and integrity are independently verified, and the loading status of its predecessor dependent components is confirmed. When the signature and integrity verification of a component passes (i.e., independent verification passes), and all its predecessor dependent components are successfully loaded, the loading operation for that component is executed, and the loading status of the component is marked as successfully loaded. When the signature or integrity verification of a component fails, or its predecessor dependent components fail to load, the loading operation for that component is not executed, and the loading status of the component is marked as unsuccessfully loaded. When the loading status of all components in the offline capability package is successfully loaded, the installation of the offline capability package is determined to be successful; when the loading status of any component in the offline capability package is unsuccessfully loaded, the installation of the offline capability package is determined to be unsuccessful, the installation data of the offline capability package is deleted, and the above-mentioned capability package import and installation process is re-executed.

[0130] The loading state of a component in the component loading sequence can be determined by the following formula: ; in, For the first The loading status of each component. This indicates that the component has been successfully loaded. This indicates that the component has failed to load. This is the independent verification result for this component; This is the set of predecessor dependent components of this component; Indicates the first Precursor dependencies of each component Loading successful; This indicates an indicator function.

[0131] By performing component-level validation, confirming predecessor dependencies, and recording loading status, the integrity and reliability of offline capability packages in inactive slots can be improved.

[0132] In one embodiment, the preset activation conditions include at least: the offline capability package completing the multi-dimensional verification or the verification result of the multi-dimensional verification remaining valid, the key components being successfully loaded, and the local verification result meeting the preset requirements; wherein, the local verification includes one or more of local self-testing, offline replay testing, and regression testing. The offline capability package governance module is also used to monitor the anomaly rate, control latency, thermal load, and stable control pass rate within a preset observation window after performing atomic switching, and automatically roll back to the previous stable active slot when any indicator within the observation window triggers the rollback condition.

[0133] In other words, after all components of the offline capability package have been successfully loaded, the offline capability package governance module is also used to perform the following steps: S11: Perform local self-test, offline playback test and regression test on the offline capability package on the inactive slot to test whether the inactive slot meets the preset activation conditions.

[0134] After the capability package is installed, a local self-test is performed on the offline capability package in the inactive slot to verify the collaborative relationship between components and the basic operating capability of the control system. Offline playback and regression tests are also performed in the inactive slot to test whether the inactive slot meets the preset activation conditions.

[0135] The local self-test includes signature verification, integrity verification, and loading status checks and basic function verification of each component within the offline capability package to verify component legitimacy and loadability. Offline playback testing simulates the offline capability package based on historical input data from the control system to verify the correctness of the control system's operational logic. Regression testing runs the offline capability package based on preset test cases, compares its output results with reference results to determine the regression test pass rate (i.e., the pass rate of the test cases), and evaluates whether its performance meets preset requirements.

[0136] The regression test pass rate can be expressed by the following formula: ; in, For regression test pass rate; The number of test cases; For the first The error between the output of each test case and the reference result; The allowable error threshold; This indicates an indicator function.

[0137] The preset activation conditions include: the signature and integrity verification of each component in the offline capability package is passed, the loading status is all completed (i.e., the component dependency topology is loaded successfully), and the regression test pass rate is greater than or equal to the preset threshold (i.e., the minimum pass rate). In addition, the preset activation conditions may also include that the offline playback test result shows that the system's operating status is normal during the playback test.

[0138] In other words, the offline capability package in an inactive slot is determined to meet the preset activation conditions only if the offline capability package satisfies signature and integrity verification, dependency topology loading passes, and the regression test pass rate is not lower than a preset threshold (and the control system operates normally during playback testing). This allows for atomic switching at the robot control cycle boundary or safe stop point. Conversely, if the offline capability package fails signature or integrity verification, or if any component's loading status is incomplete, or the regression test pass rate is lower than a preset threshold (or the control system operates abnormally during playback testing), the offline capability package in an inactive slot is determined not to meet the preset activation conditions. Therefore, atomic switching is not performed, and the current stable active slot is maintained. The preset activation conditions include at least: the offline capability package completing signature and integrity verification, completing dependency topology loading, and completing local self-test, offline playback test, and regression test.

[0139] This solution combines local self-inspection (signature, integrity, component loading status, and basic function verification), offline replay testing (simulating the correctness of the running logic based on historical data), and regression testing (comparing the error between preset test case outputs and reference results and evaluating the pass rate). This multi-level and multi-dimensional verification of the offline capability package in an offline environment comprehensively assesses its legality, functional correctness, and performance compliance. Only capability packages that pass all tests can proceed to the subsequent atomic switching process, significantly reducing the possibility of runtime logic errors, performance anomalies, or security risks, and improving the reliability and security of system updates.

[0140] S12: When the preset activation conditions are met, an atomic switch is performed at the boundary of the control cycle of the control system to switch the inactive slot to the active slot, and the system operating parameters in the preset observation window are monitored after the switch. The system operating parameters in the observation window include the abnormality rate, control delay, thermal load and stable control pass rate.

[0141] The control cycle refers to the time interval required for the control system to execute a complete control algorithm once. The control cycle boundary refers to the switching moment between two adjacent control cycles, or the safe moment when the control loop allows version switching. The safe stop point refers to the controlled moment when the robot is stationary, at low speed, with stable load, or when the task phase ends naturally, which can be used as the preferred execution time for atomic switching. When the offline capability package in the inactive slot meets the above preset activation conditions, the control cycle boundary or safe stop point of the control system is detected. When the control cycle boundary or safe stop point is detected, atomic switching is performed at that moment, so that the switching action does not change the control instructions already issued in the current cycle, thereby avoiding control jitter, instruction truncation, or state inconsistency during the switching process. That is, the active slot A corresponding to the currently running capability package of the same type is switched to the inactive slot of the control system, and the inactive slot B with the offline capability package is switched to the active slot of the control system, so that the control system can execute the skill corresponding to the offline capability package to achieve atomic-level skill switching of the control system.

[0142] The active slot switching process of the control system can be represented by the following formula: ; in, Indicates the activity slot for the next stage; This indicates an inactive slot, i.e. a candidate slot with an offline capability package installed. Indicates the currently active slot; This indicates the signature and integrity verification results during local self-check; This indicates the loading status and basic capability check results of each component during local self-test; Indicates the pass rate of the regression test; This indicates the preset threshold, i.e., the minimum pass rate; This indicates that the offline capability package meets the signature and integrity verification, the dependency topology loading is successful, and the regression test pass rate is not lower than the preset threshold. In other words, the offline capability package in the inactive slot meets the preset activation conditions.

[0143] After performing the atomic switch, the system enters a preset observation window and monitors the operating parameters of the control system, such as the anomaly rate, control delay, thermal load, and stable control pass rate. The system operating parameters within the observation window are obtained, and then based on the system operating parameters such as the anomaly rate, control delay, thermal load, and stable control pass rate, it is determined whether the performance of the control system after running the offline capability package meets the requirements, thereby determining whether the rollback conditions for capability rollback are met.

[0144] S13: When the system operating parameters in the observation window trigger the rollback condition, the system will automatically roll back to the previous stable active slot. S14: When the system running parameters in the observation window do not trigger the rollback condition, load the resources required for the current operation in the current active slot, including model files, map resources, strategy files, configuration files and ontology adaptation parameters, to complete the runtime warm-up.

[0145] Within the observation window, when the system operating parameters do not trigger the rollback condition (i.e., the abnormality rate or number of abnormal events does not exceed the corresponding abnormality threshold, the control delay does not exceed the corresponding delay threshold, the heat load or temperature rise index does not exceed the corresponding temperature rise threshold, and the stable control pass rate or safe projection pass rate is not lower than the corresponding pass rate threshold), it is determined that the operating performance of the control system after loading the offline capability package on the active slot meets the requirements. Then, the model file, map resources, strategy file, configuration file, and ontology adaptation parameters required for the current operation are loaded in the current active slot to complete the runtime preheating and enter closed-loop control after preheating.

[0146] Within the observation window, when any of the following system operating parameters triggers the rollback condition—namely, the anomaly rate (or number of abnormal events), control delay, thermal load (or temperature rise index), and stable control pass rate (or safe projection pass rate)—is greater than or equal to the corresponding parameter threshold, the rollback condition is determined. This indicates that the operating performance of the control system after installing the offline capability package on the active slot does not meet the requirements, and the system automatically rolls back to the previous stable active slot. Specifically, the previously stable active slot A, which was switched to an inactive slot, is switched back to the active slot of the control system, and the active slot B currently equipped with the offline capability package is switched to an inactive slot of the control system. Automatic rollback only allows reverting to the previously verified stable slot locally, ensuring the recoverability of the control system performance and ensuring the normal and stable operation of the control system.

[0147] The rollback condition can be triggered by the following formula: ; in, This is a rollback condition; To observe the number of abnormal events or the abnormality rate within the observation window; This refers to the threshold for the abnormal quantity corresponding to the number of abnormal events or the threshold for the abnormal rate, i.e., the maximum number of abnormal events or the maximum abnormal rate allowed by the control system. This represents the average control delay of the control system. This is the time delay threshold, which is the maximum control time delay allowed by the control system. This refers to the average heat load or temperature rise index. This is the temperature rise threshold corresponding to the average heat load or temperature rise index, i.e., the maximum heat load or maximum temperature rise allowed by the control system. To ensure stable control of the throughput or safe projection throughput; The threshold for the pass rate, which is the minimum pass rate allowed by the control system, is used to stabilize the pass rate or the safe projection pass rate. This indicates an indicator function.

[0148] Among them, when When the system's operating parameters trigger the rollback condition, it automatically rolls back to the previous stable active slot and sends the slot rollback event and the reason for triggering the rollback to the local audit module, so that the local audit module records it as a critical event in a chain of audit logs. If the system's operating parameters do not trigger rollback conditions, the system loads the necessary model files, map resources, strategy files, configuration files, and ontology adaptation parameters into the current active slot, completing runtime warm-up. After warm-up or rollback to the previous stable active slot, offline control of the robot is implemented based on the active slot's capability package to enter the robot's closed-loop control process, enabling the robot to operate normally and execute corresponding tasks. During normal robot operation, key events and abnormal events occurring in each functional module are sent to the local audit module, which records them as chained audit logs for subsequent data traceability.

[0149] The reliability of the offline capability package activation process can be improved through local verification, atomic switching, observation window monitoring, and automatic rollback.

[0150] In conjunction with the above, such as Figure 4 As shown, in a specific embodiment, the offline capability package governance module is specifically used to perform the following steps to implement the capability package installation and activation process: S101: Import the offline capability package to the memory isolation area via offline media; S102: The offline capability package enters the memory isolation area; S103: Parse the list of capability packages for this offline capability package; S104: Perform multi-dimensional verification on the offline capability package based on the parsed information; S105: Determine whether the multi-dimensional verification has passed; otherwise, proceed to step S106; otherwise, proceed to step S107. S106: Refuses to install the offline capability package and records the installation refusal event and verification result to the local audit module, forming a chain of audit records; S107: Install the offline capability pack to an inactive slot; S108: Load components of the offline capability package according to the dependency topology order; S109: Prioritize loading critical components; The topology order is determined based on dependencies, and components of the offline capability package are loaded according to this order, with critical components loaded first and ordinary components loaded on demand. It is then determined whether the critical components of the offline capability package have been successfully loaded; if so, step S110 is executed; otherwise, the current stable active slots are maintained.

[0151] S110: Perform local self-test, offline playback test and regression test in inactive slots; S111: Determine whether the activation condition is met based on the test results; if yes, proceed to step S112; otherwise, proceed to step S115. S112: Perform an atomic switch at the control cycle boundary or safe stop point to switch an inactive slot to an active slot; S113: Monitor and control the operating status of the offline capability package in the active slot within the observation window to obtain system operating parameters; S114: Determine whether the system operating parameters trigger the rollback condition; if yes, proceed to step S115; otherwise, proceed to step S116.

[0152] S115: Maintain the current active slot or roll back to the previous stable active slot; S116: Load runtime resources and complete runtime preheating; S117: Operate normally and record audits.

[0153] The control system operates normally based on the capability package of the active slot and records key events and abnormal events during normal operation to the local audit module, forming a chain of audit records.

[0154] The offline capability package governance module performs multi-dimensional verification of capability packages, installation in inactive slots, and sequential loading of dependency topologies within a memory isolation area. Key components are loaded first, and local self-checks and regression tests are performed after loading to execute atomic switching at control cycle boundaries or safe stop points. After atomic switching, a preset observation window is entered, and status monitoring is performed within the observation window to trigger conditional rollback. This process, combined with chained recording from the local audit module, achieves secure isolation of capability packages in an offline environment, zero-downtime upgrades, component dependency reliability, pre-deployment risk verification, automatic rollback of operational anomalies, and full-process anti-tampering traceability. This significantly improves the security, recoverability, and autonomous governance capabilities of the robot control system.

[0155] The multimodal perception and preprocessing module is used to acquire multimodal observation data of the robot body and perform preprocessing to obtain multimodal perception data. The preprocessing includes time synchronization, coordinate registration, missing data repair, noise estimation, and outlier removal.

[0156] During the operation of the control system, observation data from sensors on the robot body are collected through the multimodal perception and preprocessing module to obtain multimodal observation data of the robot. Preprocessing operations such as time synchronization, coordinate registration, missing value repair, noise estimation, and outlier removal are performed on the collected multimodal observation data to obtain multimodal perception data.

[0157] The sensors on the robot body include vision sensors, depth sensors, laser sensors, inertial sensors, force sensors, tactile sensors, and encoders, as well as other sensors on the robot body. A single sensor may include multiple observation channels, and a single observation channel may correspond to multiple sensors.

[0158] To address the issue of inconsistent sampling times across different sensors, for each observation channel in the multimodal observation data, the sampling point closest to the reference time is found. Then, through interpolation resampling or nearest neighbor alignment methods, the original observation data from different channels in the multimodal observation data are aligned to a unified time, resulting in time-aligned multi-channel observation data. Subsequently, coordinate registration, missing data repair, noise estimation, and outlier removal are performed on the time-aligned multi-channel observation data to obtain multimodal sensing data, providing an accurate data foundation for subsequent data calculations.

[0159] Among them, for the first in multimodal sensing data The raw observation data from each observation channel, at the reference time Aligned observation data can be represented by the following formula: ; ; in, Indicates the first Raw observation data from each observation channel; Indicates the first The raw observation data of each observation channel at the reference time Aligned observation data; This represents an interpolation, resampling, or nearest neighbor alignment function; For the first In the raw observation data of each observation channel, relative to the reference time The closest original sampling time.

[0160] In one embodiment, outlier removal includes: employing an innovative gating algorithm (such as the chi-square threshold algorithm) to perform outlier gating processing on each observation channel in the time-aligned multi-channel observation data to obtain multimodal sensing data. Specifically, the outlier gating processing involves, for each observation channel in the time-aligned multi-channel observation data, calculating the difference between the actual observation data and the predicted observation data for that channel based on the predicted state at the current moment, determining the gating variable for that observation channel based on the difference data, and then performing anomaly detection and outlier removal based on the gating variable. If an anomaly is detected, the outlier data for the observation channel is removed; if normal, the normal data is used for subsequent fusion calculations.

[0161] The specific process includes: acquiring the actual observation data of the observation channel after time alignment at the current moment; calculating the predicted observation data of the observation channel at the current moment based on the predicted state of the control system at the current moment using the corresponding observation model; calculating the difference data between the actual observation data and the predicted observation data, and converting multiple difference values ​​in the difference data into a difference value matrix for the observation channel; transposing the difference value matrix and normalizing the difference value normalization of the innovative covariance of the difference value matrix to obtain a statistically significant deviation measure, such as Mahalanobis distance; comparing the deviation measure with a preset statistical threshold, and using the comparison result as a gating variable. If the deviation measure is less than or equal to the preset statistical threshold, i.e., the gating variable equals 1, the observation data of the observation channel is determined to be normal and recorded as normal data; if the deviation measure is greater than the threshold, i.e., the gating variable equals 0, the observation channel is determined to be abnormal and recorded as abnormal data to be removed in subsequent fusion. Each observation channel undergoes the above outlier removal process. After anomaly gating of the observation data for each observation channel, outlier observation channel data is removed, resulting in observation data for each normal and usable observation channel, thus forming multimodal sensing data.

[0162] The gate variables in the exception gating process can be represented by the following formula: ; in, Indicates the first Each observation channel time The actual observed values; For the control system at any time The predicted state; For the first An observation model for each observation channel; For the first Each observation channel at time The predicted observations; To innovate the covariance matrix; This represents the Mahalanobis distance function, which is used to calculate a measure of deviation. The chi-square threshold is the preset statistical threshold. This is a gate variable used in the anomaly gating process. If the gate variable is equal to 0, it means that the data of the observation channel is judged to be abnormal or unusable at the current moment and needs to be removed. If the gate variable is equal to 1, it means that the data of the observation channel is judged to be normal or usable at the current moment and is used for subsequent weight and fusion state calculations.

[0163] By using anomaly gating to remove anomalous observation data, the stability of subsequent fusion state estimation can be improved.

[0164] The Fusion State Estimation and World Representation module is used to fuse multimodal perception data to obtain the robot's fused state, and to construct the environment based on the multimodal perception data to obtain a world representation that includes occupied space, semantic objects, and topological relationships.

[0165] The fusion state estimation and world representation module is used to calculate the weights of each observation channel in the multimodal sensing data based on the gating variables, channel confidence, and time delay of each observation channel, thus obtaining the fusion weights of each observation channel. Based on the fusion weights of each observation channel, the data from different observation channels in the multimodal sensing data are fused to obtain the robot's fusion state. It is important to understand that each observation channel data includes multiple actual observation values.

[0166] Specifically, for each observation channel in the multimodal sensing data, that is, for the observation data of each normal and usable observation channel obtained after anomaly gating processing, the fusion weight is calculated based on the gating variable, channel confidence, noise and time delay to obtain the fusion weight of each observation channel.

[0167] The specific calculation process for the fusion weights includes: for each available observation channel, obtaining multiple evaluation indicators at the current time, including: channel confidence, noise measurement, time synchronization deviation, or transmission delay (i.e., latency); based on the multiple evaluation indicators of the observation channel at the current time, calculating the comprehensive evaluation value at the current time, and obtaining the comprehensive evaluation value of different observation channels; normalizing the comprehensive evaluation values ​​of each observation channel so that the sum of all channel weights is 1, thus obtaining the fusion weight of each observation channel. In calculating the fusion weights, the positive impact of confidence on the weights can be increased, and a penalty mechanism can be introduced for noise and latency to suppress their effect on the weights; for channels determined to be unusable during the anomaly detection phase, their fusion weights are directly set to 0.

[0168] The fusion weights for each observation channel can be calculated using the following formula: ; in, Indicates the first Each observation channel at time The fusion weight; These are gate variables used in the exception gating process. For the first Each observation channel at time Channel confidence; For the first Each observation channel at time Noise measurement; For the first Each observation channel at time The delay can be a time synchronization deviation or a measure of transmission delay; These are the first available observation channels. Each observation channel at time Channel confidence; This is the confidence gain coefficient. This is the noise penalty factor. This is the delay penalty coefficient; represents the total number of available observation channels; exp is an exponential function used to map the comprehensive evaluation value to a positive number and amplify the differences.

[0169] Then, the control system can perform weighted fusion of the data from each available observation channel based on the fusion weight of each observation channel to obtain the fusion state of the multimodal sensing data.

[0170] In one embodiment, the control system can further use the multimodal sensing data after anomaly gating and the fusion weights of each observation channel as input to construct a unified optimization objective. Based on the fusion weights of each observation channel, the observation residuals of each observation channel are weighted and accumulated, and state prior constraints are introduced to construct a joint optimization objective function. The joint optimization objective function is solved using weighted least squares, MAP estimation (maximum a posteriori probability estimation), extended Kalman filtering, unscented Kalman filtering, or factor graph optimization algorithms to obtain the optimal state estimate of the robot at the current moment, which serves as the robot's current fusion state. The state variables of the control system include the robot's position, attitude, and velocity. The optimization objective is to find an optimal state among all possible state variables of the robot that can both maximize the explanation of the actual observations of each observation channel and conform to the time evolution law of the control system state.

[0171] The fusion state of the robots, i.e., the objective function of joint optimization, can be expressed by the following formula: ; in, The merged state, i.e., the state variable At any moment The optimal state estimation results; Represents the state variables of the control system Solve the optimization objective; Indicates the first Each observation channel at time The fusion weight; Indicates the first Each observation channel time The actual observed values; For the control system at any time The predicted state; For the first An observation model with multiple observation channels is used to convert state variables... Mapped to the observation space; Indicates the first The observation residuals of each observation channel; For the first The observation noise covariance matrix is ​​used to reflect the measurement uncertainty of this observation channel; These are the state priors for state evolution; The state prior constraint function for state evolution is used to measure the deviation between the current state and the state at the previous time step; This is the prior weight, used to adjust the balance between observation consistency and temporal continuity; it is a constant value. This represents the total number of available observation channels.

[0172] This scheme constructs a unified optimization objective function that integrates multi-channel observation residuals and state evolution prior constraints. Based on gating and adaptive weighting, it achieves joint state estimation of multimodal observations, thereby introducing time continuity constraints while ensuring observation consistency and improving the accuracy and stability of state estimation.

[0173] Simultaneously, the fusion state estimation and world representation module is also used to reconstruct environmental information based on multimodal sensing data, constructing a world representation that includes occupied space, semantic objects, and topological relationships. Based on the multimodal sensing data at the current moment and the fused state obtained, the world representation at the previous moment is recursively updated to obtain the world representation at the current moment. Among them, the world representation includes three sub-representations: spatial occupancy representation (referred to as occupancy representation), semantic object representation, and topological relationship representation.

[0174] Specifically, the fusion state estimation and world representation module recursively updates the world representation from the previous time step based on the multimodal sensing data at the current time step and the fused state obtained by fusion, including the following steps: Acquire the world representation of the previous moment, the multimodal sensing data of the current moment, and their corresponding fusion states.

[0175] The multimodal perception data at the current moment is identified and classified to obtain the robot's occupancy status in the local space, the set of semantic objects, changes in the spatial structure and object distribution of the robot's space, and the set of dynamic targets. The set of dynamic targets consists of the robot's dynamic obstacles or interactive targets.

[0176] Based on the current multimodal perception data and its corresponding fusion state, the occupancy state of the local space is updated, including writing new obstacles, updating free space, and correcting historical occupancy information, to obtain the updated space occupancy representation.

[0177] Based on the currently identified set of semantic objects, existing semantic objects are matched, updated, or added, including object category confirmation, location update, and attribute correction, to obtain the updated semantic object representation.

[0178] Based on the changes in the spatial structure and object distribution in the robot's space, the spatial topological relationship representation is updated, including the connection relationships between different regions and / or objects, such as reachability, adjacency, or interaction relationships, to obtain the updated topological relationship features.

[0179] The updated spatial occupancy representation, semantic object representation, and topological relation representation are then combined and merged into the world representation at the current moment.

[0180] The update process of the world representation at the current moment can be represented by the following formula: ; in, For the current moment The world's representation; The semantic set of the currently identified objects. It is a set of dynamic targets, including dynamic obstacles and sets of dynamic interactive targets; For time representation, update function; This represents the world at the previous moment; This represents the current state of fusion.

[0181] In the process of updating the above representation, for each dynamic target in the dynamic target set (such as dynamic obstacles and interactive targets), its state can be tracked and its time-varying characteristics can be reflected in the spatial occupancy representation and semantic object representation, so as to avoid the dynamic target being mistakenly solidified into a static structure.

[0182] In the process of identifying a dynamic target set, the fusion state estimation and world representation module can also calculate the degree of environmental change at each spatial location in the robot's space, and based on the degree of environmental change at different spatial locations, identify dynamic obstacles or dynamic interactive targets to obtain a dynamic target set.

[0183] For example, if the intensity of environmental change at a certain spatial location is lower than the intensity threshold, the control system classifies it as a static environment (walls, pillars, etc.) and updates it directly to the map; if the intensity of environmental change is higher than the intensity threshold, the control system will treat the point cloud or features of that spatial location as dynamic obstacles or dynamic interactive targets and classify them into the dynamic target set (such as walking people or moving cars).

[0184] Specifically, the calculation process for the degree of environmental change at each spatial location is as follows: Within the local map area of ​​the robot's current location, the difference in spatial occupancy representation between the current moment and the previous moment is compared location by location to obtain the magnitude of the change in occupancy status for each spatial location; the magnitudes of the change in occupancy status for all spatial locations are accumulated and normalized according to the area size to obtain the average degree of occupancy change within a unit area, which is taken as the intensity of environmental change at the current moment. This intensity of environmental change reflects the changes in obstacles or the dynamics of the scene in the environment and can be used for subsequent decision-making or strategy adjustment.

[0185] The intensity of dynamic environmental change is calculated using the following formula: ; in, For the current moment The intensity of dynamic environmental changes; This refers to the current local map area. This represents the difference in spatial occupancy between the current time and the previous time, i.e., position. The range of change in the occupied state; Location in the current local map area At the present moment Spatial occupancy representation; For position The spatial occupancy representation in the previous moment.

[0186] This scheme constructs a unified world representation by integrating spatial occupancy information, semantic information, and topological relationships. It then recursively updates this representation based on state estimation, current observations, and dynamic targets. Simultaneously, it obtains the intensity of environmental changes by statistically calculating changes in the spatial occupancy representation. This enables structured modeling and change perception of the dynamic environment, effectively distinguishing between static obstacles and dynamic objects. It ensures that the robot can maintain an accurate and real-time internal world model even when offline and under resource constraints, providing an effective data foundation for subsequent world representation updates and the determination of comprehensive uncertainty indicators.

[0187] The task understanding and condition vector construction module is used to determine the task objective. It encodes the fusion state, world representation, task objective, security constraint representation and ontology constraint representation into a condition vector to eliminate control deviations caused by inconsistencies in input representations between modules.

[0188] The task understanding and condition vector construction module is used to determine the task objective, which can be a task instruction written offline to the robot or a preset task script. This module parses, disambiguates, and structures the offline task instruction or preset task script to determine the executable task objective.

[0189] In one embodiment, the task understanding and condition vector construction module is further used to project and concatenate the fused state, world representation, task objective, safety constraint representation, and robot ontological constraint representation onto a unified feature space through linear transformation or embedding mapping to form a uniformly encoded condition vector. This eliminates control deviations caused by inconsistencies in input representations between different modules such as skill path generation, action generation, and execution control. This condition vector is also available for use by both the skill graph planning module and the uncertainty assessment and action sequence generation module.

[0190] Specifically, after forming condition vectors, the task understanding and condition vector construction module outputs the condition vectors to the skill graph planning module, enabling the skill graph planning module to search for target skill paths based on the condition vectors. The module then outputs both the condition vectors and the target skill paths to the uncertainty assessment and action sequence generation module to generate candidate action sequences. Subsequently, the safety constraint execution module performs safety projection or constraint correction based on the candidate action sequences to obtain safe and executable actions. The ontology abstraction and adaptation module converts these safe and executable actions into low-level control instructions. Thus, the condition vectors form a unified data transmission link between task understanding, skill planning, action generation, safety execution, and ontology adaptation.

[0191] Among them, safety constraint representation and robot body constraint representation are physical limiting parameters for robot behavior. Safety constraint representation includes, but is not limited to, minimum safe distance threshold, maximum permissible contact force, maximum permissible speed, maximum permissible driving torque, and prohibited area set. Robot body constraint representation includes upper limit of joint position, lower limit of joint position, upper limit of joint speed, and body control and interface constraint parameters; these body control and interface constraint parameters include body driving protocol, control frequency, interface format, and limit description parameters, etc.

[0192] Since the fusion state, world representation, task objective, security constraint representation, and ontology constraint representation belong to different types of data with inconsistent dimensions, scales, and semantic spaces, they cannot be directly fused. Therefore, the control system processes the various input information at the current moment separately through the task understanding and condition vector construction module. For each type of data, a corresponding linear transformation matrix or embedding mapping is introduced, projecting it onto a unified feature space to obtain the feature vectors corresponding to each type of data, ensuring consistency in numerical scale and expression. The linear transformation or embedding mapping not only achieves dimension alignment but also extracts key features from each data type, providing structured input for subsequent fusion.

[0193] Secondly, for complex world representations, which contain multiple layers of information such as occupancy, semantics, and topology, and are typically graph-structured or multi-dimensional data, vectorization operations can be used to convert them into one-dimensional or fixed-dimensional computable representations, encoding them as vectors. Then, feature extraction and spatial alignment are performed using corresponding mapping matrices, enabling them to participate in fusion with other information. After completing the unified mapping of various information types, the transformed feature vectors are concatenated in a preset order to form the final unified condition vector. This concatenation process preserves the independent features of each information source while achieving information aggregation within the same vector space. This allows subsequent modules to simultaneously acquire state information, task objectives, environmental information, and various constraints through a single input interface.

[0194] The condition vector of the unified coding can be represented by the following formula: ; in, For at any time The condition vector obtained by unified encoding; For robots at all times The state of fusion; For robots at all times The world's representation; For at any time Task objectives; Represented as a safety constraint; This represents the robot's ontological constraints; vec is the vectorization operation, which converts matrices, tensors, or structured data into one-dimensional vectors; Concat is the vector concatenation function. , , , , It is a linear transformation matrix or an equivalent embedding map.

[0195] The encoding process of this condition vector involves feature mapping of state, task, environment, and constraint information, followed by unified concatenation to form a fusion input representation that combines multi-source information with a unified expression form. Through this encoding process, the control system achieves the transformation from multi-source heterogeneous information to a unified condition expression, effectively avoiding semantic deviations or control errors caused by inconsistent input expressions between different modules, and providing a consistent and computable input basis for subsequent action generation, policy reasoning, or execution control.

[0196] The skill graph planning module is used to search for the optimal path in a preset skill graph based on condition vectors to obtain the target skill path.

[0197] Specifically, after obtaining the condition vector uniformly encoded by the aforementioned modules, the skill graph planning module performs an optimal path search based on the condition vector and the preset skill graph. This allows it to select the execution path with the lowest overall cost from multiple candidate path branches, while satisfying the constraints, as the target skill path. The condition vector is a feature vector obtained by uniformly encoding the fusion state, world representation, task objective, safety constraint representation, and robot body constraint representation. In other words, the skill graph planning module uses the task objective, fusion state, world representation, safety constraint representation, robot body constraint representation, and the preset skill graph to perform an optimal path search, determining a target skill path that satisfies the task objective, robot state, surrounding environment constraints, safety constraint representation, and robot body constraints.

[0198] During path search, the feasibility of switching between skills can be determined based on preconditions or postconditions. A scoring function, incorporating factors such as matching degree, risk, energy consumption, and uncertainty, is used to search for the target skill path with the lowest overall cost through a pre-defined graph search algorithm. The pre-defined graph search algorithm can be either A* or Dijkstra's algorithm.

[0199] The skill graph planning module constructs a preset skill graph based on the robot's offline executable skills and the switchable relationships between these skills. The nodes (i.e., skill nodes) of this skill graph represent offline executable skills (such as grasping, moving, and obstacle avoidance), and the edges represent the switchable relationships between skills. Each edge in the skill graph contains three key attributes: preconditions, postconditions, and switching costs.

[0200] Specifically, the process by which the skill graph planning module searches for the optimal path to obtain the target skill path includes the following steps: S21: Using the current unified encoding condition vector as input, starting from the starting skill node of the skill graph, initialize the path to be searched to form the initial candidate path branches.

[0201] After determining the task objective, the skill graph planning module takes the current unified encoded condition vector as input and starts from the starting skill node of the skill graph to initialize the search path: for all candidate skill nodes after the current skill node, it judges whether their preconditions are met one by one, and only retains the skill nodes that meet the preconditions as the expandable direction of the path, thus forming the initial candidate path branches.

[0202] Each candidate path branch consists of multiple skill nodes and their edges. Specifically, starting from the initial skill node, the edge relationships in the skill graph are traversed step by step. For any path branch from the skill node... Point to skill node edge All conditions are determined based on the current condition vector and its preceding conditions; only when... From the current skill node Switch to skill node Only when the prerequisites are met is it permissible to modify the edge and its corresponding skill node. Candidate paths are incorporated to form candidate path branches. Simultaneously, during path selection, resource constraints (such as energy budget and security constraints of condition vectors, ontology constraints, etc.) are used to constrain and filter candidate skill nodes and their edges, thereby gradually constructing a set of candidate path branches that satisfy both the task objective and the constraints. This process essentially involves constrained graph search within the skill graph, and pruning the search space through precondition checks to ensure that skill switching within the candidate paths is feasible under the current state, environment, and constraints.

[0203] S22: During the path expansion process, perform quality evaluation on each candidate skill node after the current skill node to obtain the quality score of each candidate skill node. Convert the quality score of the candidate skill node into part of the path cost and accumulate it with the skill switching cost and time cost to obtain the cumulative cost of the current candidate path branch.

[0204] After obtaining multiple candidate path branches, during the path expansion process, the quality of each candidate skill node following the current skill node is evaluated, resulting in a quality score for each candidate skill node. This quality score reflects the degree of matching between the skill and the task objective, its consistency with the current state and environment, and factors such as the risk, energy consumption, and uncertainty of executing the skill. The higher the quality score, the better the skill. Based on this, the quality score of the candidate skill node is converted into part of the path cost and accumulated with the skill switching cost and time cost to obtain the cumulative cost of the current candidate path branch.

[0205] Specifically, for each candidate skill node following the current skill node in each candidate path branch, a quality score for each candidate skill node is calculated based on the current task objective and the environmental context information of that candidate skill node. This includes: calculating the matching degree between the skill corresponding to the candidate skill node and the target task to obtain the target matching degree; determining the consistency between the skill corresponding to the candidate skill node and the environmental context information (including the current fusion state and world representation) to obtain context consistency; calculating the resource consumption required to execute the skill corresponding to the candidate skill node using historical energy consumption data to obtain the energy cost; estimating the potential safety risks or failure risks associated with executing the skill corresponding to the candidate skill node based on safety constraints and environmental risk assessment results to obtain the risk cost; and estimating the uncertainty of the skill execution result after executing the skill corresponding to the candidate skill node based on the robot's ontology model or historical skill execution data to obtain the uncertainty cost. Based on the target matching degree, context consistency, risk cost, energy cost, and uncertainty cost of the candidate skill node, a quality score for that candidate skill node is calculated. For example, the above scoring factors can be weighted and summed to obtain the quality score of the candidate skill node. The quality score of a skill node can be determined in the following way: ; in, Assess the quality of candidate skill nodes; Target matching degree, used to represent the degree of matching between skills and target tasks; For contextual consistency, it is used to represent the consistency between skill and environmental context information; Risk cost, used to represent the safety or failure risks that may arise from the execution of skills; Energy cost, used to represent the degree of resource consumption required to perform a skill; The cost of uncertainty is used to characterize the uncertainty of skill execution results; The weights for the corresponding parameters are all constant values ​​greater than 0, and the sum of the weights equals 1. The weights of each scoring factor are non-negative and their sum is normalized to ensure that different evaluation factors work together proportionally on the skill node score, guaranteeing the rationality, stability, and interpretability of the quality score.

[0206] While determining the quality scores of all candidate skill nodes, the switching cost and estimated time cost of switching from the current skill point to a candidate skill node can be determined; the estimated time cost can be determined based on the historical time data required for the skill switch. Based on the quality scores of each candidate skill node, the switching cost between skill nodes, and the estimated time cost, the cumulative cost of the candidate path branch is calculated, resulting in the cumulative cost of multiple candidate path branches. Specifically, the edges between two adjacent skill nodes in a candidate path branch can be considered as sub-paths. For each sub-path, the switching cost, estimated time cost, and the quality score of the sub-path endpoint are weighted and summed to obtain the cost of that sub-path. The costs of all sub-paths in the candidate path branch are then summed to obtain the cumulative cost of the candidate path branch. This cumulative cost is calculated using the following cost function: ; in, Indicates candidate path branches The cumulative cost; To start from skill nodes Switch to skill node The switching cost is used to reflect the complexity or cost of switching between different skills; To start from skill nodes Switch to skill node The estimated time cost; Skill Node Quality rating; The weights are non-negative.

[0207] This scheme incorporates the quality score of skill nodes into the path cost function and, combined with skill switching costs and time costs, accumulates the costs of each sub-path in the candidate path branch to obtain the cumulative cost of that candidate path branch. Modeling this cumulative cost achieves a balance between skill quality, switching costs, and execution efficiency, ensuring that the quality of a path depends not only on its reachability but also on its execution quality and cost level, thus improving the accuracy of the cumulative cost.

[0208] S23: Using a preset graph search algorithm, based on the cumulative cost of each candidate path branch, perform optimal path search on all candidate path branches to determine the candidate path with the minimum comprehensive cost, which is then used as the target skill path.

[0209] During path search, paths with lower cumulative costs and closer proximity to the target skill node are prioritized for further expansion; that is, paths with the lowest total cost are prioritized for further expansion. As the search progresses, when a candidate path branch reaches the target skill node and its total cost is the lowest among all searched paths (or satisfies the optimality condition), this candidate path is determined as the target skill path. The target skill node is the skill node mapped to the task objective.

[0210] Taking the task of grasping a workpiece and moving it to a target location as an example, the initial skill node where the robot is currently located is determined according to the preset skill map, and the target skill node corresponding to the task objective is determined accordingly. Furthermore, between the initial skill node (i.e., the task start point) and the target skill node (i.e., the task end point), there are at least the following skill nodes: Skill 1 (corresponding to the skill: move to point A), Skill 2 (corresponding to the skill: move to point B), Skill 3 (corresponding to the skill: grasp workpiece), Skill 4 (corresponding to the skill: obstacle avoidance), Skill 5 (corresponding to the skill: move workpiece), and Skill 6 (corresponding to the skill: place workpiece). The edges of each skill node are as follows: Figure 5 As shown, the process of determining the target skill path includes: starting from the task's starting point, using skill node 1 and skill node 2 as candidate skill nodes for precondition determination; when the preconditions for switching from the task's starting point to skill node 1 and skill node 2 are both met, the path from the task's starting point to skill node 1 is designated as candidate path branch 1, and the path from the task's starting point to skill node 2 is designated as candidate path branch 2. The cumulative cost of the two candidate path branches is calculated separately. If the cumulative cost of candidate path branch 1 is less than the cumulative cost of candidate path branch 2, then candidate path branch 1 is designated as a sub-path of the target skill path. After determining this sub-path, the path is expanded downwards based on the endpoint of this sub-path, repeating the above precondition determination and cumulative cost calculation process, selecting the path that meets the preconditions and has the lowest cumulative cost as the sub-path, until the task's endpoint is reached, thus forming the target skill path. Wherein, as... Figure 5 As shown, the target skill path includes: task start point - move to point A - grab the workpiece - transport the workpiece - place the workpiece - task end point. The target skill path and its determination process in this embodiment are only for illustrative purposes; in other embodiments, the target skill path and its determination process may also be other content determined according to the target task, which will not be elaborated here.

[0211] In one embodiment, Dijkstra's algorithm can be used to perform a global optimal search across multiple candidate path branches based on the cumulative cost of each candidate path branch. Alternatively, the A* algorithm can be used to perform a heuristic search across multiple candidate path branches based on the cumulative cost of each candidate path branch, thereby accelerating the search process and improving efficiency.

[0212] When using the A* algorithm for heuristic search, for each search node (i.e., candidate skill node) in the candidate path branches: based on the distance residual and energy consumption from the search node to the target skill node, the path cost from the search node to the target skill node is estimated as the future estimated cost of the candidate path branch; the sum of the cumulative cost and the future estimated cost of the candidate path branch is taken as the total cost of the candidate path branch; the candidate path branch with the lowest total cost is selected as the optimal path and the path is continuously expanded, thereby guiding the search to prioritize expansion in directions closer to the target and with lower costs. When expanding to the target skill node, the total cost of each candidate path branch is calculated, and the candidate path branch with the lowest total cost (or satisfying the optimality condition) is selected as the target skill path. Through heuristic guidance, the search space can be significantly reduced while ensuring optimality or near-optimality.

[0213] The estimated cost for the future can be determined using the following heuristic function: ; in, For search nodes The estimated future cost of the candidate path branch; The distance residual from the search node to the target skill node; For search nodes The estimated energy consumption to reach the target skill node, i.e. the remaining energy consumption estimate, is used to characterize the total amount of energy or resources required to execute subsequent skills from the search node until the target skill node is reached. These are preset weight values.

[0214] In one embodiment, the uncertainty assessment and action sequence generation module is used to adaptively adjust the sampling noise range and denoising step size in the robot action generation process based on the fusion state uncertainty, world representation uncertainty, intensity of dynamic environment change and positioning recognition confidence, so as to generate candidate action sequences corresponding to the target skill path.

[0215] In one specific embodiment, the uncertainty assessment and action sequence generation module is used to determine a comprehensive uncertainty index based on the fusion state uncertainty, world representation uncertainty, and the intensity of dynamic environmental changes and positioning and recognition confidence in the robot's space. Then, it adaptively adjusts the sampling noise range and denoising step size in the robot action generation process according to the comprehensive uncertainty index to generate candidate action sequences corresponding to the target skill path.

[0216] The candidate motion sequence includes one or more of the following: end-effector trajectory, chassis speed trajectory, desired contact force trajectory, or joint reference sequence.

[0217] The process of determining the comprehensive uncertainty index specifically includes: First, performing a trace operation on the covariance matrix of the fused state to obtain the uncertainty index of the fused state (i.e., fused state uncertainty). The fused state uncertainty represents the robot's confidence in its own position, velocity, and other states; the greater the fused state uncertainty, the more uncertain the robot's state. Second, performing environmental modeling on the world representation to obtain the uncertainty matrix of the world representation, and performing a trace operation on the uncertainty matrix to obtain the uncertainty index of the world representation (i.e., fused state uncertainty); the world representation uncertainty reflects the credibility of the map or environmental model. Third, performing a weighted summation of the fused state uncertainty, the world representation uncertainty, the intensity of dynamic environmental changes, and the confidence in localization and recognition to obtain the comprehensive uncertainty index. The fused state uncertainty is obtained by performing a trace operation on the covariance matrix of the fused state.

[0218] The comprehensive uncertainty index is calculated using the following formula: ; in, To comprehensively assess uncertainty; The covariance matrix of the fusion state; The uncertainty matrix represents the world. This indicates a trace operation; The fusion state is uncertain; This characterizes uncertainty in the world; The intensity of dynamic environmental changes; To determine the confidence level for location identification; The weights are non-negative.

[0219] The uncertainty matrix of the world representation includes uncertainty matrices for spatial occupancy representation, semantic object representation, and topological relationship representation. Specifically, the spatial occupancy probability of different locations can be determined based on the spatial occupancy representation, and the uncertainty matrix of the spatial occupancy representation can be constructed using the variance of the spatial occupancy probability at different locations. The category probability distribution or tracking error of semantic objects can be determined based on the semantic object representation, and the uncertainty matrix of the semantic object representation can be constructed based on the category probability distribution or tracking error. The connection relationships of the topological structure can be determined based on the topological relationship representation, and the uncertainty matrix of the topological relationship representation can be constructed based on the instability or update frequency of the connection relationships.

[0220] The positioning and recognition confidence score is a comprehensive confidence score obtained by fusing the positioning confidence score and the recognition confidence score. It reflects the reliability of the current control system's understanding of its own state and environment. For example, the comprehensive confidence score can be obtained by weighted summation or weighted average of the positioning confidence score and the recognition confidence score. The positioning confidence score can be inversely derived from the robot's state estimation covariance (such as the positioning error variance), with a smaller error resulting in a higher confidence score. The recognition confidence score can be the classification probability or detection confidence score output by a visual or multimodal perception model.

[0221] The specific process of generating candidate action sequences corresponding to the target skill path based on comprehensive uncertainty indicators includes: S31: Normalize the comprehensive uncertainty index to obtain the normalized comprehensive uncertainty index.

[0222] Specifically, the comprehensive uncertainty index can be mapped to the [0,1] interval through normalization and truncation functions, thus obtaining a normalized comprehensive uncertainty index for subsequent scheduling. The specific normalization process can be referred to the following formula: ; in, This is the normalized comprehensive uncertainty index; To comprehensively assess uncertainty; This is a truncation function; , This represents the maximum and minimum values ​​of the comprehensive uncertainty index.

[0223] S32: Based on the normalized comprehensive uncertainty index, determine the initial noise covariance matrix and step noise intensity in the action generation process.

[0224] Specifically, based on the normalized comprehensive uncertainty index, a preset base noise covariance matrix can be adjusted to obtain an initial noise covariance matrix, which describes the overall sampling range of the motion generation process. Simultaneously, based on the normalized comprehensive uncertainty index, a preset base noise amplitude can be adjusted to obtain the step noise intensity; this step noise intensity is used to control the perturbation amplitude at each step in the motion generation process. The initial noise covariance matrix and the step noise intensity can be determined using the following formulas: ; ; in, The initial noise covariance matrix; This represents the step noise intensity. The basic noise covariance matrix; The basic noise amplitude; This is the normalized comprehensive uncertainty index; and The adjustment coefficient corresponding to the comprehensive uncertainty index is a preset constant value.

[0225] Specifically, when the normalized overall uncertainty index is large (e.g., close to 1), the initial sampling noise covariance and step noise intensity will automatically increase due to the amplification effect of the preset adjustment coefficient, thereby allowing for a wider range of exploration in the action generation space. When the normalized overall uncertainty index is small (e.g., close to 0), the initial sampling noise covariance and step noise intensity will decrease, making the sampling more concentrated, thus generating a smoother and more stable action sequence.

[0226] The adaptive adjustment of the above parameters is achieved by proportionally amplifying the basic noise covariance matrix and the basic noise amplitude, giving the motion generation process environmental adaptability.

[0227] S33: Generate robot actions based on conditional vectors and target skill paths. During the action generation process, conditional asymptotic denoising is performed based on the initial covariance matrix and step noise intensity to iteratively generate candidate action sequences for the robot.

[0228] Based on the adaptive adjustment of the above parameters, and using the adjusted initial noise covariance matrix and step noise intensity, the actions corresponding to each skill segment in the target skill path are sampled and optimized. Conditional constraints are then applied to the sampled actions based on condition vectors, thereby generating candidate action sequences. The specific process includes: At the start of action generation, the initial covariance matrix is ​​sampled using a Gaussian distribution to obtain the initial noise latent variables for action generation, which serve as the initial action sequence. This initial action sequence represents actions with relatively strong randomness. That is: ; in, The initial action sequence is represented by an initial noise latent variable. The initial action sequence includes the first... The sequence of actions leading to step 0; Indicates a Gaussian distribution; This is the initial noise covariance matrix.

[0229] Subsequently, under the constraints of the conditional vector and the target skill path, the initial action sequence undergoes multi-step sampling and iterative updates to gradually remove noise and obtain the final structured action sequence as a candidate action sequence. The action generation process employs conditional asymptotic denoising, with each iteration including: predicting the current noise component based on the current noisy action sequence, conditional vector, and target skill path using a noise estimation function or denoising model; correcting the generated action based on the predicted noise component to gradually converge towards satisfying the target constraint, which is the target action constraint mapped from the target skill path; and introducing random noise controlled by the step noise intensity to maintain a certain level of exploration capability. By gradually reducing the noise impact during multi-step iterations, the action gradually converges from a random distribution to an executable trajectory, achieving conditional asymptotic denoising.

[0230] Among them, the Step to the first The step-by-step denoising iterative process can be represented by the following formula: ; ; in, For the first Action sequences obtained by step sampling; For the first Action sequences obtained by step sampling ; For the first The scheduling parameters for each step represent the proportion of signal retention at each step; Indicates the cumulative signal retention coefficient; This is a noise estimation function or a denoising model; For the target skill path; It is a conditional vector; This represents the current denoising step number, i.e., the time step. The current noise components are predicted; This represents the step noise intensity. For the first The random noise introduced in the step is obtained by sampling through a standard Gaussian distribution.

[0231] After completing the iteration, i.e. after convergence at step 0, the noisy latent variables (used to represent actions) at step 0 are decoded using the action decoding function based on the condition vector and the target skill path to obtain the candidate action sequence.

[0232] When the iteration is complete, the candidate action sequence can be decoded using the following action decoding function: ; in, Indicates a candidate action sequence; The temporal length of the action prediction; This is the action decoding function; The noise latent variable obtained from the sampling in step 0; For the target skill path; This is the condition vector.

[0233] As mentioned above, such as Figure 6 As shown, the process of generating candidate action sequences specifically includes the following steps: S301: Input condition vector and input comprehensive uncertainty index; S302: Adaptively set noise sampling parameters in the action generation process based on the comprehensive uncertainty index, including the initial noise covariance matrix and step noise intensity; S303: Initial sampling action representation based on noise sampling parameters: initial noise latent variable; S304: Input the target skill path; S305: Under the constraints of the conditional vector and the target skill path, starting from the k-th step, perform denoising iterative calculation on the initial noise latent variables to iteratively generate candidate actions; S306: Determine whether the termination condition for completing the iteration has been met. If yes, proceed to step S307; otherwise, repeat step S305. S307: Decode the noise latent variables output when the termination condition is met to obtain the candidate action sequence; S308: Output candidate action sequence.

[0234] In one embodiment, when the runtime resource adaptive scheduling module detects that the control system has insufficient runtime resources, the time delay constraint is tightening, or the above-mentioned action generation has not converged within a limited time, the uncertainty assessment and action sequence generation module can downgrade the action generation mode to a deterministic generation method based on skill templates and local optimization, thereby generating candidate action sequences and executing them. This effectively ensures that the robot can maintain a minimum safe control frequency even when computing resources are limited or real-time performance is tight in an offline environment, avoiding control failures caused by action generation timeouts or resource exhaustion, and significantly improving the real-time performance and operational robustness of the system.

[0235] The safety constraint execution module is used to perform projection solutions on candidate actions in the candidate action sequence to satisfy preset safety constraints, thereby obtaining safe and executable actions. When no feasible solution is found, i.e., no safe and executable action is obtained, the module triggers the execution of preset safety handling strategies. These preset safety handling strategies include at least one of emergency stop, safety action rollback, or safety degradation.

[0236] The safety constraint execution module receives candidate action sequences from the uncertainty assessment and action sequence generation module. Each candidate action may not fully consider the physical feasibility or environmental safety constraints at the current moment. Therefore, for each received candidate action, the safety constraint execution module solves a projection optimization problem with rigid safety constraints, mapping it to a feasible action space that satisfies preset safety constraints, and outputs the final safe and executable action. The core function of this module is to perform safety constraint verification and correction on the candidate actions output by the upper-level module, outputting safe and executable actions that meet full-dimensional rigid safety requirements, so as to preserve the intent and performance of the original candidate actions as much as possible while ensuring the robot's operational safety.

[0237] Specifically, using the safe and executable action to be solved as the optimization variable, a projection optimization problem for the current candidate action is constructed, which aims to minimize the weighted projection distance. The preset safety constraints are transformed into computable functions with respect to the optimization variables, resulting in multiple safety constraint functions. Using these multiple safety constraint functions as constraints and minimizing the weighted projection distance as the objective, a preset solver (such as a convexity solver) is called to solve the projection optimization problem for the current candidate action, resulting in a safe and executable action with rigid constraints.

[0238] The projection optimization problem for the current candidate action can be expressed by the following formula: ; in, The goal is to solve for the objective, namely, the safe and executable action after projection optimization. This is the current candidate action; To optimize the variables, i.e. the safe and executable actions to be solved; Transpose of the representation matrix; The positive definite projection cost matrix; This represents the objective function used to calculate the weighted projective distance, which is half the square of the weighted Euclidean distance between the optimization variable and the current candidate action.

[0239] A positive definite projection cost matrix is ​​used to assign different correction weights to motion components of different dimensions. For example, larger weights can be set for critical joints or high-precision motion axes to suppress deviations. The positive definite projection cost matrix is ​​a symmetric positive definite matrix, which ensures that the objective function is a strictly convex function, thus the optimization problem has a unique global optimum when the feasible region is non-empty.

[0240] In one embodiment, the preset safety constraints include collision safety distance limits, contact force limits, speed limits, torque limits, joint limits, and no-entry zone constraints. That is, when the safety constraint execution module performs projection solving on the candidate action, it must at least satisfy the preset safety constraints such as collision safety distance constraints, contact force constraints, speed constraints, torque constraints, no-entry zone constraints, and joint limit constraints.

[0241] The safety constraint functions corresponding to collision safety distance limits, contact force limits, speed limits, torque limits, joint limits, and no-entry zone constraints are as follows: ; ; ; ; ; ; in, To optimize the variables, i.e. the safe and executable actions to be solved; This refers to the current body configuration of the robot (including joint angles and end-effector pose). The set of environmental obstacles currently perceived (including static map obstacles and dynamic obstacles); To predict the distance to obstacles, i.e., to perform actions The minimum distance between the robot and the obstacle; This is a preset safe distance threshold. The collision safety distance limit ensures that the robot maintains a distance of at least the safe distance threshold from any obstacle after performing an action.

[0242] in, To predict contact force, i.e. to execute actions The contact force between the robot's end effector and the environment; It is the infinite norm, which means taking the maximum absolute value of each component of the vector; Maximum contact force; contact force limit ensures that the contact force in each dimension does not exceed the upper limit allowed by the hardware or task, preventing damage to the workpiece or joints. To predict speed, i.e., to execute actions The speed of each joint or end effector of the robot; This is the maximum speed; this speed limit ensures that the robot's movement speed does not exceed the motor's rated or dynamic stability limits. To predict torque, i.e. to execute actions The driving torque of each joint of the robot at the end; This is the maximum torque. The torque limit sets a safe upper limit for the torque of the robot joints to prevent overload or damage.

[0243] in, To predict spatial location, i.e. to perform actions The robot's spatial position afterward; This is a pre-defined set of prohibited areas, such as high-voltage areas, hazardous work areas, and sensor blind spots. This prohibited area constraint is a set membership constraint, ensuring that the robot will not enter any prohibited areas and guaranteeing robot safety. For the action The generated joint velocities can then be used to analyze the robot's movements using its kinematic model. Obtained by mapping; To control the cycle or predict the step time; These represent the physical limits of the joint angles, specifically the minimum and maximum joint angles, respectively. These joint angle limits ensure that the robot's joint angles do not exceed the hardware limits within the next control cycle.

[0244] In practical applications, the above process is repeated in each control cycle of the control system to achieve real-time execution of robot safety constraints. This design reduces the risk of the robot executing actions that violate preset safety constraints, while preserving the original intent of candidate actions to the greatest extent possible, thus achieving a balance between safety and task performance.

[0245] In this embodiment, by applying multi-dimensional safety constraints such as collision safety distance, contact force, speed, torque, joint limit, and restricted area to each action in the candidate action sequence and performing projection solution, a safe and executable action is output only when a feasible solution exists. If no solution exists, preset handling strategies such as emergency stop, safe action rollback, or safety degradation are automatically triggered. This effectively ensures the physical safety and compliance of robot actions in an offline environment, avoids risks such as collisions, overloads, or intrusion into restricted areas caused by violation of constraints, and significantly improves the system's safety tolerance under uncertain conditions.

[0246] Furthermore, when solving the projection optimization problem, if multiple feasible solutions exist, the solution with the smallest weighted distance to the current candidate action is selected as the safe and executable action. If no feasible solution exists (also known as no feasible solution), meaning there are no optimization variables that can simultaneously satisfy all preset safety constraints, the safety constraint execution module can also preset safety handling strategies. These preset safety handling strategies include at least one of emergency stop, reverting to the previous safe action, or entering a safety degradation mode. For example, based on the robot's current state and the environment, the current state priority can be determined. Based on the current state priority, the corresponding preset safety handling strategy is selected and executed according to the priority of emergency stop, maintaining pose, reverting to the previous safe action, or entering a safety degradation mode, and the infeasibility reason code is recorded.

[0247] Specifically, this includes: if a collision or joint overshooting is predicted based on the robot's current state and the environment, and the current state is determined to be of the highest priority, then the highest priority emergency stop strategy is executed, immediately sending a stop command to brake all moving parts to achieve an emergency stop. If the emergency stop strategy is unavailable or the robot is detected to be in a special state (such as a suspended state), and the current state is determined to be of the second priority, then a higher priority pose-maintaining strategy is executed, keeping the current joint angle and velocity at zero and maintaining the existing pose. Then, when the current state is of the third priority, a safe action rollback strategy is executed, using the safe and executable action successfully executed in the previous control cycle as the current output, while reducing the expected speed and attempting to gradually exit the danger zone. When the current state is of the fourth priority, a safety degradation strategy is executed, notifying the upper-level decision module to switch to a safety degradation mode, such as using a more conservative skill template, reducing movement speed, increasing the safe distance threshold, etc., while triggering an alarm to request manual intervention or automatically execute an escape strategy. During the execution of the aforementioned preset security policy, an execution log of the preset security policy is recorded. This execution log includes a timestamp, infeasibility reason code, and current state snapshot, which is used for post-event analysis and control system improvement.

[0248] This solution selects the solution with the smallest weighted distance to the current candidate action when multiple feasible solutions exist in projection optimization. This preserves the original action intent to the greatest extent while satisfying all safety constraints. When no feasible solution exists, the solution adaptively selects an emergency stop, safety action rollback, or safety degradation strategy based on the priority of the robot's current state and records the infeasibility reason code. This achieves flexible constraint processing that ensures both physical safety and task execution as much as possible in an offline environment. It also provides traceable evidence for post-fault analysis and significantly improves the system's safety adaptive capability and reliability.

[0249] The body abstraction and adaptation module is used to map safe and executable actions to the underlying control instructions of the robot body, and to perform position control, speed control, force control or force-position hybrid control according to the task type during the instruction generation process.

[0250] The body abstraction and adaptation module is located at the lowest level of the robot control system. Its core function is to map the safe and executable actions output by the upper-level safety constraint execution module into low-level control commands for the specific robot body, and drive the actuators to execute these low-level control commands. At the same time, this module also automatically selects position control, speed control, force control, or force-position hybrid control mode according to the task type to achieve precise and stable control of the robot body.

[0251] To support heterogeneous robots (such as mobile chassis, single-arm robotic arms, dual-arm composite robots, etc.), this module adopts an ontology abstraction layer design. It shields the differences in kinematics, dynamics, and drive protocols of different robot bodies through a unified abstract action semantic interface, so that the upper-level planning module can output a unified action description, which is then mapped to specific low-level control commands by the ontology adaptation module.

[0252] The ontology abstraction and adaptation module first embeds the safe executable actions output by the safety constraint execution module into abstract action semantics through the abstract action semantics interface. This abstract action semantics includes action type, target pose or region, desired force, task tolerance, and task time limit. This abstract action semantics is represented by the following structure: ; in, Indicates time Abstract action semantics, i.e., moment Semantic representation of safe and executable actions; Indicates the type of action that is safe to perform; Indicates the target pose or target area corresponding to a safe and executable action; This represents the expected force, which is the expected contact force or force threshold corresponding to a safe and executable action. This indicates the task tolerance corresponding to a safe and executable action; This indicates the task time limit, that is, the deadline for completing a safe and executable action, i.e., the duration of the action.

[0253] Then, the ontology abstraction and adaptation module determines the hardware and software layer parameters of the robot ontology based on its type. Based on these parameters, a preset mapping function is used to map the abstract motion semantics to the robot's underlying control commands. The hardware layer parameters include the robot's kinematic, dynamic, and geometric parameters. The software layer parameters include the drive protocol, control frequency (e.g., position, velocity, torque loop refresh rate), interface description (e.g., instruction format, data byte order, verification method), and limit parameters. Specifically, for the first... The robot-like body, its underlying control commands are represented as follows: ; in, For the first Mapping functions for robot-like applications; These are the hardware layer parameters of the robot body, such as its kinematic, dynamic, and geometric parameters. These are the software layer parameters of the robot body, such as drive protocol, control frequency, interface description, and limit parameters; For the first The underlying control commands for the robot-like body.

[0254] For mobile robots, the underlying control commands can be chassis linear velocity and angular velocity commands; for robotic arm robots, the underlying control commands can be joint control commands, such as joint position commands, joint velocity commands, or joint torque commands; for composite robots, the underlying control commands can be joint control commands for both the chassis and the robotic arm. In other words, during the process of mapping safe and executable actions to the underlying control commands of the robot body, the ontology abstraction and adaptation module generates chassis linear velocity and angular velocity commands for mobile robots, joint position, joint velocity, or joint torque commands for robotic arms, and joint control commands for both the chassis and the robotic arm for composite robots.

[0255] Specifically, the generation process of the underlying control instructions is as follows: First, the abstract action semantics of the safe and executable action are parsed to obtain the action type and task requirements. These task requirements include the target pose or region, desired force, task tolerance, and task time limit. Then, the robot's state data, hardware layer parameters, and software layer parameters are acquired. The robot's state data includes the current joint state (including joint angles and velocities), contact force, and end-effector pose. Based on the action type and task requirements, the robot's control mode is determined. Then, based on the task requirements of the safe and executable action, as well as the state data, hardware layer parameters, and software layer parameters, the mapping function corresponding to the robot's control mode is called to perform action-instruction mapping, obtaining the underlying control instructions corresponding to the safe and executable action.

[0256] The control modes include position control, speed control, force control, or force-position hybrid control. Correspondingly, the ontology abstraction and adaptation module includes a position control submodule, a speed control submodule, a force control submodule, and a force-position hybrid control submodule, each of which stores the mapping function corresponding to its respective control mode.

[0257] Specifically, for tasks involving contact operations, assembly, insertion / removal, pressing, or force-controlled grasping that require simultaneous constraints on position and force, a force-position hybrid control mode is adopted. Based on the task requirements for safe and executable actions, as well as state data, hardware layer parameters, and software layer parameters, the force-position hybrid control submodule is invoked to perform action-instruction mapping for the safe and executable actions, obtaining the corresponding underlying control instructions. That is, during the process of the body abstraction and adaptation module mapping safe and executable actions to the underlying control instructions of the robot body, force-position hybrid control is executed in assembly, insertion / removal, pressing, or force-controlled grasping tasks to ensure assembly accuracy in the position control direction and limit the contact force in the force control direction to not exceed a preset threshold. In the force-position hybrid control mode, the force-position hybrid control submodule uses the following mapping function to map and obtain joint control commands as the underlying control commands: ; in, Indicates joint control commands; This refers to the current robot body configuration; The Jacobian matrix of the current robot body configuration is used to map Cartesian space forces / torques to joint space torques. These are the target end-effector pose, desired end-effector velocity, and desired contact force, respectively. These are the robot's current end-effector pose, current end-effector velocity, and feedback contact force, respectively. These are gain matrices (such as positive definite diagonal matrices) for position, velocity, and feedback contact force, respectively, used to adjust the response intensity of position error, velocity error, and force error; The feedforward compensation term includes gravity compensation, Coriolis force compensation, friction compensation, etc., and can be calculated through a dynamic model.

[0258] in, These are the position control selection matrix and the force control selection matrix, set according to the task's degrees of freedom, and are the core of the force / position hybrid control; both are diagonal matrices, with diagonal elements being 0 or 1, satisfying a complementary relationship: ; This means that each degree of freedom, whether in position or force, can only be controlled by either position or force; they cannot be controlled simultaneously. For example, in a socketing task, force control is used along the socket axis, and position control is used in the perpendicular direction. The task degree of freedom refers to the number and type of independent motion directions required to complete the current task.

[0259] In other embodiments, for tasks involving point-to-point motion and trajectory tracking in barrier-free spaces that only use position or velocity closed loops, a position control mode is adopted. Based on the task requirements for safe and executable actions, along with state data, hardware layer parameters, and software layer parameters, the position control submodule is invoked to perform action-instruction mapping for the safe and executable actions, obtaining the corresponding underlying control commands. For pure speed control tasks involving mobile chassis or conveyors, a speed control mode is adopted. Based on the task requirements for safe and executable actions, along with state data, hardware layer parameters, and software layer parameters, the speed control submodule is invoked to perform action-instruction mapping for the safe and executable actions, obtaining the corresponding underlying control commands. For action tasks such as constant-force grinding that do not control position but only adjust force, a force control mode is adopted. Based on the task requirements for safe and executable actions, along with state data, hardware layer parameters, and software layer parameters, the force control submodule is invoked to perform action-instruction mapping for the safe and executable actions, obtaining the corresponding underlying control commands.

[0260] Finally, the ontology abstraction and adaptation module sends the generated low-level control commands to the robot's actuators (such as drivers or servo control systems) through the actuator interface to achieve robot control. The above steps are repeated in each control cycle to form a closed-loop feedback control until the task completion conditions are met (such as position error less than and force error less than tolerance, or time limit exceeded).

[0261] In this embodiment, the action type and task requirements (target pose or region, expected force, task tolerance, and task time limit) are obtained by parsing the abstract action semantics of safe and executable actions. Combined with the robot's joint state, contact force, end-effector pose and other state data, as well as hardware and software layer parameters, the robot adaptively determines control modes such as position, velocity, force or force-position hybrid, and calls the mapping function of the corresponding sub-module to generate low-level control instructions. In particular, force-position hybrid control is used for contact operations such as plugging and unplugging and assembly. This enables efficient, accurate and task-constrained instruction conversion from high-level skills to low-level execution in an offline environment, significantly improving the robot's adaptability and control accuracy for complex contact tasks.

[0262] The runtime resource adaptive scheduling module is used to dynamically adjust the operating configuration of the control system under the constraints of latency, power consumption, temperature rise, memory, and minimum safe control frequency.

[0263] The runtime resource adaptive scheduling module is a core component in the robot control system responsible for online optimization of resource allocation. It can monitor the robot's computing resources (such as embedded GPU, CPU, and memory), real-time performance (i.e., latency), power consumption, temperature rise, and safety control frequency in real time. Under multiple constraints such as latency, power consumption, temperature rise, memory usage, and minimum safety control frequency, it dynamically adjusts the operating configuration of the control system, dynamically selecting the most suitable operating configuration to balance task accuracy and control system stability, ensuring the robot continuously and reliably executes its tasks.

[0264] The runtime resource adaptive scheduling module takes a set of candidate running configurations as input and optimizes these configurations under constraints of computing power, latency, power consumption, temperature rise, memory, and minimum safe control frequency. It outputs the optimal target running configuration set at the current moment. Simultaneously, it monitors the operating status of the control system in real time and triggers configuration switching when the robot's resources are strained or the environment changes.

[0265] The running configurations in the candidate running configuration set and the target running configuration set include, but are not limited to, parameters such as model accuracy, sensor sampling rate, planning time domain length (i.e., the time window length for action planning), number of action generation steps (i.e., number of denoising steps), log level, and playback switch. The playback switch indicates whether to record sensor data for offline playback; the log level indicates the level of detail in the control system's running log. Specifically, the runtime resource adaptive scheduling module is used to select the current running configuration from the candidate running configuration set consisting of model accuracy, sensor sampling rate, planning time domain length, number of action generation steps, log level, and playback switch.

[0266] Specifically, the runtime resource adaptive scheduling module performs the following steps: Get multiple candidate run configuration sets, and the run configuration values ​​in the multiple candidate run configuration sets are different.

[0267] For each candidate operating configuration set, the predictive control system performs the current task with the following parameters: accuracy, latency, power consumption, temperature rise (or thermal load), memory usage, and safety margin, thereby obtaining the control system performance data corresponding to the candidate operating configuration set.

[0268] Safety margin characterizes the degree of redundancy of the control system in a given configuration regarding the likelihood of violating safety constraints (e.g., control frequency margin, torque margin). Execution accuracy characterizes the accuracy of task completion under a set of candidate operating configurations (e.g., object recognition rate, reciprocal of trajectory tracking error).

[0269] Based on the control system performance data corresponding to the candidate operating configuration set, the performance score of the control system of the candidate operating configuration combination is calculated using a performance scoring function, and the safe control frequency that the control system can achieve under the candidate operating configuration set is predicted.

[0270] Based on the control system performance score and safety control frequency of each candidate operating configuration combination, the optimal configuration is solved within the candidate operating configuration set. The candidate operating configuration set with the highest control system performance score and that meets preset performance constraints is output as the target operating configuration set. These preset performance constraints include a delay limit less than or equal to the upper limit of power consumption, a temperature rise limit (or thermal load), a memory usage limit, and a safety control frequency threshold greater than or equal to the upper limit of the lower limit of ...

[0271] The current operating configuration of the control system is adjusted based on the target operating configuration set to improve the operating performance of the control system, enhance task accuracy, and increase the stability of the control system.

[0272] The performance score of the control system can be calculated using the following performance scoring function: ; in, The system performance is scored for the control system under candidate operating configuration combinations; This refers to the normalized execution accuracy. This refers to the normalized time delay; This represents the normalized power consumption. This refers to the normalized temperature rise or heat load. This represents the normalized memory usage. This represents the normalized safety margin. These are the weights of the corresponding performance parameters.

[0273] In one embodiment, when excessive temperature rise, excessive power consumption, latency deterioration, or memory usage approaching the upper limit is detected, the runtime resource adaptive scheduling module degrades the runtime configuration in the following order: reducing the accuracy of non-critical visual models, reducing the level of non-critical logs, shortening the number of action generation iterations, shortening the planning time domain length, and reducing the refresh rate of non-critical sensors. The configurations related to the safety projection module, execution control kernel, and minimum safety control frequency are either not degraded, or are only degraded last if all other degradation measures have been implemented and the constraints are still not met. Specifically, the runtime resource adaptive scheduling module is further configured to: when excessive temperature rise, excessive power consumption, latency deterioration, or memory usage approaching the upper limit is detected, perform resource degradation in the order of reducing the accuracy of non-critical visual models, reducing the level of non-critical logs, shortening the number of action generation iterations, shortening the planning time domain length, and reducing the refresh rate of non-critical sensors, while keeping the configurations related to the safety constraint execution module and minimum safety control frequency either undegraded or degraded last.

[0274] This solution effectively ensures that the system can prioritize maintaining core security functions and minimum control frequency even when resources are scarce in an offline environment, avoiding security failure due to resource exhaustion, and significantly improving the system's real-time robustness and security reliability under high load.

[0275] The local audit module is used to record key and abnormal events during the operation of each functional module in the control system, and to generate chained audit records by combining monotonic counting and store them locally.

[0276] Specifically, the local audit module combines monotonic count values ​​to generate a chain-like audit record for each type of critical event during the operation of various functional modules of the control system, preventing rollback. These critical events include offline boundary failure events, as well as abnormal events occurring during the capability package governance process, control execution process, and each process. Examples include events such as offline capability package deployment, import, verification, installation, loading, activation, and rollback during capability package governance, and degradation, emergency stop, manual takeover, action rollback, and decommissioning during control execution. Furthermore, it also includes abnormal events occurring during the capability package governance process and control execution process. Offline boundary failure events refer to events that prevent the control chain from starting when the preset offline operating conditions are not met.

[0277] The local audit module is responsible for creating irreversible, tamper-proof, and traceable chained audit logs of critical security events during robot operation. This chained audit log includes audit records of multiple critical events of the same type. Each audit record is bound to the previous one using a chained hash structure, and a monotonic count value maintained by a local root of trust is used to prevent log replay, overwriting, or rollback attacks. Each audit record forms a digest chain, with the head typically stored in secure hardware or protected storage. This chained audit log provides reliable evidence for post-event security analysis, fault location, compliance review, and legal evidence collection.

[0278] Each audit record generated by the local audit module includes at least a summary of the previous audit record, event type, local timestamp, monotonic count value, slot identifier, version information, and result code or exception code, forming a chain of audit records to prevent rollback.

[0279] Among them, the The summary of each audit record is calculated using the following formula: ; in, For the first A summary of the audit record; This is a summary of the previous audit record; Event type; Local timestamp; This is a monotonic count value maintained by the local root of trust; The slot identifier at the time the event occurred; For relevant version information; This is either a result code or an exception code.

[0280] In one embodiment, the offline capability package governance module is further configured to perform offline cross-platform consistency verification and cross-entity adaptability verification on the offline capability package before it is put into use after installation, in order to confirm the behavioral consistency and safe executability of the same capability package on different computing platforms and different robot bodies. Only after both the cross-platform consistency verification and the cross-entity adaptability verification are passed will the control system activate the offline capability package, that is, execute the robot's offline control tasks based on the capability package in the active slot.

[0281] The core objective of the offline capability package governance module in performing offline cross-platform consistency verification and cross-robot body adaptability verification is to ensure that the same capability package exhibits a consistent fusion state, target skill path, and control command output when running on different hardware platforms (CPU architecture, operation control system, ABI, driver version) and different robot bodies (kinematic and dynamic differences), while simultaneously meeting safety constraints and task success rate requirements. Only capability packages that pass verification can be put into formal use, and the verification results are written into the local audit chain along with the installation record, forming traceable compliance evidence. This module is a key line of defense for ensuring the portability, determinism, and security of robot software, and is particularly suitable for heterogeneous clusters or long-term operation and maintenance scenarios.

[0282] Offline cross-platform consistency verification specifically includes: The architecture information of all computing platforms (i.e., computing environments) within the robot body is acquired to form a set of platforms to be verified. This set includes the architecture information of multiple computing platforms. The architecture information of a computing platform includes four dimensions: instruction set architecture, operation control system or real-time operation control system, application binary interface, and driver configuration set. The application binary interface determines the function calling convention and control system call number; the driver configuration set contains the driver versions and parameters for sensors, actuators, and communication interfaces. The architecture information of the computing platform characterizes the computing environment within the control system and directly affects the compilation, linking, and runtime behavior of the offline capability package. The set of platforms to be verified should cover all variations of the target deployment environment.

[0283] Based on the architecture information of the computing platform, the computing platform to be tested is determined. A unified offline test suite is used to perform control system operation tests on the computing platform based on offline capability packages, obtaining the output data of each computing platform. The output consistency deviation between the two computing platforms is calculated based on their output data. If the output consistency deviation is less than or equal to a deviation threshold, the two computing platforms are determined to have passed cross-platform consistency verification; if the output consistency deviation is greater than the deviation threshold, the two computing platforms are determined to have failed consistency verification.

[0284] The output data of the computing platform includes at least the fusion state, the target skill path, and control commands. Therefore, the output consistency deviation includes state consistency deviation, planning consistency deviation, and action consistency deviation. State consistency deviation, planning consistency deviation, and action consistency deviation can be calculated using the following formulas: ; ; ; in, They represent computing platforms. With computing platform State consistency deviation, action consistency deviation, and planning consistency deviation; The number of test samples; They represent computing platforms respectively. Computing platform In the Fusion status under each test sample; Represents computing platform Computing platform In the Control commands under each test sample; Represents computing platform Computing platform In the The target skill path under each test sample. It is the Euclidean norm; For weighted norm; It is a positive definite weight matrix, which is usually consistent with the cost matrix in the safety constraint enforcement module.

[0285] Specifically, if the state consistency deviation, planning consistency deviation, and action consistency deviation are all less than or equal to their respective deviation thresholds, the two computing platforms are determined to have passed the consistency verification. Conversely, if any one of the state consistency deviation, planning consistency deviation, or action consistency deviation exceeds its respective deviation threshold, the two computing platforms are determined to have failed the consistency verification.

[0286] Then, all computing platforms are aggregated. If all computing platforms pass the consistency verification, the offline capability package is determined to have passed the cross-platform consistency verification. If any computing platform fails the consistency verification, the offline capability package is determined to have failed the cross-platform consistency verification.

[0287] This solution acquires architectural information such as instruction set architecture, operating system, application binary interface, and driver configuration of all computing platforms to form a set of platforms to be verified. It then uses a unified offline test set to run offline capability packages on each platform, outputting data such as fusion status, target skill path, and control commands. The solution then calculates the consistency deviation of cross-platform output status, planning, and action generation. Verification is only passed when the deviation does not exceed a threshold. This effectively ensures that the same capability package behaves consistently and is securely executable on different computing platforms in an offline environment, avoiding operational deviations or anomalies caused by platform architecture differences. This significantly improves the cross-platform compatibility and deployment reliability of capability packages.

[0288] Specifically, cross-ontology compatibility verification includes: Obtain the set of robot bodies to be verified. This set includes the hardware and software parameters of multiple robot bodies. Obtain the parameter information of the robot bodies and summarize the different types of robot bodies to obtain the set of robot bodies to be verified. The specific content of the hardware and software parameters is described above and will not be repeated here.

[0289] For each robot body, multiple independent tests are conducted based on its hardware and software parameters to determine the safety pass rate and task success rate when executing the offline capability package. If both the safety pass rate and task success rate are greater than or equal to their respective preset thresholds, the robot body is determined to have passed the body compatibility verification. Conversely, if either the safety pass rate or the task success rate is less than the corresponding preset threshold, the robot body is determined to have failed the body compatibility verification.

[0290] Specifically, based on multiple test data, the number of tests required for the robot to satisfy all safety constraints and the number of tests required to complete the target task when executing the offline capability package can be determined. Then, the safety pass rate can be obtained by dividing the number of tests that satisfy all rigid safety constraints by the total number of tests; the task pass rate can be obtained by dividing the number of tests that complete the target task by the total number of tests.

[0291] The aforementioned independent tests can employ one or more of the following methods: offline replay data, standard test vectors, simulation execution, and actuator no-load verification.

[0292] Next, all robot bodies are aggregated. If all robot bodies pass the body adaptability verification, the offline capability package is determined to have passed the cross-body adaptability verification. If any robot body fails the body adaptability verification, the offline capability package is determined to have failed the cross-body adaptability verification.

[0293] This solution acquires the hardware and software parameters of the robot body to be verified, performs multiple independent tests on each body (using offline playback, standard test vectors, simulation operation, or actuator no-load verification), and calculates the safety pass rate and task success rate. Only when both reach a preset threshold is the cross-body adaptability verification considered passed. This effectively ensures the safety and task effectiveness of the same capability package when executed on different robot bodies in an offline environment, avoiding safety constraint violations or task failures caused by differences in body parameters, and significantly improving the general deployment capability and operational reliability of capability packages across multiple robot platforms.

[0294] Finally, the control system activates the offline capability package only after it passes cross-platform consistency verification and cross-entity adaptability verification. This triggers the relevant modules in the core control layer to execute the robot's offline control tasks based on the offline capability package in the active slot. If the offline capability package fails cross-platform consistency verification or cross-entity adaptability verification, it is refused activation and automatically rolled back to the last stable active slot. Furthermore, after obtaining the cross-platform consistency verification results and cross-entity adaptability verification results, the verification results are written to the local audit log along with the capability package installation record.

[0295] In this embodiment, by performing offline cross-platform consistency verification and cross-entity adaptability verification after the capability package is installed and before it is put into use, it is ensured that the same capability package behaves consistently and is safe and executable on different computing platforms and different robot bodies. This effectively avoids operational anomalies, performance degradation or security risks caused by differences in platform architecture or entity parameters in the offline environment, and significantly improves the versatility and deployment reliability of the capability package.

[0296] The unified anomaly handling module receives anomaly information reported by other functional modules and, based on the received anomaly information and preset anomaly classification and state transition rules, performs global state switching and anomaly classification handling to improve the recoverability of the control system in offline environments. Specifically, the unified anomaly handling module switches between normal, degraded, hold or pause, emergency stop, and rollback states according to the anomaly type.

[0297] Specifically, the pre-defined anomaly classification and state transition rules include: defining a system fault set covering the entire stack of perception, planning, control, hardware, and software; and classifying the operating states of the control system, including abnormal states such as degraded state, hold state, emergency stop state, and rollback state, as well as normal state. Through state transition functions, abnormal information in the system fault set is mapped to the corresponding target operating state, and executable processing strategies are formulated for each type of abnormal state, including predictive compensation, channel gating, resource degradation, backup templates, emergency stop, and rollback. These rules ensure that the robot can automatically and safely respond to various anomalies, avoid loss of control, and recover to a normal operating state when conditions permit.

[0298] Among them, the system fault set It contains the following types of exception information: ; in, This indicates a sensor malfunction, meaning a sensor malfunction was detected in the robot itself. This indicates a time synchronization anomaly, meaning there is an error in the time alignment process between the multimodal sensing and preprocessing modules. This indicates that the state estimate is diverging, meaning that the trace of the fused state covariance matrix calculated by the fused state estimate and the world representation module is greater than a preset divergence threshold. This indicates that path planning has failed, meaning that the target skill path planning in the skill graph planning module has timed out or failed. This indicates that action generation has timed out or failed, and that the candidate action sequence generation in the uncertainty assessment and action sequence generation module has timed out or failed. This indicates that safe projection is not feasible, meaning that the safety constraint execution module has not solved for a safe and executable action; This indicates an actuator or drive malfunction, or a joint position exceeding its limit. This indicates abnormal temperature rise or power consumption, meaning that the runtime resource adaptive scheduling module has detected abnormal temperature rise or power consumption in the control system. This indicates an anomaly in capability package governance: anomalies in capability package verification / installation / switching, meaning that the capability package governance module has encountered anomalies in the governance process, such as abnormal verification, installation, switching, and the status observed in the post-switching observation window. This indicates an audit chain anomaly, meaning that the chain of audit records in the local audit module is abnormal.

[0299] The state transition function is expressed by the following formula: ; in, The target state to be migrated; Indicates a normal state; Indicates a downgraded status; Indicates a state of holding or pausing; Indicates an emergency stop status; Indicates a rollback state; Indicates a collision risk warning sign; Indicates a rollback request flag; This indicates the current abnormal information of the control system.

[0300] As shown in the state transition function above, when no safe executable action is found, or the actuator malfunctions, or there is a risk of collision, the control system enters an emergency stop state; when the capability package governance or audit chain is abnormal, the control system enters a rollback state; when the robot's local sensors malfunction, the data time synchronization is abnormal, the state estimation diverges, or the temperature rise or power consumption is abnormal, the control system enters a degraded state; when path planning fails or action generation fails, the control system enters a hold state or a pause state; when the control system is running without faults, the control system remains in a normal state.

[0301] It is important to understand that the anomaly detection, global state switching, and anomaly hierarchical handling process in this embodiment adopts a hybrid architecture of distributed anomaly detection by each functional module, centralized state switching and hierarchical handling by a unified anomaly handling module, and unified traceability by a local auditing module. This reduces the functional disorder that may be caused by each functional module performing anomaly detection and handling independently, and ensures the security and stability of system operation.

[0302] In other words, in practical applications, each of the above functional modules is equipped with an anomaly detection submodule, which is used to detect its own operational anomalies, obtain anomaly information, and report it to the unified anomaly handling module. The anomaly information detected by each functional module is described above.

[0303] The unified exception handling module is specifically used for: receiving exception information reported by each functional module and recording the received exception information into the system fault set; based on the exception information in the system fault set at the current moment, using the above-mentioned state transition function to map and obtain the target operating state to be switched; and based on the target operating state and the type of the exception information (i.e. exception type), generating and executing a global state switching and exception classification handling strategy.

[0304] Specifically, based on a global state switching and anomaly classification handling strategy, the global state can be switched to the target running state, and control instructions for the corresponding functional modules can be generated to trigger the corresponding functional modules to execute the control instructions, thereby realizing global state switching and anomaly classification handling. When there are multiple target running states to be switched at the current time, the state switching and anomaly classification handling strategies corresponding to each target running state are executed sequentially according to their priority. Specifically, when there are multiple target running states to be switched at the current time, the corresponding state switching and anomaly classification handling strategies are executed sequentially according to the priority of emergency stop state, rollback state, hold or pause state, and degraded state.

[0305] Each functional module performs operational anomaly detection through its own anomaly detection submodule and reports the detected anomaly information to the unified anomaly handling module. In addition, upon receiving the control command generated by the unified anomaly handling module based on the corresponding handling strategy, the module executes the control command to achieve hierarchical processing.

[0306] Specifically, the strategies for handling global state transitions and anomaly classifications include: If a capability package governance anomaly is received from the capability package governance module, such as capability package verification / installation / switching failure, or an audit chain anomaly is received from the local audit module, such as audit chain breakage or version rollback anomaly, the system status will be controlled to enter rollback state, triggering the capability package governance module to refuse to activate offline capability packages, lock the update channel, and automatically roll back to the previous stable slot.

[0307] If the system receives information from the safety constraint execution module that the safety projection is not feasible or that there is a collision risk, or from the ontology abstraction adaptation module (or the actuator or drive control system) that is abnormal, such as an actuator or drive system failure or a joint position exceeding the limit, the control system will enter an emergency stop state and immediately send an emergency stop or torque suppression command to the actuator or drive control system to trigger the actuator or drive control system to take emergency stop actions.

[0308] If a path planning timeout or failure message is received from the skill graph planning module, or an action generation timeout or failure message is received from the uncertainty assessment and action sequence generation module, the control system enters a hold or pause state, triggering the uncertainty assessment and action sequence generation module to call the local skill template to generate actions, and triggering the actuator or drive control system to execute the previous safety action to maintain or triggering the local obstacle avoidance controller to avoid obstacles.

[0309] If a sensor first-type fault information is received from the multimodal sensing and preprocessing module, such as short-term frame loss or temporary data unavailability, the multimodal sensing and preprocessing module is triggered to maintain continuous data control of the sensor by predicting the state, and the fusion state estimation and world characterization module is triggered to reduce the weight of the corresponding observation channel of the sensor. If a sensor second-type fault information is received from the multimodal sensing and preprocessing module, such as the number of consecutive frame losses of the sensor exceeding a threshold, the system state enters a degraded state, the multimodal sensing and preprocessing module is triggered to block or stop using the observation channel corresponding to the sensor, and the fusion state estimation and world characterization module is triggered to reduce or set the fusion weight of the observation channel to zero.

[0310] If a time synchronization anomaly is received from the multimodal sensing and preprocessing module, such as when the time synchronization deviation of the data exceeds the upper limit, the control system enters a degraded state, triggering the multimodal sensing and preprocessing module to discard the abnormal sensor data and re-align the acquired data. Additionally, the multimodal sensing and preprocessing module can be triggered to reduce the sampling rate of the corresponding observation channel of that sensor.

[0311] If state estimation divergence information is received from the fusion state estimation and world representation module, such as the trace of the fusion state covariance matrix being greater than the state divergence threshold, the control system enters a degraded state, triggering the map-related modules in the control system to perform positioning reset, local map reconstruction, or enter a low-speed safety mode.

[0312] If the system receives abnormal status information from the runtime resource adaptive scheduling module, such as abnormal temperature rise, power consumption, latency, or memory usage of the control system, the control system enters a degraded state, triggering the runtime resource adaptive scheduling module to perform resource degradation operations. In addition, it can also trigger the control system to shut down non-critical modules.

[0313] Through distributed anomaly detection and centralized state switching, capability package anomalies, security anomalies, planning anomalies, perception anomalies, and resource anomalies can be handled in a graded manner.

[0314] In one specific embodiment, the embodied intelligent robot can be an indoor mobile inspection robot. Taking the application of this control system to this indoor mobile inspection robot as an example, an offline control method for the indoor mobile inspection robot is provided. This control system, in a completely offline environment, controls the indoor mobile inspection robot to complete control processes such as indoor map construction and localization, task navigation, target detection and offline anomaly storage, autonomous obstacle avoidance, and safe speed limiting. The specific control process of the control system includes at least the following: After the control system of the indoor mobile inspection robot is powered on, the external communication isolation control module is first activated, all wireless and wired external communication interfaces are shut down, and after verifying that the offline boundary is established, the local root of trust is read from the local root of trust in the monotonic counting module, and the offline capability package management module is activated to load the inspection capability package in the active slot. The inspection capability package includes a laser SLAM model, a visual target detection model, an inspection skill map, safety constraint parameters, and robot body adaptation parameters.

[0315] When a robot version update is required, an offline capability package can be imported via offline media. The imported offline capability package is then loaded using the offline capability package governance module to complete the version update. Specifically, the offline capability package is imported via an encrypted USB drive, containing the updated map and detection model. After import, it first enters an isolation zone, where the system sequentially performs signature verification, hash tree integrity verification, version rollback prevention verification, dependency integrity verification, and hardware compatibility verification. Once all verifications pass, it is installed in an inactive slot. Offline replay testing and regression testing are performed in the inactive slot. Once the pass rate reaches a preset threshold, an atomic switch is performed at the control cycle boundary or during task idle periods. After the switch, a preset observation window is opened. If the control latency exceeds a preset latency threshold, or the security projection pass rate falls below a preset pass rate threshold, the system automatically rolls back to the previous stable version. The local audit module records all events from capability package import to installation and switchover, forming a chain of audit records.

[0316] During system operation, the robot's battery level and temperature are monitored in real time by the runtime resource adaptive scheduling module. When the battery level is below 20% or the temperature is above 60°C, the system automatically switches to a low-power configuration, reducing the accuracy of the visual inspection model and the sampling rate of non-critical sensors, but ensuring that the safety control frequency is not lower than the preset minimum safety control frequency.

[0317] In densely populated areas, the uncertainty assessment and action sequence generation module automatically increases the noise sampling range and generates more candidate detour paths due to the increase in dynamic obstacles. Subsequently, the safety projection layer constrains the robot's speed and safety distance, ensuring that the human-machine safety distance is not less than the preset safety distance threshold.

[0318] In one specific embodiment, the embodied intelligent robot can also be an industrial manipulator robot with a six-axis robotic arm. Taking the application of this control system to an industrial manipulator robot with a six-axis robotic arm as an example, an offline control method for an industrial manipulator robot is provided. This control system controls the robotic arm of the industrial manipulator robot to perform tasks such as workpiece gripping, handling, assembly, and button operation in an environment without an external network connection.

[0319] In the operation of this industrial robot, the task understanding and condition vector construction module encodes the fused state, world representation, task objective, safety constraint representation, and ontological constraint representation into a unified condition vector. This unified condition vector includes information such as the end-effector pose, contact force threshold, no-entry zone constraint, and joint limit constraint. The contact force threshold and no-entry zone constraint belong to the safety constraint representation, while the joint limit constraint belongs to the ontological constraint representation. The uncertainty assessment and motion sequence generation module outputs candidate motion sequences, including candidate end-effector trajectories and force control targets. Then, the safety constraint execution module performs projection optimization on the candidate end-effector trajectories and force control targets based on constraints such as collision risk, joint limits, and torque limits, thereby obtaining a safe and executable motion sequence.

[0320] The body abstraction and adaptation module maps safe and executable action sequences into low-level control instructions for industrial robots. For precision operation tasks such as assembly, a force-position hybrid control strategy is adopted to ensure assembly accuracy in the position control direction and ensure that the contact force does not exceed the preset contact force threshold in the force control direction, thereby reducing the risk of damaging the workpiece or the robot body.

[0321] The offline capability package governance module performs cross-ontology consistency verification on capability packages before execution. This allows high-level skills in the same grasping-transferring-placement category to be automatically mapped onto robotic arms with different joint configurations, requiring only the configuration of corresponding kinematic parameters, drive protocols, and joint limit parameters. The offline capability package governance module can also perform offline updates of tool parameters and control gains. Before updating, cross-ontology consistency verification is performed to ensure that the safety pass rate and task success rate on the target robotic arm reach their respective preset thresholds. In one example, the safety pass rate threshold and task success rate threshold can be set according to the task safety level. If an anomaly occurs during the update process, the system automatically rolls back to the previous stable version.

[0322] As can be seen, the full lifecycle offline embodied intelligent control system provided by the embodiments of the present invention can be applied to different types of robots. By constructing an offline closed-loop control link with communication isolation, trusted verification deployment, offline updates, task execution, and local auditing, the control system effectively solves the problem that the robot control system cannot achieve full lifecycle offline operation due to reliance on external devices. It realizes the continuous and stable operation and safe evolution of the robot in complex environments, and improves the stability and reliability of the robot.

[0323] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0324] It should be noted that the information interaction and execution process between the above-mentioned control system and the method of the present invention are based on the same concept as the method of the present invention. For the specific implementation process and technical effects of the method of the present invention, please refer to the embodiment section of the above-mentioned device / module, which will not be repeated here.

[0325] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of the present invention. The specific working process of the units and modules in the above control system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0326] This invention also provides an electronic device, such as... Figure 7 As shown, the electronic device includes: at least one processor, a memory, a sensor interface, an actuator interface, and an offline media interface, as well as a computer program / program instructions stored in the memory and executed by the processor; when the processor executes the computer program / program instructions, it implements the aforementioned full lifecycle offline embodied intelligent control method. Those skilled in the art will understand that... Figure 7 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or combine certain components, or different components. For example, an electronic device may also include input / output devices, network access devices, buses, etc.

[0327] The processor mentioned above can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. Memory can be an internal storage unit of the electronic device, such as a hard drive or RAM. Memory can also be an external storage device of the electronic device, such as a plug-in hard drive, smart memory card, secure digital card, flash memory card, etc. Furthermore, memory can include both internal and external storage units of the electronic device.

[0328] This invention also provides a readable storage medium storing a computer program / program instructions thereon, which, when executed by a processor, implements the above-described full lifecycle offline embodied intelligent control method.

[0329] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention can implement all or part of the processes in the methods described above by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the functions of the aforementioned functional modules. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include entities or devices capable of storing computer program code, recording media, computer memory, read-only memory, random access memory, USB flash drive, portable hard drive, magnetic disk, or optical disk, etc., and other non-transitory storage media.

[0330] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail or in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present invention.

[0331] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0332] The above-described embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the embodiments of the present invention have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the embodiments of the present invention.

Claims

1. A full life cycle offline embodied intelligent control system applied to an embodied intelligent agent, the embodied intelligent agent having a sensor, an actuator and a body control interface, characterized in that, The control system is configured to establish an external communication isolation boundary during the startup phase or before the loading of runtime resources, and to perform capability resource governance and local closed-loop control within the external communication isolation boundary. The local closed-loop control includes: generating control actions based on locally available capability resources, according to at least one of observation data, task objectives, environmental characterization, and constraint information; and outputting low-level control commands after performing security processing and ontology adaptation on the control actions. The control system includes: The external communication isolation control module is used to keep the communication path to external networks, external servers, external inference services or external application interfaces in a closed, shielded, physically isolated, logically isolated or unreachable state, and to allow the subsequent local control chain to start when the external communication isolation boundary is verified to meet the preset offline conditions. The local trust root and monotonic counting module is used to store at least one of the following: local trust root information, monotonic counting information, equivalent rollback prevention status information, and slot or resource space information. The local trust root information is used to support source trust verification and / or integrity verification of capability resources. The monotonic counting information or equivalent rollback prevention status information is used to support version rollback prevention and / or audit rollback prevention. The slot or resource space information is used to support the activation, switching, and / or rollback of capability resources. The offline capability package management module is used to receive candidate offline capability packages through offline media, and / or determine candidate capability resources from locally preset, historically installed, or currently inactive capability resources; perform trusted verification on the candidate offline capability packages or candidate capability resources, and after the verification is passed, install, register, or load them into at least one of non-current running area, candidate resource space, or inactive slot; and enable the candidate capability resources or switch the candidate slot to an active slot when a preset activation condition is met, or restore to the previous stable capability resource or the previous stable active slot when a rollback condition is triggered. The offline control execution module is used to load capability resources from currently enabled capability resources, activity slots or local storage, obtain ontology observation data and / or environmental observation data of the embodied intelligent agent, construct fusion state and / or environmental representation, and form a unified control representation based on at least one of the observation data, fusion state, task objective, environmental representation, security constraint representation and ontology constraint representation, and generate candidate control actions according to the unified control representation; The safety constraint and ontology adaptation module is used to perform safety processing on the candidate control actions to satisfy preset safety constraints, obtain safe executable actions, and map the safe executable actions into low-level control instructions that adapt to the ontology control interface of the embodied intelligent agent. The control system further includes at least one of a runtime resource adaptive scheduling module, a unified exception handling module, and a local auditing module.

2. The control system of claim 1, wherein, The external communication isolation control module is specifically used to: determine whether the external communication isolation boundary is established based on the external routing status, the monitoring interface status facing the external address, the external application interface enable status, and / or the external communication hardware status; and allow the control system to enter the capability package selection, runtime resource loading, and control loop startup phase when the external communication path is in a closed, shielded, isolated, or unreachable state.

3. The control system of claim 1, wherein, The local trust root and monotonic counting module includes at least one of the following: root public key, integrity verification benchmark, version count, audit count, equivalent rollback prevention status, active slot identifier, inactive slot identifier, candidate resource space identifier, and stable resource identifier; the version count or equivalent rollback prevention status is used to perform version rollback prevention verification on offline capability packages or capability resources, the audit count is used to generate a chain of audit records for rollback prevention, and the active slot identifier, inactive slot identifier, candidate resource space identifier, and stable resource identifier are used to support the activation, atomic switching, recovery, and / or rollback of capability resources.

4. The control system of claim 1, wherein, The offline capability package includes a capability package list, a capability resource set, and signature information; the capability resource set includes one or more of the following: model files, policy files, configuration files, map resources, dependency list, interface description files, and test vector set; the capability package list includes one or more of the following: capability package identifier, version number, anti-rollback index, supported computing platform set, supported embodied intelligent agent ontology set, file hash table, validity period, required dependency set, security policy parameter set, and minimum resource requirement set.

5. The control system of claim 1, wherein The offline capability package governance module is specifically used for: importing the offline capability package or candidate capability resources into the isolation area, candidate resource space, or inactive slot, parsing the capability resource list, and performing one or more trusted verifications including signature verification, integrity verification, version rollback prevention verification, dependency verification, compatibility verification, validity period verification, space verification, and security policy verification; after the trusted verification meets the preset installation conditions, installing, registering, or loading the offline capability package or candidate capability resources into at least one of the inactive slot, candidate resource space, or non-current running area; determining the component loading order based on at least one of component dependency, loading priority, security level, or running requirements, and loading components according to the component loading order; Perform at least one local verification on the loaded candidate resources, including local self-check, offline replay test, regression test, simulation test, and no-load verification; and write the governance results to the local audit module.

6. The control system according to claim 5, characterized in that, The preset activation conditions include at least one of the following: the verification result of the trusted verification is valid, the key component is successfully loaded, and the verification result of the local verification meets a preset threshold. The offline capability package governance module is also used to monitor at least one of the following operating indicators in a preset observation window after enabling candidate capability resources or performing slot switching: anomaly rate, number of abnormal events, control latency, thermal load, power consumption, memory usage, stable control pass rate, and security processing pass rate. When the operating indicator does not meet the corresponding threshold conditions, it automatically restores or rolls back to the previous stable capability resource or the previous stable active slot.

7. The control system according to claim 1, characterized in that, The offline control execution module includes a multimodal perception and preprocessing module; the multimodal perception and preprocessing module is used to acquire at least one of visual, depth, laser, inertial, force, tactile and encoder observation data, and to perform at least one of time synchronization, coordinate registration, missing data repair, noise estimation, anomaly gating and outlier removal on the acquired observation data.

8. The control system according to claim 1, characterized in that, The offline control execution module includes a fusion state estimation and world representation module; the fusion state estimation and world representation module is used to perform state estimation based on preprocessed observation data, obtain the fusion state, and construct or update the world representation; the world representation includes at least one of the following: occupied space representation, semantic object representation, and topological relationship representation.

9. The control system according to claim 1, characterized in that, The offline control execution module includes a task understanding and condition vector construction module. This module projects, encodes, or encapsulates at least a portion of the fused state, task objective, world representation, security constraint representation, and ontology constraint representation into a unified control representation through at least one of linear transformation, embedding mapping, rule encoding, or structured encapsulation. The unified control representation includes condition vectors, structured state representations, rule-encoded representations, or combinations thereof, and is output to the skill planning process and / or action generation process, so that the skill planning results, action generation results, security processing results, and ontology adaptation results are transmitted based on the same unified control representation.

10. The control system according to claim 1, characterized in that, The offline control execution module includes a skill graph planning module; the skill graph planning module is used to search for target skill paths in a preset skill graph based on the unified control representation. The preset skill graph includes skill nodes and switchable relationships between skill nodes. The search process determines the path cost of candidate skill paths based on at least one of the following: preconditions, postconditions, switching costs, target matching degree, risk costs, energy consumption costs, and uncertainty costs.

11. The control system according to claim 1, characterized in that, The offline control execution module includes an uncertainty assessment and action sequence generation module. This module determines uncertainty information based on at least one of fusion state uncertainty, world representation uncertainty, dynamic environment change intensity, and location recognition confidence. It then adjusts the action generation parameters according to the uncertainty information to generate candidate action sequences corresponding to the target skill path. The action generation parameters include at least one of sampling range, sampling noise, denoising step size, planning time domain length, and action generation iteration steps. When runtime resources are insufficient, latency constraints tighten, or action generation fails to converge within a limited time, the action generation mode is downgraded to a deterministic generation method based on skill templates, rule bases, trajectory bases, and / or local optimization.

12. The control system according to claim 1, characterized in that, The safety constraint and ontology adaptation module includes a safety constraint execution module; the safety constraint execution module is used to perform projection solving or constraint correction on candidate control actions to obtain safe and executable actions; the preset safety constraints include at least one of collision safety distance constraints, contact force constraints, velocity constraints, torque constraints, restricted area constraints, and joint limit constraints; when no feasible solution exists, the safety constraint execution module triggers at least one of the following actions according to preset priority: emergency stop, maintaining posture, reverting to the previous safe action, and entering a safety degradation mode.

13. The control system according to claim 1, characterized in that, The safety constraint and ontology adaptation module includes an ontology abstraction adaptation module. This module maps safe, executable actions to low-level control commands based on at least one of the following: ontology type, ontology kinematic parameters, ontology dynamic parameters, drive interface description, and control mode. The low-level control commands include one or more of movement control commands, joint control commands, end-effector control commands, force control commands, and combinations thereof. In assembly, insertion / removal, pressing, force-controlled grasping, or other contact tasks, the ontology abstraction adaptation module performs position control, speed control, force control, or a force-position hybrid control to meet task accuracy requirements in the position control direction and / or limit the contact force to no more than a preset threshold in the force control direction.

14. The control system according to claim 1, characterized in that, The control system includes a runtime resource adaptive scheduling module, which selects the current running configuration from a set of candidate running configurations consisting of one or more of the following: model accuracy, sensor sampling rate, planning time domain length, action generation steps, log level, and playback switch. When excessive temperature rise, excessive power consumption, latency deterioration, or memory approaching the upper limit are detected, at least one of the following non-critical visual model accuracy, non-critical log level, action generation iteration steps, planning time domain length, and non-critical sensor refresh rate is adjusted according to a preset resource degradation strategy. The system ensures that the configurations related to safety constraint execution, execution control kernel, and minimum safety control frequency are not degraded, or is only degraded in the final step if the preset constraints are still not met after all non-critical running configurations have been degraded.

15. The control system according to claim 1, characterized in that, The offline capability package governance module is also used to perform offline cross-platform consistency verification and / or cross-ontology adaptability verification before the offline capability package is put into use; wherein, the offline cross-platform consistency verification is used to verify whether the fusion state, skill path or control command output of the same offline capability package on different computing platforms meets the consistency threshold, and the cross-ontology adaptability verification is used to verify whether the security pass rate and / or task success rate of the same offline capability package on different embodied intelligent agent ontologies meet the preset threshold.

16. The control system according to claim 1, characterized in that, The control system includes a unified anomaly handling module. The anomaly information received by the unified anomaly handling module includes at least one of the following: sensor failure, time synchronization anomaly, state estimation divergence, planning failure, action generation timeout, safety projection infeasibility, actuator or drive failure, temperature rise or power consumption anomaly, capability package verification anomaly, capability package installation anomaly, capability package switching anomaly, and audit chain anomaly. The unified anomaly handling module is used to switch between normal state, degraded state, hold or paused state, emergency stop state, and rollback state according to the anomaly type.

17. The control system according to claim 1, characterized in that, The control system includes a local audit module. Each chain audit record generated by the local audit module includes at least one of the following: summary of the previous audit record, event type, local timestamp, monotonic count value, slot identifier, version information, and result code or exception code. The local audit chain is formed through hash chain binding to prevent tampering and rollback.

18. A full lifecycle offline embodied intelligent control method, applied to the control system of an embodied intelligent agent, characterized in that, include: An external communication isolation boundary is established during the startup phase of the control system or before the loading of operating resources, and subsequent local control chains are allowed to start when the external communication isolation boundary meets preset offline conditions. Read local trusted state information, which includes at least one of local trust root information, monotonic count information, equivalent anti-rollback state information, and slot or resource space information. When a candidate offline capability package imported via offline media is received, or when a candidate capability resource that is locally pre-installed, historically installed, or currently not enabled is identified, a trusted verification is performed on the candidate offline capability package or the candidate capability resource. After the trusted verification meets the preset installation conditions, the candidate offline capability package or candidate capability resource is installed, registered or loaded into at least one of the candidate resource space, non-current running area or inactive slot. The loading order is determined according to at least one of the dependency relationship, loading priority, security level or running requirements, and local verification is performed on the loaded candidate resources. When the candidate resource meets the preset activation conditions, the candidate resource is enabled or the candidate slot is switched to the active slot, and when the recovery or rollback conditions are triggered, it is restored to the previous stable capability resource or the previous stable active slot. Load the required capability resources for the current operation from currently enabled capability resources, active slots, or local storage; Acquire ontological observation data and / or environmental observation data of the embodied intelligent agent, and construct a fused state and / or environmental representation based on the observation data; A unified control representation is constructed based on at least one of the observation data, fusion state, mission objective, environmental characterization, security constraint representation, and ontology constraint representation. Candidate control actions are generated based on the unified control representation; The candidate control actions are processed for safety to obtain safe and executable actions, and the safe and executable actions are mapped to low-level control instructions; During operation, perform at least one of the following: runtime configuration adjustment, anomaly handling, and local auditing.

19. The method according to claim 18, characterized in that, Performing trusted verification on the offline capability package includes: after importing the offline capability package into the isolation area, performing at least one of the following: signature verification, integrity verification, version rollback prevention verification, dependency verification, compatibility verification, validity period verification, space verification, and security policy verification; after the trusted verification meets the preset installation conditions, loading the components in the offline capability package based on the dependency graph topology sorting, and performing at least one of the following local verifications: local self-test, offline replay test, and regression test.

20. The method according to claim 18, characterized in that, Constructing the unified control representation includes: projecting, encoding, or encapsulating at least a portion of the fusion state, task objective, world representation, security constraint representation, and ontology constraint representation through at least one of linear transformation, embedding mapping, rule encoding, or structured encapsulation to obtain the unified control representation; the unified control representation includes condition vectors, structured state representations, rule-encoded representations, or combinations thereof; and inputting the unified control representation into the skill planning process and / or action generation process to generate a target skill path and / or a candidate action sequence corresponding to the target skill path.

21. The method according to claim 18, characterized in that, The safety handling of the candidate control action includes: performing projection solution or constraint correction on the candidate control action to satisfy at least one of the following constraints: collision safety distance constraint, contact force constraint, speed constraint, torque constraint, restricted area constraint and joint limit constraint; when no feasible solution exists, triggering at least one of the following actions: emergency stop, maintaining posture, reverting to the previous safety action, or entering a safety degradation mode.

22. A fully lifecycle offline embodied intelligent robot or embodied intelligent device, characterized in that, It includes an embodied intelligent agent body and a full lifecycle offline embodied intelligent control system according to any one of claims 1 to 17.

23. An electronic device, characterized in that, It includes a processor, a memory, a sensor interface, an actuator interface, and an offline media interface, wherein the memory stores program instructions executable by the processor, which, when executed, implement the method according to any one of claims 18 to 21.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 18 to 21.

25. A computer program product, characterized in that, It includes a computer program or program instructions that, when executed by a processor, implement the method according to any one of claims 18 to 21.

Images