Characteristic network topology structure-based intelligent multi-mode perception robustness enhancement method
By constructing a sparse and robust feature network and utilizing community structure and adaptive threshold parameters, the robustness problem of multimodal perception technology under sensor failure or noise interference is solved, enabling embodied intelligent robots to achieve efficient and stable perception and operation in complex environments.
Patent Information
- Application Number
- CN202511884019.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-04-10
AI Technical Summary
Existing multimodal sensing technologies are fragile under sensor failure or noise interference, lack structured interpretation and robust mechanisms, and cannot effectively utilize semantic redundancy between modalities, leading to recognition errors and control divergence.
By structuring multimodal data into a dynamic feature network, introducing adaptive threshold parameters and community repair mechanisms, a sparse and robust feature network is constructed. The community structure is used to enhance the stability of the system and achieve information repair.
It improves the perception robustness and operational accuracy of embodied intelligent robots in complex environments, and enhances the system's operational efficiency and stability under sensor failure or noise interference.
Smart Images
Figure CN121835740A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and robot control technology, specifically to a multimodal perception method for workstation-level embodied intelligent robots, and particularly to a method that utilizes complex network theory to construct a characteristic topology structure and enhances the robustness of the perception system under sensor failure or noise interference through a community redundancy mechanism. Background Technology
[0002] With the rapid development of embodied intelligence technology, robotic systems are gradually moving out of enclosed spaces and into open, dynamic, and unpredictable complex environments to perform tasks. In these scenarios, single-modal perception often has limitations. For example, vision is easily affected by changes in lighting and smoke or dust, tactile perception is only effective upon contact, and force perception is easily interfered with by mechanical vibrations. Therefore, multimodal perception systems that integrate vision, touch, hearing, and proprioception have become standard features of embodied intelligent agents.
[0003] However, existing multimodal perception technologies mainly rely on deep neural networks for end-to-end feature fusion (such as simple concatenation fusion or attention mechanism fusion). This black-box fusion method suffers from significant perceptual fragility issues:
[0004] 1. Lack of structured explanation
[0005] Existing models implicitly assume that all modes are available and equally important at all times. Once a mode fails due to physical occlusion (such as oil blocking the lens), electromagnetic interference (causing sensor signal loss), or equipment failure, the entire fused feature space will be drastically distorted, leading to recognition errors or control divergence.
[0006] 2. Lack of robustness mechanism
[0007] Traditional fusion methods fail to effectively utilize semantic redundancy between modalities. For example, in a screw-tightening scenario, "visually seeing the screw rotate" and "sensing a change in torque" are strongly correlated. If vision fails, theoretically, the visual state should be inferred from the force perception, but existing end-to-end models lack such explicit topology inference and repair mechanisms.
[0008] 3. Lack of a structured systems perspective
[0009] Existing robustness research largely focuses on feature extraction or decision-making, lacking an examination of the perception process from a system topology perspective. In fields such as the Industrial Internet of Things (IIoT), research has shown that network topology (e.g., scale-free properties, community structure) plays a decisive role in system resilience. A few key nodes in the network support overall connectivity. If this theory can be introduced into the field of embodied intelligent sensing, modeling heterogeneous features as complex networks, and enhancing the system's intrinsic stability by optimizing the network topology, it would be a safer, more efficient, and interpretable new approach than information repair.
[0010] Therefore, there is an urgent need for a method that can model heterogeneous multimodal features into a unified network and utilize the network's community structure and topological redundancy to achieve adaptive information repair, so as to improve the robot's survivability and operational accuracy in complex industrial environments. Summary of the Invention
[0011] The purpose of this invention is to provide an embodied intelligent multimodal perception robustness enhancement method based on feature network topology, which structures multimodal data into a dynamic feature network and introduces an adaptive threshold parameter. Together with community repair mechanisms, we can solve the problem of sensing failure in existing technologies under modal absence conditions.
[0012] The technical solution adopted by the present invention to achieve the above objectives is as follows:
[0013] A robustness enhancement method for embodied intelligent multimodal perception based on feature network topology includes the following steps:
[0014] 1) Collect multi-source perception data from a workstation-level embodied intelligent robot, extract high-dimensional feature vectors for each modality through a heterogeneous encoder, and map them to a unified semantic metric space to generate a set of feature nodes. ;
[0015] 2) Based on the semantic association strength between each pair of feature nodes, an initial fully connected graph is constructed and pruned to create a sparse and robust feature network. ;
[0016] 3) Community detection algorithm for feature network By dividing the data, we obtain a set of characteristic communities. ;
[0017] 4) Perform anomaly detection on nodes in the feature network and capture failed nodes;
[0018] 5) Repair information on failed nodes within the community;
[0019] 6) Input the complete feature network containing the repair features into the downstream task decoder to obtain the robot's perception results or control commands.
[0020] Step 1) further includes the following steps:
[0021] The position and velocity of the end effector and each joint in Cartesian space are calculated using the robot's forward kinematic equations. These are then encoded into kinematic nodes using a heterogeneous encoder. These nodes serve as skeleton nodes in the feature network, which are used to physically constrain the topological deformation of the feature network when visual features fail.
[0022] Step 2) includes the following steps:
[0023] 2.1) If the similarity between two feature nodes in the semantic space or the co-occurrence probability in the physical space is higher than the threshold, then construct one edge between the two nodes to build an initial fully connected graph.
[0024] 2.2) Dynamically calculate the topology sparsity parameters of the initial fully connected graph based on environmental information entropy. ;
[0025] 2.3) Based on parameter a, an adaptive thresholding formula is used to prune the initial fully connected graph, generating a sparse and robust feature network. ,in, It is the set of all modal feature nodes. It is a set of semantically related edges between nodes. It is a set of edge weights.
[0026] Step 2.2) specifically refers to:
[0027] Real-time calculation of the information entropy of the multimodal feature distribution within the current time window. ;
[0028] According to the formula Dynamically adjust parameters ,in, This serves as the baseline value for the topology sparsity parameter. For use in characterizing parameters The entropy-sensitive adjustment coefficient for the intensity of response to environmental uncertainty. This is the target information entropy baseline or the expected entropy threshold.
[0029] Step 2.3) specifically refers to:
[0030] The threshold is calculated using an adaptive threshold formula, namely:
[0031] ;
[0032] in, For connection threshold, This represents the mean similarity between nodes. The standard deviation of the similarity between nodes;
[0033] Iterate through all potential node pairs and retain only those that satisfy the edge weight. The connecting edges.
[0034] Step 3) includes the following steps:
[0035] 3.1) Use the Louvain algorithm to divide the communities;
[0036] 3.2) For the identified communities, the formula is used... Increase the connection weight between nodes within the community, whereby... To enhance the edge weights, The original edge weights between two nodes within the community. This represents the community cohesion enhancement coefficient, and ;
[0037] 3.3) Identify edge nodes that do not belong to any core community and whose degree value is below the threshold, classify them as noise, and remove them directly from the network.
[0038] Step 4) specifically refers to:
[0039] The activation status of nodes in the feature network is monitored in real time. When the activation value of a node is lower than the confidence threshold or a sudden change occurs, it is determined to be a failed node.
[0040] Step 5) specifically involves:
[0041] Locate the target community to which the failed node belongs, retrieve all surviving neighbor nodes within that community, and generate a repair feature vector for the failed node by weighted aggregation using the feature vectors of the neighbor nodes and their corresponding edge weights, and replace the original failure feature vector with this vector.
[0042] The repair feature vector Specifically:
[0043] ;
[0044] in, A failed node. For target club Surviving neighbor nodes within, The edge weights between nodes. To the failed node The set of connected, surviving neighbor nodes. Failed node The club or organization you belong to.
[0045] An embodied intelligent multimodal perception robustness enhancement system based on feature network topology includes:
[0046] The heterogeneous feature node module is used to collect multi-source perception data from workstation-level embodied intelligent robots. It extracts high-dimensional feature vectors of each modality through a heterogeneous encoder and maps them to a unified semantic metric space to generate a set of feature nodes. ;
[0047] The dynamic feature network construction module is used to build an initial fully connected graph based on the semantic association strength between each pair of feature nodes, and then prunes it to create a sparse and robust feature network. ;
[0048] The cross-modal community mining module is used by community detection algorithms to analyze the feature network. By dividing the data, we obtain a set of characteristic communities. ;
[0049] The topology sensing and anomaly detection module is used to detect anomalies in nodes in the feature network and capture failed nodes.
[0050] The information repair module based on community redundancy is used to repair information on failed nodes in the community.
[0051] The robust decision output module is used to input the complete feature network containing repair features into the downstream task decoder to obtain the robot's perception results or control commands.
[0052] The present invention has the following beneficial effects and advantages:
[0053] 1. This invention establishes an embodied intelligent multimodal perception model based on feature network topology, providing a tool for robustness analysis of robot perception in complex unstructured environments. By mapping heterogeneous modal data to unified feature nodes and edges, a shift from data fusion to structured topology analysis is achieved.
[0054] 2. This invention differs from traditional generative repair methods that only address missing data and simple attention feature splicing optimization methods. By analyzing the characteristics of each node in the feature network environment (such as betweenness centrality and community contribution) and constraining the feature propagation path (such as pruning noisy edges and strengthening intra-community connections), it helps improve the operating efficiency and stability of embodied intelligence systems under interference such as line-of-sight occlusion and sensor noise.
[0055] 3. To meet the real-time application requirements of workstation-level embodied intelligent robots, this invention introduces an adaptive threshold adjustment and community cohesion enhancement algorithm based on information entropy to optimize the perception topology, achieving robust enhancement of endogenous characteristics without increasing additional generative computational overhead. Attached Figure Description
[0056] Figure 1 This is the overall process architecture diagram of the method;
[0057] Figure 2 It is an adaptive threshold parameter A diagram illustrating the impact on network topology;
[0058] Figure 3 This is a schematic diagram of a cross-modal community structure;
[0059] Figure 4 This is a comparison diagram of the feature networks before and after information repair. Detailed Implementation
[0060] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0061] This invention proposes an embodied intelligent multimodal perception robustness enhancement method based on characteristic network topology. This method, unlike traditional data repair or generation methods, enhances system robustness by optimizing the connection structure through network modeling of multimodal perception data, based on the analysis of characteristic network topology (such as community structure and core node identification).
[0062] Includes the following steps:
[0063] Step 1: Heterogeneous Feature Node Generation. Collect multi-source perception data from the workstation-level embodied intelligent robot, including but not limited to visual images, industrial time-series signals, production log text, and robot kinematic states; extract high-dimensional feature vectors for each modality using a heterogeneous encoder, and map them to a unified semantic metric space to generate a set of feature nodes. .
[0064] Step 2: Dynamic Feature Network Construction. Calculate the semantic association strength between each pair of feature nodes to construct an initial fully connected graph; dynamically calculate the topology sparsity parameters based on environmental information entropy. Using the adaptive threshold formula Pruning the initial graph generates a sparse and robust feature network. .
[0065] Step 3: Cross-modal community mining. In the feature network The algorithm for detecting communities is applied to divide the network into several characteristic communities that are tightly connected internally and sparsely connected externally. Each community represents a cross-modal physical entity or operational event.
[0066] Step 4: Topology Awareness and Anomaly Detection. The activation status of nodes in the feature network is monitored in real time. When the activation value of a node falls below the confidence threshold or undergoes a sudden change, it is identified as a failed node.
[0067] Step 5: Information Repair Based on Community Redundancy. Locate the target community to which the failed node belongs, retrieve all surviving neighbor nodes within that community, and use the feature vectors of these neighbor nodes and their corresponding edge weights to generate a repair feature vector for the failed node through weighted aggregation. This repair feature vector is then used to replace the original failure feature vector.
[0068] Step 6: Robust Decision Output. The complete feature network containing the repair features is input into the downstream task decoder, which outputs the robot's perception results or control commands.
[0069] In step S1, the feature extraction of the kinematic state combines prior physical knowledge: the position and velocity of the end effector and each joint in Cartesian space are calculated using the robot's forward kinematic equations and encoded as kinematic nodes; the kinematic nodes serve as skeleton nodes in the feature network and are used to limit the topological deformation of the feature network through physical constraints when visual features fail.
[0070] The topology sparsification parameters The settings employ a dynamic entropy weight adjustment mechanism:
[0071] (1) Calculate the information entropy of the multimodal feature distribution within the current time window in real time. ;
[0072] (2) According to the formula Dynamically adjust parameters ;
[0073] (3) Among them, A higher value indicates greater environmental uncertainty. Automatically reduce, thereby retaining more weak connection edges to increase the information redundancy of the network; The lower the value, the clearer the environment. It automatically increases in size to retain the strongest semantic connections and reduce computational load.
[0074] The community detection algorithm uses the Louvain algorithm:
[0075] The community structure constitutes a robust unit of perception, with multimodal features within a single community serving as backups for each other.
[0076] The repair feature vector The calculation formula is:
[0077]
[0078] in, A failed node. For target club Surviving neighbor nodes within, The edge weights between nodes are defined; this step ensures that only semantically related nodes from the same community are used for repair, avoiding the introduction of irrelevant noise.
[0079] A workstation-level embodied intelligent robot system, comprising:
[0080] (1) Multimodal perception module: including depth camera, torque sensor, microphone array and joint encoder;
[0081] (2) Feature network processor: Configured to execute an embodied intelligent multimodal perception robustness enhancement method based on feature network topology, and to construct and maintain the feature network topology;
[0082] (3) Robust control module: Receives the repaired feature network output and controls the robotic arm to perform assembly, welding or handling tasks.
[0083] Example
[0084] like Figure 1 As shown, the specific steps include:
[0085] Step 1: Network model the multimodal perception data of the embodied intelligent robot.
[0086] In this approach, feature nodes are defined as the encoded feature vectors of visual, force, auditory, and proprioceptive kinematic data. If the similarity between two feature nodes in the semantic space or their co-occurrence probability in the physical space exceeds a certain standard, then there is an edge connecting these two nodes. Based on this process, a multimodal perceptual feature network can be constructed.
[0087] For example, in a robot assembly station, the image features captured by industrial cameras, the contact force features captured by force sensors, and the angular features of the robotic arm joints are all mapped to nodes in the network.
[0088] Step 2: Define the characteristics of multimodal sensing feature networks according to the definition in complex networks.
[0089] Perceptual feature networks are abstracted into complex weighted graphs. ,in It is the set of all modal feature nodes. It is a set of semantically related edges between nodes. It is a set of edge weights. Robustness in this invention refers to the network's ability to maintain its core semantic structure without collapsing when faced with node attacks (i.e., sensor failure or noise interference). The effectiveness of features can be measured by the cohesion density of key communities in the network and the betweenness centrality of core nodes.
[0090] Set the set of interfering nodes in the network as (e.g., high-frequency vibration noise characteristics). Robustness enhancement aims to improve performance through topology optimization. The connection with the core network was severed, and the core semantic community was also cut off. The connectivity remains stable.
[0091] Step 3: Based on the settings in Step 2, establish a topology optimization and robustness enhancement method based on structure mining.
[0092] First, calculate the information entropy of the global feature distribution in the network. This indicator is used to dynamically adjust the topology filtering parameters. .
[0093] Adaptive threshold filtering (e.g.) Figure 2 (As shown): Set connection threshold In each step of the graph construction process, we calculate the mean similarity between nodes. and standard deviation .
[0094] When environmental entropy When the parameters are high (e.g., due to complex operating conditions or numerous interferences), the algorithm automatically reduces the parameters. (like Figure 2 (On the right side), more connections are retained, utilizing redundancy to resist interference.
[0095] When environmental entropy When the parameters are low (e.g., the operating conditions are clear), the algorithm automatically increases them. (like Figure 2 (Left side) Only the strongest backbone connections are retained, and potential weak noise is removed through sparsification to improve computational efficiency.
[0096] During this process, the algorithm will traverse all potential node pairs and retain only those that satisfy the condition. The edges are connected to form a dynamically changing sparse network structure.
[0097] After steps three and four, some strongly interfering nodes may still exist in the network, and the core semantic community may not be tightly knit enough. Therefore, this step will perform community structure analysis on the feature network and implement a "community cohesion strengthening" strategy.
[0098] Community segmentation: Using the Louvain algorithm to identify strongly related communities in the network (such as...) Figure 3 (As shown). For example, in insertion and removal operations, the visual feature of "socket alignment" and the tactile feature of "disappearance of resistance" are grouped into the same category.
[0099] Cohesion reinforcement: For identified communities, the algorithm artificially increases the connection weights between nodes within the community, using the following formula: The purpose of this operation is to group heterogeneous features describing the same physical event together, making it difficult for them to be dispersed by external noise as a whole.
[0100] Structured pruning: Simultaneously, it identifies edge nodes that do not belong to any core community and have extremely low degree values (such as...). Figure 4 The isolated point on the left is identified as noise and removed directly from the network.
[0101] This step can significantly purify the feature network, ensuring that the information input into the downstream decision model is a structured and validated high-confidence feature.
[0102] Step 5: After obtaining the optimized feature network, this step outputs the network to the robot's decision-making module.
[0103] After the network undergoes the aforementioned topology reconstruction, its node characteristics not only represent the original sensor readings but also contain global topological context information.
[0104] For example, even if the visual sensor is briefly occluded (the visual node features become weaker), since the visual node and the strong force node are bound to the same reinforcement community in step four, the activation of the force node can support the semantic representation of the visual node through the strong connections within the community, preventing the decision layer from misjudging it as having no target.
[0105] It is worth noting that this invention abandons complex generative repair, that is, it does not fabricate data out of thin air, but relies entirely on existing, reliable topological structures to maintain the continuity of perception.
[0106] Through this step, we will obtain the final robust perception results, which will guide the robot to perform precise operations.
[0107] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0108] This invention is described with reference to a process flowchart according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A device for a function specified in one or more processes.
[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.
[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.
[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A robustness enhancement method for embodied intelligent multimodal perception based on feature network topology, characterized in that, Includes the following steps: 1) Collect multi-source perception data from a workstation-level embodied intelligent robot, extract high-dimensional feature vectors for each modality through a heterogeneous encoder, and map them to a unified semantic metric space to generate a set of feature nodes. ; 2) Based on the semantic association strength between each pair of feature nodes, an initial fully connected graph is constructed and pruned to create a sparse and robust feature network. ; 3) Community detection algorithm for feature network By dividing the data, we obtain a set of characteristic communities. ; 4) Perform anomaly detection on nodes in the feature network and capture failed nodes; 5) Repair information on failed nodes within the community; 6) Input the complete feature network containing the repair features into the downstream task decoder to obtain the robot's perception results or control commands.
2. The embodied intelligent multimodal perception robustness enhancement method based on feature network topology according to claim 1, characterized in that, Step 1) further includes the following steps: The position and velocity of the end effector and each joint in Cartesian space are calculated using the robot's forward kinematic equations. These are then encoded into kinematic nodes using a heterogeneous encoder. These nodes serve as skeleton nodes in the feature network, which are used to physically constrain the topological deformation of the feature network when visual features fail.
3. The embodied intelligent multimodal perception robustness enhancement method based on feature network topology according to claim 1, characterized in that, Step 2) includes the following steps: 2.1) If the similarity between two feature nodes in the semantic space or the co-occurrence probability in the physical space is higher than the threshold, then construct one edge between the two nodes to build an initial fully connected graph. 2.2) Dynamically calculate the topology sparsity parameters of the initial fully connected graph based on environmental information entropy. ; 2.3) Based on parameter a, an adaptive thresholding formula is used to prune the initial fully connected graph, generating a sparse and robust feature network. ,in, It is the set of all modal feature nodes. It is a set of semantically related edges between nodes. It is a set of edge weights.
4. The embodied intelligent multimodal perception robustness enhancement method based on feature network topology according to claim 3, characterized in that, Step 2.2) specifically refers to: Real-time calculation of the information entropy of the multimodal feature distribution within the current time window. ; According to the formula Dynamically adjust parameters ,in, This serves as the baseline value for the topology sparsity parameter. For use in characterizing parameters The entropy-sensitive adjustment coefficient for the intensity of response to environmental uncertainty. This is the target information entropy baseline or the expected entropy threshold.
5. The embodied intelligent multimodal perception robustness enhancement method based on feature network topology according to claim 3, characterized in that, Step 2.3) specifically refers to: The threshold is calculated using an adaptive threshold formula, namely: ； in, For connection threshold, This represents the mean similarity between nodes. The standard deviation of the similarity between nodes; Iterate through all potential node pairs and retain only those that satisfy the edge weight. The connecting edges.
6. The embodied intelligent multimodal perception robustness enhancement method based on feature network topology according to claim 1, characterized in that, Step 3) includes the following steps: 3.1) Use the Louvain algorithm to divide the communities; 3.2) For the identified communities, the formula is used... Increase the connection weight between nodes within the community, whereby... To enhance the edge weights, The original edge weights between two nodes within the community. This represents the community cohesion enhancement coefficient, and ; 3.3) Identify edge nodes that do not belong to any core community and whose degree value is below the threshold, classify them as noise, and remove them directly from the network.
7. The embodied intelligent multimodal perception robustness enhancement method based on feature network topology according to claim 1, characterized in that, Step 4) specifically refers to: The activation status of nodes in the feature network is monitored in real time. When the activation value of a node is lower than the confidence threshold or a sudden change occurs, it is determined to be a failed node.
8. The embodied intelligent multimodal perception robustness enhancement method based on feature network topology according to claim 1, characterized in that, Step 5) specifically involves: Locate the target community to which the failed node belongs, retrieve all surviving neighbor nodes within that community, and generate a repair feature vector for the failed node by weighted aggregation using the feature vectors of the neighbor nodes and their corresponding edge weights, and replace the original failure feature vector with this vector.
9. The embodied intelligent multimodal perception robustness enhancement method based on feature network topology according to claim 8, characterized in that, The repair feature vector Specifically: ； in, A failed node. For target club Surviving neighbor nodes within, The edge weights between nodes. To the failed node The set of connected, surviving neighbor nodes. Failed node The club or organization you belong to.
10. An embodied intelligent multimodal perception robustness enhancement system based on feature network topology, characterized in that, include: The heterogeneous feature node module is used to collect multi-source perception data from workstation-level embodied intelligent robots. It extracts high-dimensional feature vectors of each modality through a heterogeneous encoder and maps them to a unified semantic metric space to generate a set of feature nodes. ; The dynamic feature network construction module is used to build an initial fully connected graph based on the semantic association strength between each pair of feature nodes, and then prunes it to create a sparse and robust feature network. ; The cross-modal community mining module is used by community detection algorithms to analyze the feature network. By dividing the data, we obtain a set of characteristic communities. ; The topology sensing and anomaly detection module is used to detect anomalies in nodes in the feature network and capture failed nodes. The information repair module based on community redundancy is used to repair information on failed nodes in the community. The robust decision output module is used to input the complete feature network containing repair features into the downstream task decoder to obtain the robot's perception results or control commands.