Multi-dimensional situation awareness based embodied intelligent robot task planning system and method

Abstract

The application discloses a multi-dimensional situation awareness-based embodied intelligent robot task planning system and method, and particularly relates to the technical field of embodied intelligent robot task planning, and performs space-time alignment on asynchronous and heterogeneous data generated by a plurality of modal sensor channels, extracts cross-modal space-time features through a cross-modal feature fusion network, generates a fusion situation matrix, updates the correlation strength between the plurality of modal data using dynamic causal modeling, identifies and traces abnormalities through a counterfactual reasoning engine, and outputs abnormal trace results; the fusion situation matrix and the abnormal trace results are used to dynamically adjust the reliability weight of each sensor channel through a multi-modal credibility evaluation model; based on the reliability weight, the fusion situation matrix is input into a real robot dynamics model and a digital twin virtual model, cooperative predictive control is performed, and when the safety score is lower than a threshold value, an adaptive rule evolution mechanism is started to solve the fusion matrix distortion problem in dynamic obstacle avoidance and precision grasping tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of embodied intelligent robot task planning, more particularly, the present application relates to a multi-dimensional situation awareness based embodied intelligent robot task planning system and method. BACKGROUND

[0002] Embodied intelligent robots performing tasks in dynamic unstructured environments, such as industrial sorting manipulators, disaster rescue robots, etc., need to rely on visual, tactile and laser radar and other multi-modal sensors to perceive the environmental situation in real time, and generate safe and continuous action instructions; However, the prior art has significant defects, mainly manifested in the lack of cooperative processing mechanism of heterogeneous multi-modal signals, resulting in inaccurate perception and decision-making in dynamic physical interaction scenarios; Specifically, traditional methods use linear interpolation or fixed delay compensation to realize multi-modal data alignment, but cannot eliminate the spatio-temporal misalignment of high-frequency tactile signals and low-frequency visual data, for example, when tactile perception contact remains, vision has determined that the obstacle has disappeared, resulting in millisecond-level spatio-temporal deviation in the fusion situation matrix in dynamic obstacle avoidance, precision grasping and other tasks.

[0003] In addition, the static confidence evaluation rule does not fuse the correlation between physical field parameters and multi-modal semantics, and cannot quantify the reliability difference of sensors in dynamic interaction, resulting in the generation of oscillation instructions or high-risk actions by the task planner when there is signal conflict, such as frequent start-stop or grasp force exceeding the limit in a narrow channel, such problems are particularly prominent in scenarios with frequent environmental disturbances and significant sensor heterogeneity, seriously restricting the autonomy, efficiency and safety of robots, and constitute a technical bottleneck that needs to be broken through. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the present application provides a multi-dimensional situation awareness based embodied intelligent robot task planning system and method, which solves the problems raised in the above background art by spatio-temporal misalignment of multi-modal data, improving the accuracy of sensor reliability evaluation in dynamic scenarios, and ensuring the safety of task planning in complex environments.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a multi-dimensional situation awareness based embodied intelligent robot task planning method, comprising the following steps:

[0006] Step one, use a pulse neural network to perform spatio-temporal alignment on asynchronous and heterogeneous data generated by multi-modal sensor channels, and extract cross-modal spatio-temporal features through a cross-modal feature fusion network to generate a fusion situation matrix as the input of step two and step three;

[0007] Step two, based on the fusion situation matrix generated in step one, use dynamic causal modeling to update the correlation strength between multi-modal data, and use a counterfactual reasoning engine to identify and trace the abnormality, and output the abnormality tracing result to step three;

[0008] Step 3: Using the fused situational matrix generated in Step 1 and the anomaly tracing results output in Step 2, process the dynamic field strength distribution map and real-time physical field parameters through a graph neural network, dynamically adjust the reliability weights of each sensing channel through a multimodal reliability assessment model, and output to Step 4.

[0009] Step 4: Based on reliability weights, the fused situation matrix is ​​input into the real robot dynamics model and the digital twin virtual model to execute collaborative predictive control. When the safety score is lower than the threshold, an adaptive rule evolution mechanism is activated to ensure the safety of the decision.

[0010] Preferably, the asynchronous heterogeneous data includes at least asynchronous data streams from tactile sensing channels, optical sensing channels, and three-dimensional ranging sensing channels. The spiking neural network uses a frequency-domain adaptive interpolation algorithm to achieve data synchronization and dynamically adjusts the interpolation accuracy according to the target's motion speed. The cross-modal feature fusion network includes a spatiotemporal attention mechanism and a recursive feature extraction unit. It establishes the feature correlation between different sensing channels through a cross-attention mechanism and generates a fusion situation matrix containing multimodal semantic features by combining temporal dependency modeling. For example, during the operation of the frequency-domain adaptive interpolation algorithm, the interpolation accuracy is dynamically adjusted to match the changes in the object's motion speed in the scene. The self-attention mechanism is used to quantify the correlation between tactile sensor data and visual sensor data, and the spatial topological features of the lidar sensor data are fused under the cross-attention mechanism to generate a fusion situation matrix, ensuring high-precision cross-modal feature extraction in dynamic scene recognition.

[0011] The explanation is as follows: the multimodal sensor channel adopts a distributed array layout, including a piezoresistive tactile sensor, a depth camera, and a lidar sensor; each sensor achieves time synchronization through a hardware trigger signal, and the trigger signal deviation is less than a preset threshold; the data interface adopts the robot operating system protocol, and the message format follows the point cloud data standard;

[0012] Preferably, the asynchronous heterogeneous data includes low-frequency sensor data and high-frequency sensor data. The frequency domain adaptive interpolation algorithm includes the following steps: performing frequency domain decomposition on the low-frequency sensor data to extract the dominant frequency component; encoding the high-frequency sensor data into a pulse sequence, with the pulse firing threshold dynamically set according to the sensor type, and the firing frequency nonlinearly correlated with the target motion speed; adjusting the interpolation window length based on the motion speed, and using spline interpolation to fill in the missing frames of the low-frequency data; verifying the timestamp alignment error of the multi-channel data, and triggering synchronous calibration when the error exceeds the tolerance.

[0013] Preferably, the dynamic causal modeling adopts a Bayesian weight update algorithm to reconstruct the causal relationship between multimodal sensor data in real time. When the causal correlation between sensor data deviates from the preset dynamic error threshold, anomaly detection is triggered. For example, when the causal edge weight deviation between tactile sensor data and lidar sensor data exceeds the preset dynamic threshold, anomaly identification is triggered. The counterfactual reasoning engine simulates the data distribution under abnormal conditions through generative adversarial networks and outputs anomaly tracing results containing anomaly type labels and propagation paths by combining historical anomaly pattern database.

[0014] Preferably, the graph neural network generates a dynamic field strength distribution map based on synchronous localization and mapping technology. The multimodal reliability assessment model integrates an environmental physical field analysis unit and a sensing error compensation unit. Specifically, the environmental physical field analysis unit constructs the dynamic field strength distribution map through electromagnetic field gradient monitoring and acoustic disturbance detection; for example, it integrates real-time monitored electromagnetic field strength gradient data and acoustic Doppler effect characteristics to construct the dynamic field strength distribution map. The sensing error compensation unit adopts a multi-physics coupling correction algorithm to generate a sensing data correction matrix based on real-time physical field parameters and calculates the reliability weight of each sensing channel by combining semantic segmentation confidence index.

[0015] Preferably, the safety score is obtained as follows: The digital twin virtual model is constructed through a physics engine, including state synchronization, environment mapping, and latency compensation mechanisms. It receives real-time joint state data of the real embodied robot and generates a 3D occupancy map of the virtual environment. A trajectory similarity measurement algorithm is used to calculate the spatial deviation between the real trajectory point sequence and the virtual trajectory point sequence, and a trajectory deviation degree is generated through a normalized mapping function. An obstacle occupancy probability model is constructed based on the LiDAR data after multi-physics coupling correction. The model is dynamically corrected by fusing the confidence level of the visual sensor and the pressure gradient features of the tactile sensor. The collision risk probability is calculated by superimposing a 3D risk field. The safety score is generated by weighting the trajectory deviation degree and the collision risk probability.

[0016] Preferably, the process of calculating the reliability weight includes:

[0017] Let i represent the sensor channel number index, and N represent the total number of sensor channels;

[0018] Spatiotemporal consistency quantification: Based on multimodal consistency testing, calculate the spatiotemporal consistency score of the i-th sensing channel. ,in Let be the real-time measurement vector of the i-th sensing channel. The cross-modal prediction value is the prediction value generated by the Transformer-LSTM hybrid network through data from other sensor channels; cos(·) is the vector cosine similarity calculation function, used to quantify the directional consistency between real-time data and prediction values;

[0019] Environmental interference intensity quantification: The environmental interference intensity of the i-th sensing channel is calculated using the following formula:

[0020] ;

[0021] in, The electromagnetic field gradient magnitude was obtained through differential calculation using a triaxial magnetic field sensor. fL is the sound pressure power spectral density, extracted from the acoustic sensor array via fast Fourier transform; fH is the lower limit frequency of integration, representing the lowest effective frequency threshold for calculating the sound pressure power spectral density integral; fH is the upper limit frequency of integration, representing the highest effective frequency threshold for calculating the sound pressure power spectral density integral. , The sensitivity coefficient is related to the sensor type and is a scaling factor obtained by fitting the interference-error curve in the sensor calibration experiment. For example, in actual industrial scenarios, fL can be set to 50Hz (to avoid power frequency interference) and fH can be set to 15kHz (higher than the upper limit of human hearing).

[0022] Task Relationship Measurement: Based on the security requirement level of the current task phase. (Dynamically assigned values ​​based on the robot's current operation type, discrete values: {1: non-safety critical, 2: collaborative operation, 3: high-risk operation}) and operation accuracy requirements. The value ranges from 0.1 to 1.0, used to calculate the task criticality factor. ; Indicates task priority;

[0023] Dynamic weight composition: will be denoted as The reliability weight of the i-th sensing channel is calculated using a weighted fusion function. ,in A small constant to prevent division by zero errors;

[0024] ;

[0025] in, It is a small positive number, and in this embodiment of the invention, its value range is 0.01-0.0001, which is used to prevent calculation overflow caused by an excessively small denominator.

[0026] Preferably, the collaborative predictive control calculates the output deviation between the physical model and the digital twin virtual model through a trajectory similarity measurement algorithm. When the output deviation exceeds a preset trajectory fault tolerance threshold, safety constraint optimization is activated. The adaptive rule evolution mechanism includes rule credibility assessment and logical structure optimization. The effectiveness of the rules is verified in the simulation environment through a reinforcement learning mechanism, and the constraints and optimization objectives in the decision rule base are dynamically updated. The adaptive rule evolution mechanism includes the following steps: constructing an anomaly pattern knowledge graph: based on historical anomaly events from tactile, visual, and lidar sensor data, spatiotemporal correlation features are extracted and a topological structure of anomaly propagation paths is generated; incremental optimization of the rule base: the similarity between real-time anomaly scenarios and historical knowledge graphs is matched through a contrastive learning algorithm, and the constraints in the decision rule base are dynamically updated, prioritizing the retention of rules whose effectiveness has been verified; unsupervised verification mechanism: extreme scenarios are simulated in the digital twin virtual model, and the stability of newly added rules is verified, eliminating redundant rules that cause a decrease in safety scores; for example, when a similarity of more than 85% with historical tactile anomaly patterns is detected, the corresponding decision rule is automatically loaded; and anomaly handling rules from industrial assembly scenarios are adapted to warehousing and handling scenarios through transfer learning.

[0027] Preferably, the multi-physics coupling correction algorithm includes an interference source localization unit and a dynamic weight redistribution unit. When the electromagnetic field intensity gradient exceeds the device's anti-interference threshold or the acoustic disturbance frequency enters the sensor's sensitive frequency band, the interference source localization unit is activated to perform error compensation. The interference source localization unit includes: establishing a spatial distribution probability model of the interference source based on three-dimensional vector data of the electromagnetic field gradient and combining it with the Lorentz force equation to calculate the maximum likelihood electromagnetic interference source coordinates; using a generalized cross-correlation algorithm to process the time delay estimation of the acoustic sensor array, and constructing the acoustic interference source location information interval through the sound wave arrival time difference; and a dynamic weight redistribution unit, which triggers a weight optimization algorithm based on Lyapunov stability theory to redistribute the reliability weights of the multimodal sensor data when the Frobenius norm of the disturbance coefficient matrix composed of environmental physical field parameters exceeds an adaptive threshold.

[0028] Preferably, the dynamic evolution of symbol rules is based on historical verification data and simulated scenarios. Through self-supervised learning, it analyzes historical abnormal patterns in tactile sensor data, visual sensor data, and lidar sensor data, automatically optimizes the decision rule base, and improves the robot's task planning ability in unpredictable scenarios.

[0029] Preferably, the multi-physics coupling compensation mechanism monitors the temperature change rate and electromagnetic interference intensity in real time. When the temperature change rate or electromagnetic interference intensity exceeds a preset threshold, an online calibration algorithm is triggered to correct the tactile sensor data, visual sensor data, and lidar sensor data, thereby suppressing the impact of environmental noise on the reliability weight allocation and ensuring the accuracy of weight adjustment.

[0030] To achieve the above objectives, the present invention provides the following technical solution: a task planning system for embodied intelligent robots based on multi-dimensional situational awareness, comprising:

[0031] The spatiotemporal alignment module uses a pulse neural network to perform frequency domain adaptive interpolation on asynchronous heterogeneous data from multimodal sensor channels (sensor devices include at least tactile, visual, and lidar sensors) to eliminate spatiotemporal misalignment of high-frequency / low-frequency signals, output synchronized multimodal data, and output it to the feature fusion module.

[0032] The feature fusion module utilizes a cross-modal feature fusion network (including a spatiotemporal attention mechanism and recursive units) to extract spatiotemporal correlation features from multimodal data, generate a fusion situation matrix containing semantic information (including multimodal features such as tactile pressure gradient, visual target recognition, and LiDAR spatial topology), and transmit it to the causal modeling module and the reliability assessment module.

[0033] The causal modeling module, based on dynamic causal modeling (Bayesian weight update algorithm), analyzes the strength of causal associations between multimodal data in real time, detects sensor data conflicts (such as the contradiction between tactile residue and the disappearance of visual impairment), outputs an updated sensor causal association map (including abnormal trigger markers), and transmits it to the abnormal source tracing module.

[0034] The anomaly tracing module simulates the distribution of anomalous data through a counterfactual reasoning engine (built based on adversarial networks and a historical anomaly database), identifies anomaly types (such as electromagnetic interference and sensor failure), generates interpretable anomaly tracing results (including propagation paths and type labels), outputs anomaly type labels and propagation path reports, and transmits them to the trust assessment module.

[0035] The reliability assessment module, combining the dynamic field strength distribution map (electromagnetic gradient, acoustic disturbance) and anomaly tracing results, calculates the spatiotemporal consistency score and environmental interference intensity of each sensor channel through a multimodal reliability assessment model, dynamically adjusts the reliability weights, outputs the reliability weight matrix of the sensor channel (e.g., tactile weight decreases, lidar weight increases), and transmits it to the collaborative control module.

[0036] The collaborative control module, based on reliability weights and a fusion situation matrix, performs collaborative predictive control (trajectory similarity measurement + collision risk calculation) by combining the real robot dynamics model and the digital twin virtual model, generating a safety score and outputting action commands; it outputs robot joint control commands and safety scores, and triggers the rule evolution module when the safety score is lower than a threshold.

[0037] The rule evolution module uses reinforcement learning and anomaly pattern knowledge graphs to verify and optimize the decision rule base in the digital twin environment (such as adding obstacle avoidance constraints and adjusting grip force thresholds), dynamically updates rules to adapt to extreme scenarios, and outputs an updated safety decision rule base (such as emergency shutdown rules and dynamic obstacle avoidance strategies). The optimized rule base is fed back to the collaborative control module to form a closed-loop iteration.

[0038] The technical effects and advantages of this invention are as follows:

[0039] (1) The embodied intelligent robot task planning method based on multidimensional situational awareness proposed in this invention realizes the spatiotemporal alignment of multimodal data through spiking neural network and frequency domain adaptive interpolation algorithm. Combined with the spatiotemporal attention mechanism of cross-modal feature fusion network, it dynamically matches the asynchronous data streams of tactile, visual and lidar, and eliminates the millisecond-level spatiotemporal deviation between high-frequency tactile signals and low-frequency visual data. Through the spline interpolation window adjustment of motion speed adaptive, it maintains the synchronization accuracy of multimodal features in the scene of object speed change, solves the problem of fusion matrix distortion caused by spatiotemporal misalignment in dynamic obstacle avoidance and precision grasping tasks, and enhances the situational awareness capability in complex scenes.

[0040] (2) The embodied intelligent robot task planning method based on multidimensional situational awareness proposed in this invention reconstructs the correlation strength of sensor data in real time and dynamically adjusts the weights by integrating physical field parameters through dynamic causal modeling and multimodal credibility evaluation model. The counterfactual reasoning engine based on Bayesian network can trace the abnormal propagation path. The dynamic field strength distribution map is constructed by combining electromagnetic field gradient and acoustic disturbance parameters. In the sensor conflict scenario, the weight allocation is optimized by Lyapunov stability theory, which effectively solves the problem of grasping force exceeding the limit and trajectory planning inaccuracy caused by environmental disturbance in traditional methods. Attached Figure Description

[0041] Figure 1 This is a flowchart of the intelligent robot task planning method of the present invention.

[0042] Figure 2 This is a block diagram of the robot task planning system of the present invention. Detailed Implementation

[0043] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0044] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0045] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0046] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0047] In existing technologies, embodied intelligent robots operating in dynamic unstructured environments face the challenge of collaborative processing of multimodal sensor data. Traditional methods rely on fixed-frequency interpolation to achieve data synchronization, which makes it difficult to eliminate the spatiotemporal misalignment between high-frequency tactile signals and low-frequency visual data. When the robotic arm contacts an object, the tactile sensor has already detected the pressure change, but the vision system has not yet updated the obstacle position information, resulting in millisecond-level deviations in the fused situation matrix. Static confidence assessment rules do not consider physical field disturbances such as electromagnetic interference or temperature changes, and are prone to generating oscillation commands or high-risk actions when sensor signals conflict, such as the robotic arm repeatedly starting and stopping in a narrow channel or applying gripping force beyond the target's tolerance.

[0048] To address the aforementioned issues and the spatiotemporal misalignment of multimodal data, the inventors recognized that traditional linear interpolation cannot adapt to scenarios with varying motion speeds, necessitating the establishment of an interpolation mechanism with a nonlinear correlation to the target's motion speed. By analyzing the relationship between the pulse emission characteristics of tactile sensors and the object's motion speed, an adaptive interpolation method based on frequency domain decomposition was proposed. Regarding sensor reliability assessment, it was found that the fixed threshold weighting rule did not integrate environmental physical field parameters. Therefore, a dynamic correlation model between multimodal data and electromagnetic field gradients needs to be constructed. Combining this with the virtual-real linkage mechanism, a collaborative predictive control architecture between a digital twin virtual model and a real robot was designed to achieve dynamic verification of safety decisions.

[0049] Example 1, see Figure 1 The present invention provides a flowchart of an intelligent robot task planning method. Figure 1 The illustrated task planning method for embodied intelligent robots based on multidimensional situational awareness includes:

[0050] Step 1: Use a spiking neural network to perform spatiotemporal alignment on the asynchronous heterogeneous data generated by the multimodal sensor channels, and extract cross-modal spatiotemporal features through a cross-modal feature fusion network to generate a fusion situation matrix, which serves as the input for Steps 2 and 3.

[0051] Step 2: Based on the fusion situation matrix generated in Step 1, use dynamic causal modeling to update the correlation strength between multimodal data, and use the counterfactual reasoning engine to identify and trace anomalies, and output the anomaly tracing results to Step 3;

[0052] Step 3: Using the fused situational matrix generated in Step 1 and the anomaly tracing results output in Step 2, process the dynamic field strength distribution map and real-time physical field parameters through a graph neural network, dynamically adjust the reliability weights of each sensing channel through a multimodal reliability assessment model, and output to Step 4.

[0053] Step 4: Based on the reliability weights output in Step 3, input the fusion situation matrix generated in Step 1 into the real robot dynamics model and the digital twin virtual model to execute collaborative predictive control. When the safety score is lower than the threshold, activate the adaptive rule evolution mechanism to ensure the safety of the decision.

[0054] In this embodiment of the invention, it is necessary to further explain that the asynchronous heterogeneous data includes at least asynchronous data streams from tactile sensing channels, optical sensing channels, and three-dimensional ranging sensing channels. The spiking neural network uses a frequency-domain adaptive interpolation algorithm to achieve data synchronization and dynamically adjusts the interpolation accuracy according to the target's motion speed. The cross-modal feature fusion network refers to a deep learning architecture that integrates the spatiotemporal features of different modal sensors. It includes a spatiotemporal attention mechanism and a recursive feature extraction unit. The feature correlation between different sensing channels is established through a cross-attention mechanism. Combined with temporal dependency modeling, a fusion situation matrix containing multimodal semantic features is generated. For example, during the operation of the frequency-domain adaptive interpolation algorithm, the interpolation accuracy is dynamically adjusted to match the changes in the object's motion speed in the scene. The correlation between tactile sensor data and visual sensor data is quantified using a self-attention mechanism, and the spatial topological features of the lidar sensor data are fused under the cross-attention mechanism to generate a fusion situation matrix, ensuring high-precision cross-modal feature extraction in dynamic scene recognition.

[0055] To explain, a spiking neural network is a neural network that uses the pulse transmission mechanism of biological neurons to process asynchronous event data. Specifically, it can be implemented using a third-generation neural network model with time coding characteristics. Its pulse firing frequency is non-linearly related to the input signal strength, making it suitable for processing high-frequency pulse signals from tactile sensors.

[0056] The explanation is as follows: the multimodal sensor channel adopts a distributed array layout, including a piezoresistive tactile sensor, a depth camera, and a lidar sensor; each sensor achieves time synchronization through a hardware trigger signal, and the trigger signal deviation is less than a preset threshold; the data interface adopts the robot operating system protocol, and the message format follows the point cloud data standard;

[0057] In this embodiment of the invention, it is necessary to further explain that the asynchronous heterogeneous data includes low-frequency sensor data and high-frequency sensor data. The frequency domain adaptive interpolation algorithm includes the following steps: performing frequency domain decomposition on the low-frequency sensor data to extract the main frequency component; encoding the high-frequency sensor data into a pulse sequence, with the pulse firing threshold dynamically set according to the sensor type, and the firing frequency nonlinearly correlated with the target's motion speed; adjusting the interpolation window length based on the motion speed, and using spline interpolation to fill in the missing frames of the low-frequency data; verifying the timestamp alignment error of the multi-channel data, and triggering synchronous calibration when the error exceeds the tolerance.

[0058] In this embodiment of the invention, it is necessary to further explain that the dynamic causal modeling adopts a Bayesian weight update algorithm to reconstruct the causal relationship between multimodal sensor data in real time. When the causal correlation between sensor data deviates from the preset dynamic error threshold, anomaly detection is triggered. For example, when the causal edge weight deviation between tactile sensor data and lidar sensor data exceeds the preset dynamic threshold, anomaly identification is triggered. The counterfactual reasoning engine simulates the data distribution under abnormal conditions by generating adversarial networks and combines it with the historical anomaly pattern database to output anomaly tracing results containing anomaly type labels and propagation paths.

[0059] In this embodiment of the invention, it is necessary to further explain that the graph neural network generates a dynamic field strength distribution map based on synchronous localization and mapping technology. The multimodal reliability assessment model integrates an environmental physical field analysis unit and a sensing error compensation unit. Specifically, the environmental physical field analysis unit constructs the dynamic field strength distribution map through electromagnetic field gradient monitoring and acoustic disturbance detection; for example, it integrates real-time monitored electromagnetic field strength gradient data and acoustic Doppler effect characteristics to construct the dynamic field strength distribution map. The sensing error compensation unit adopts a multi-physics coupling correction algorithm, generates a sensing data correction matrix based on real-time physical field parameters, and calculates the reliability weight of each sensing channel by combining semantic segmentation confidence index.

[0060] In this embodiment of the invention, it needs to be further explained that the security score is obtained in the following way:

[0061] The digital twin virtual model is built through a physics engine and includes state synchronization, environment mapping and latency compensation mechanisms. It receives joint state data of the real robot in real time and generates a three-dimensional map of the virtual environment.

[0062] A trajectory similarity measurement algorithm is used to calculate the spatial deviation between the real trajectory point sequence and the virtual trajectory point sequence, and the trajectory deviation degree is generated by a normalized mapping function.

[0063] An obstacle occupancy probability model is constructed based on LiDAR data after multi-physics coupling correction. The model is dynamically corrected by fusing the confidence level of the visual sensor and the pressure gradient characteristics of the tactile sensor. The collision risk probability is calculated by superimposing a three-dimensional risk field.

[0064] A safety score is generated by weighting the trajectory deviation and the probability of collision risk.

[0065] In this embodiment of the invention, it needs to be further explained that the collaborative predictive control calculates the output deviation between the physical model and the digital twin virtual model through a trajectory similarity measurement algorithm. When the output deviation exceeds a preset trajectory fault tolerance threshold, safety constraint optimization is activated. The adaptive rule evolution mechanism includes rule credibility evaluation and logical structure optimization. The effectiveness of the rules is verified in a simulation environment through a reinforcement learning mechanism, and the constraints and optimization objectives in the decision rule base are dynamically updated. The adaptive rule evolution mechanism includes the following steps:

[0066] Constructing anomaly pattern knowledge graph: Based on historical anomaly events from tactile, visual, and lidar sensor data, extract spatiotemporal correlation features and generate the topology of anomaly propagation paths;

[0067] Incremental optimization of the rule base: By comparing the similarity between real-time abnormal scenarios and historical knowledge graphs through a comparative learning algorithm, the constraints in the decision rule base are dynamically updated, and rules with proven effectiveness are retained first.

[0068] Unsupervised verification mechanism: Extreme scenarios are simulated in the digital twin virtual model to verify the stability of newly added rules and eliminate redundant rules that cause a decrease in safety score; for example, when the similarity with historical tactile anomaly patterns exceeds 85%, the corresponding decision rule is automatically loaded; through transfer learning, the anomaly handling rules of industrial assembly scenarios are adapted to warehousing and handling scenarios.

[0069] In this embodiment of the invention, it is necessary to further explain that the multi-physics coupling correction algorithm includes an interference source localization unit and a dynamic weight redistribution unit. When the electromagnetic field intensity gradient is detected to exceed the device's anti-interference threshold or the acoustic disturbance frequency enters the sensor's sensitive frequency band, the interference source localization unit is activated to perform error compensation. The interference source localization unit includes: establishing a spatial distribution probability model of the interference source based on three-dimensional vector data of the electromagnetic field gradient and combining it with the Lorentz force equation to calculate the maximum likelihood electromagnetic interference source coordinates; using a generalized cross-correlation algorithm to process the time delay estimation of the acoustic sensor array, and constructing the acoustic interference source location information interval through the sound wave arrival time difference; and a dynamic weight redistribution unit, which triggers a weight optimization algorithm based on Lyapunov stability theory to redistribute the reliability weights of the multimodal sensor data when the Frobenius norm of the disturbance coefficient matrix composed of environmental physical field parameters exceeds an adaptive threshold. The explanation is that the dynamic adjustment of sensor weights is based on multimodal semantic segmentation results and real-time physical field parameters. The multimodal semantic segmentation results are generated by fusing visual sensor data and lidar sensor data. The real-time physical field parameters include electromagnetic field gradient, sound pressure fluctuation rate, and thermal convection intensity. When the Frobenius norm of the perturbation coefficient matrix formed by the real-time physical field parameters exceeds the adaptive threshold of 0.5, a weight redistribution algorithm based on Lyapunov stability theory is triggered to optimize the weight allocation of multimodal sensor data to adapt to complex scenarios.

[0070] In this embodiment of the invention, it is necessary to further explain that the dynamic evolution of the symbol rules is based on historical verification data and simulated scenarios. Through self-supervised learning, the historical abnormal patterns of tactile sensor data, visual sensor data, and lidar sensor data are analyzed to automatically optimize the decision rule base and improve the robot's task planning ability in unpredictable scenarios.

[0071] Among them, the dynamic evolution of symbolic rules refers to the automatic adjustment of the logical structure and constraints of the decision rule base according to environmental changes. Specifically, this can be achieved using a graph embedding algorithm based on historical anomaly patterns. By encoding the spatiotemporal correlation features of anomaly events, a rule evolution path is generated to address the problem that static rule bases cannot adapt to new anomalies. Historical validation data refers to a set of sensor data validated in real-world scenarios. Specifically, a multimodal time-series database can be used to store historical anomaly event records from tactile, visual, and LiDAR sensor data, providing reliable data support for rule optimization. Simulated scenarios refer to virtual environments constructed using digital twin virtual models to verify the effectiveness of newly added rules under extreme conditions. Self-supervised learning refers to learning that does not require manual annotation. The feature extraction method can specifically employ a contrastive learning algorithm to perform unsupervised feature alignment on multimodal sensor data, and achieve cross-modal anomaly association mining by maximizing the similarity of positive sample pairs to identify composite anomaly patterns. Historical anomaly patterns refer to recurring combinations of anomaly features in sensor data. Specifically, a spatiotemporal graph convolutional network can be used to extract joint anomaly features from tactile pressure gradient changes and visual semantic segmentation results to capture implicit associations between sensor data. The automatic optimization decision rule base refers to dynamically updating the logical constraints in the rule base. Specifically, an incremental rule mining algorithm can be used to match the similarity between real-time anomaly scenarios and historical knowledge graphs, and reinforcement learning can be used to filter out effective rules and remove redundant rules to improve the adaptability of the rule base.

[0072] Specifically, a self-supervised learning algorithm is used to extract spatiotemporal correlation anomalies between tactile pressure gradient features and visual semantic segmentation results. For example, when the tactile sensor detects a sudden change in contact surface pressure but the visual system does not recognize an obstacle, the system automatically marks it as a potential abnormal event. During the rule base optimization process, a contrastive learning algorithm is used to match the real-time abnormal features with nodes in the historical knowledge graph. When the matching degree exceeds a preset threshold, a rule evolution mechanism is triggered. For example, when a high similarity to a historical tactile anomaly is detected, the corresponding force control rule is automatically loaded and the decision logic is adjusted. The optimized rule base is integrated into the original decision system through an incremental update mechanism to ensure that the robot can still generate continuous and reliable action commands when encountering unforeseen sensor conflicts or environmental disturbances.

[0073] Compared to existing technologies, traditional methods rely on fixed rule bases to handle sensor anomalies, making them unable to adapt to novel complex anomaly patterns in dynamic environments. For example, existing technologies use manually preset thresholds to determine conflicts between visual and tactile data, which can easily lead to misjudgments when encountering sudden changes in material stiffness or electromagnetic interference. This solution automatically mines implicit correlations between multimodal data through self-supervised learning and combines this with a digital twin simulation verification mechanism, enabling dynamic expansion of the rule base's processing capabilities. For instance, in disaster relief scenarios, when lidar data is missing due to dust interference, the system can automatically activate redundant decision rules and adjust motion trajectory planning strategies based on historical tactile and visual complex anomaly pattern matching.

[0074] Through the above technical solutions, this application achieves dynamic optimization of the decision rule base, solving the problem of static rules failing to match in unknown scenarios. By integrating multimodal historical anomaly features with simulated scenario verification, the robot's recognition accuracy for complex anomaly patterns is improved. The adoption of a self-supervised learning mechanism avoids the cost of manual annotation, ensuring that the rule evolution process can autonomously adapt to environmental changes. For example, in industrial sorting tasks, when the robotic arm contacts a flexible object, causing a conflict between tactile and visual data, the system can quickly adjust the gripping force control rules based on historical similar anomalies to prevent object damage or gripping failure.

[0075] In this embodiment of the invention, it is necessary to further explain that the multi-physics coupling compensation mechanism monitors the temperature change rate and electromagnetic interference intensity in real time. When the temperature change rate or electromagnetic interference intensity exceeds a preset threshold, an online calibration algorithm is triggered to correct the tactile sensor data, visual sensor data, and lidar sensor data, thereby suppressing the impact of environmental noise on the reliability weight allocation and ensuring the accuracy of weight adjustment.

[0076] Among them, the temperature change rate refers to the magnitude of change in ambient temperature per unit time. Specifically, it can be monitored in real time using thermocouple arrays or infrared temperature sensors. Its function is to capture the interference of thermodynamic effects on the pressure gradient characteristics of tactile sensors. The electromagnetic interference intensity refers to the degree of interference of the electromagnetic field in the environment on the sensor signal. Specifically, it can be dynamically measured using a magnetic field strength meter or radio frequency sensor. Its function is to identify the disturbance of electromagnetic field anomalies on lidar point cloud data or visual sensor imaging quality. The online calibration algorithm refers to the compensation method for dynamically correcting sensor data based on physical field parameters. Specifically, it can use a heat conduction model to compensate for the baseline offset of the tactile sensor caused by the temperature gradient, or use an electromagnetic shielding coefficient matrix to correct the noise data of lidar. Its function is to suppress noise propagation from the source of environmental interference and ensure that the multimodal reliability assessment model allocates weights based on the corrected data.

[0077] Specifically, when the rate of change in ambient temperature exceeds a preset threshold (e.g., baseline drift of the tactile sensor due to thermal expansion of a metal structure) or the intensity of electromagnetic interference reaches the device's sensitivity threshold (e.g., increased noise in the point cloud of a lidar system near high-voltage equipment), an online calibration algorithm is triggered to dynamically compensate for the interfered sensor data. The tactile sensor data eliminates baseline shift caused by temperature gradients using a thermal conduction model; the visual sensor data removes image noise using an electromagnetic shielding coefficient matrix; and the lidar data uses an electromagnetic interference correction algorithm based on the Lorentz force equation to repair point cloud distortion. The calibrated multimodal data is then input into a multimodal reliability assessment model, and the reliability weights of each sensing channel are recalculated using real-time physical field parameters to avoid inaccurate weight allocation due to environmental noise.

[0078] Compared with existing technologies, traditional methods, which use static compensation parameters or single physical field monitoring, cannot adapt to multi-physical field coupling interference in dynamic environments. In contrast, this solution achieves joint correction of cross-modal data through dynamic monitoring and collaborative compensation of multi-physical fields under the condition of dual threshold triggering of temperature change rate and electromagnetic interference intensity. This can effectively solve the problem of weight allocation deviation caused by multi-source interference in complex environments.

[0079] Example 2, see Figure 2 The present invention provides a block diagram of a robot task planning system, and provides the following technical solution in this embodiment: an embodied intelligent robot task planning system based on multi-dimensional situational awareness, comprising:

[0080] The spatiotemporal alignment module uses a pulse neural network to perform frequency domain adaptive interpolation on asynchronous heterogeneous data from multimodal sensor channels (sensor devices include at least tactile, visual, and lidar sensors) to eliminate spatiotemporal misalignment of high-frequency / low-frequency signals, output synchronized multimodal data, and output it to the feature fusion module.

[0081] The feature fusion module utilizes a cross-modal feature fusion network (including a spatiotemporal attention mechanism and recursive units) to extract spatiotemporal correlation features from multimodal data, generate a fusion situation matrix containing semantic information (including multimodal features such as tactile pressure gradient, visual target recognition, and LiDAR spatial topology), and transmit it to the causal modeling module and the reliability assessment module.

[0082] The causal modeling module, based on dynamic causal modeling (Bayesian weight update algorithm), analyzes the strength of causal associations between multimodal data in real time, detects sensor data conflicts (such as the contradiction between tactile residue and the disappearance of visual impairment), outputs an updated sensor causal association map (including abnormal trigger markers), and transmits it to the abnormal source tracing module.

[0083] The anomaly tracing module simulates the distribution of anomalous data through a counterfactual reasoning engine (built based on adversarial networks and a historical anomaly database), identifies anomaly types (such as electromagnetic interference and sensor failure), generates interpretable anomaly tracing results (including propagation paths and type labels), outputs anomaly type labels and propagation path reports, and transmits them to the trust assessment module.

[0084] The reliability assessment module, combining the dynamic field strength distribution map (electromagnetic gradient, acoustic disturbance) and anomaly tracing results, calculates the spatiotemporal consistency score and environmental interference intensity of each sensor channel through a multimodal reliability assessment model, dynamically adjusts the reliability weights, outputs the reliability weight matrix of the sensor channel (e.g., tactile weight decreases, lidar weight increases), and transmits it to the collaborative control module.

[0085] The collaborative control module, based on reliability weights and a fusion situation matrix, performs collaborative predictive control (trajectory similarity measurement + collision risk calculation) by combining the real robot dynamics model and the digital twin virtual model, generating a safety score and outputting action commands; it outputs robot joint control commands and safety scores, and triggers the rule evolution module when the safety score is lower than a threshold.

[0086] The rule evolution module uses reinforcement learning and anomaly pattern knowledge graphs to verify and optimize the decision rule base in the digital twin environment (such as adding obstacle avoidance constraints and adjusting grip force thresholds), dynamically updates rules to adapt to extreme scenarios, and outputs an updated safety decision rule base (such as emergency shutdown rules and dynamic obstacle avoidance strategies). The optimized rule base is fed back to the collaborative control module to form a closed-loop iteration.

[0087] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A task planning method for embodied intelligent robots based on multi-dimensional situational awareness, characterized in that, Includes the following steps: Step 1: Use a spiking neural network to perform spatiotemporal alignment on asynchronous heterogeneous data generated by multimodal sensor channels. Employ a frequency-domain adaptive interpolation algorithm to achieve data synchronization, dynamically adjust the interpolation accuracy based on the target's motion speed, and extract cross-modal spatiotemporal features through a cross-modal feature fusion network. Establish feature correlation between different sensor channels through a cross-attention mechanism, and generate a fusion situational matrix containing multimodal semantic features by combining temporal dependency modeling. Quantify the correlation between tactile sensor data and visual sensor data using a self-attention mechanism, and fuse the spatial topological features of LiDAR sensor data under the cross-attention mechanism to generate a fusion situational matrix. Step 2: Based on the fused situation matrix, use dynamic causal modeling to update the correlation strength between multimodal data and reconstruct causal relationships in real time; identify and trace anomalies based on the anomaly tracing strategy of generative adversarial networks, combine generative adversarial networks to simulate data distribution under anomaly conditions, and output anomaly tracing results; Step 3: Utilizing the fused situational matrix and anomaly tracing results, a graph neural network is used to process the dynamic field strength distribution map and real-time physical field parameters. A multimodal reliability assessment model is then used to dynamically adjust the reliability weights of each sensing channel. The graph neural network generates the dynamic field strength distribution map based on synchronous localization and mapping technology. The multimodal reliability assessment model integrates an environmental physical field analysis unit and a sensing error compensation unit. Specifically, the environmental physical field analysis unit constructs the dynamic field strength distribution map through electromagnetic field gradient monitoring and acoustic disturbance detection. The sensing error compensation unit employs a multi-physics coupling correction algorithm to generate a sensing data correction matrix based on real-time physical field parameters and calculates the reliability weights of each sensing channel by combining semantic segmentation confidence indicators. Step 4: Based on reliability weights, the fused situation matrix is ​​input into the real robot dynamics model and the digital twin virtual model to execute collaborative predictive control. The spatial deviation between the real trajectory point sequence and the virtual trajectory point sequence is calculated through a trajectory similarity measurement algorithm. The collision risk probability is calculated by combining the three-dimensional risk field and generated a safety score. When the safety score is lower than the threshold, an adaptive rule evolution mechanism is activated to ensure the safety of the decision.

2. The task planning method for embodied intelligent robots based on multi-dimensional situational awareness according to claim 1, characterized in that, Asynchronous heterogeneous data includes low-frequency sensor data and high-frequency sensor data, and the frequency domain adaptive interpolation algorithm includes the following steps: Frequency domain decomposition is performed on low-frequency sensor data to extract the dominant frequency component; High-frequency sensor data is encoded into pulse sequences, and the pulse firing threshold is dynamically set according to the sensor type. The firing frequency is non-linearly related to the target's speed. The interpolation window length is adjusted based on motion speed, and spline interpolation is used to fill in missing frames of low-frequency data; the timestamp alignment error of multi-channel data is verified, and synchronous calibration is triggered when the error exceeds the tolerance.

3. The task planning method for embodied intelligent robots based on multi-dimensional situational awareness according to claim 1, characterized in that, The dynamic causal modeling adopts a Bayesian weight update algorithm to reconstruct the causal relationship between multimodal sensor data in real time. When the causal correlation between sensor data deviates from the preset dynamic error threshold, anomaly detection is triggered. The counterfactual reasoning engine simulates the data distribution under abnormal conditions by generating adversarial networks and combines it with the historical anomaly pattern database to output anomaly tracing results containing anomaly type labels and propagation paths.

4. The task planning method for embodied intelligent robots based on multi-dimensional situational awareness according to claim 1, characterized in that, The security score is obtained as follows: The digital twin virtual model is built through a physics engine and includes state synchronization, environment mapping and latency compensation mechanisms. It receives joint state data of the real robot in real time and generates a three-dimensional map of the virtual environment. A trajectory similarity measurement algorithm is used to calculate the spatial deviation between the real trajectory point sequence and the virtual trajectory point sequence, and the trajectory deviation degree is generated by a normalized mapping function. An obstacle occupancy probability model is constructed based on LiDAR data after multi-physics coupling correction. The model is dynamically corrected by fusing the confidence level of the visual sensor and the pressure gradient characteristics of the tactile sensor. The collision risk probability is calculated by superimposing a three-dimensional risk field. A safety score is generated by weighting the trajectory deviation and the probability of collision risk.

5. The task planning method for embodied intelligent robots based on multi-dimensional situational awareness according to claim 1, characterized in that, The process of calculating the reliability weight includes: Let i represent the sensor channel number index, and N represent the total number of sensor channels; Spatiotemporal consistency quantification: Based on multimodal consistency testing, calculate the spatiotemporal consistency score of the i-th sensing channel. ,in Let be the real-time measurement vector of the i-th sensing channel. The cross-modal prediction value is the prediction value generated by the Transformer-LSTM hybrid network through data from other sensor channels; cos(·) is the vector cosine similarity calculation function, used to quantify the directional consistency between real-time data and prediction values; Environmental interference intensity quantification: The environmental interference intensity of the i-th sensing channel is calculated using the following formula: ; in, The electromagnetic field gradient magnitude was obtained through differential calculation using a triaxial magnetic field sensor. fL is the sound pressure power spectral density, extracted from the acoustic sensor array via fast Fourier transform; fH is the lower limit frequency of integration; fH is the upper limit frequency of integration. , The sensitivity coefficient is related to the sensor type and is a scaling factor obtained by fitting the interference-error curve in the sensor calibration experiment. Task Relationship Measurement: Based on the security requirement level of the current task phase. With operational accuracy requirements The value ranges from 0.1 to 1.0, used to calculate the task criticality factor. ; Indicates task priority; Dynamic weight composition: will be denoted as The reliability weight of the i-th sensing channel is calculated using a weighted fusion function. ,in A small constant to prevent division by zero errors; ； in, It is a small positive number, and in this embodiment of the invention, its value range is 0.01-0.0001, which is used to prevent calculation overflow caused by an excessively small denominator.

6. The task planning method for embodied intelligent robots based on multi-dimensional situational awareness according to claim 4, characterized in that, The collaborative predictive control calculates the output deviation between the physical model and the digital twin virtual model through a trajectory similarity measurement algorithm. When the output deviation exceeds a preset trajectory fault tolerance threshold, safety constraint optimization is activated. The adaptive rule evolution mechanism includes rule credibility evaluation and logical structure optimization. It verifies the effectiveness of rules in a simulation environment through reinforcement learning and dynamically updates the constraints and optimization objectives in the decision rule base. The adaptive rule evolution mechanism includes the following steps: Constructing anomaly pattern knowledge graph: Based on historical anomaly events from tactile, visual, and lidar sensor data, extract spatiotemporal correlation features and generate the topology of anomaly propagation paths; Incremental optimization of the rule base: By comparing the similarity between real-time abnormal scenarios and historical knowledge graphs through a comparative learning algorithm, the constraints in the decision rule base are dynamically updated, and rules with proven effectiveness are retained first. Unsupervised verification mechanism: Extreme scenarios are simulated in the digital twin virtual model to verify the stability of newly added rules and eliminate redundant rules that cause a decrease in security score.

7. The task planning method for embodied intelligent robots based on multi-dimensional situational awareness according to claim 1, characterized in that, The multi-physics coupling correction algorithm includes an interference source localization unit and a dynamic weight redistribution unit. When the electromagnetic field intensity gradient exceeds the device's anti-interference threshold or the acoustic disturbance frequency enters the sensor's sensitive frequency band, the interference source localization unit is activated to perform error compensation. The interference source localization unit includes: three-dimensional vector data based on electromagnetic field gradient, establishing a spatial distribution probability model of interference source based on Lorentz force equation, calculating the maximum likelihood electromagnetic interference source coordinates; using generalized cross-correlation algorithm to process the time delay estimation of acoustic sensor array, and constructing the acoustic interference source location information interval through the sound wave arrival time difference; The dynamic weight redistribution unit includes: when the Frobenius norm of the perturbation coefficient matrix composed of environmental physical field parameters exceeds the adaptive threshold, a weight optimization algorithm based on Lyapunov stability theory is triggered to redistribute the reliability weights of the multimodal sensor data.

8. The task planning method for embodied intelligent robots based on multi-dimensional situational awareness according to claim 1, characterized in that, The multi-physics coupling compensation mechanism monitors the temperature change rate and electromagnetic interference intensity in real time. When the temperature change rate or electromagnetic interference intensity exceeds the preset threshold, it triggers an online calibration algorithm to correct the tactile sensor data, visual sensor data, and lidar sensor data, thereby suppressing the impact of environmental noise on the reliability weight allocation.

9. A task planning system for an embodied intelligent robot based on multi-dimensional situational awareness, used to implement the task planning method for an embodied intelligent robot as described in claim 1, characterized in that, include: The spatiotemporal alignment module uses a spiking neural network to perform frequency domain adaptive interpolation on the asynchronous heterogeneous data from the multimodal sensor channels, eliminating spatiotemporal misalignment, outputting synchronized multimodal data, and then outputting it to the feature fusion module. The feature fusion module uses a cross-modal feature fusion network to extract spatiotemporal correlation features from multimodal data, generates a fusion situation matrix containing semantic information, and transmits it to the causal modeling module and the credibility assessment module. The causal modeling module, based on dynamic causal modeling, analyzes the strength of causal relationships between multimodal data in real time, detects sensor data conflicts, outputs an updated sensor causal relationship map, and transmits it to the anomaly tracing module. The anomaly tracing module simulates the distribution of anomalous data through a counterfactual reasoning engine, identifies anomaly types, generates interpretable anomaly tracing results, outputs anomaly type labels and propagation path reports, and transmits them to the trust assessment module. The reliability assessment module, combining the dynamic field strength distribution map and anomaly tracing results, calculates the spatiotemporal consistency score and environmental interference intensity of each sensor channel through a multimodal reliability assessment model, dynamically adjusts the reliability weights, outputs the reliability weight matrix of the sensor channel, and transmits it to the collaborative control module. The collaborative control module, based on reliability weights and a fusion situation matrix, combines the real robot dynamics model and the digital twin virtual model to perform collaborative predictive control, generate a safety score, and output action commands. Output robot joint control commands and safety score; trigger the rule evolution module when the safety score is below the threshold. The rule evolution module uses reinforcement learning and anomaly pattern knowledge graphs to verify and optimize the decision rule base in a digital twin environment, dynamically updates rules to adapt to extreme scenarios, and outputs an updated security decision rule base. The optimized rule base is then fed back to the collaborative control module, forming a closed-loop iteration.

Images