Robot dynamic risk assessment and decision-making system and method based on multi-modal perception

Abstract

The invention relates to the technical field of intelligent assessment and decision making, in particular to a robot dynamic risk assessment and decision making system and method based on multi-modal perception, and the system comprises a multi-modal sensor module which is used for collecting environment vision, acoustics, mechanics and position data in real time; the edge calculation unit is used for carrying out space-time alignment and feature fusion on the sensor data; the dynamic risk assessment model is used for integrating the environment uncertainty quantification module and the robot state prediction module based on a reinforcement learning framework; the decision execution interface is used for outputting a risk level and obstacle avoidance, speed reduction and shutdown instructions; by integrating visual, acoustic, mechanical and position multi-source sensor data and the like, the system can comprehensively capture various risk factors in a complex dynamic environment, so that the defect that a traditional single sensor system is insufficient in sensing dimension is overcome, and the system is particularly suitable for terrains and weather conditions with variable regions.

Description

Technical Field

[0001] The present invention relates to the field of intelligent assessment and decision-making technology, and in particular to a robot dynamic risk assessment and decision-making system and method based on multimodal perception. Background Art

[0002] With the rapid development of industrial automation and intelligent robotics technology, the ability of robots to operate autonomously in complex environments has become a research hotspot. In areas such as power inspection, intelligent manufacturing, and disaster relief, robots need to respond to changing environmental risks in real time, such as terrain changes, equipment failures, and sudden obstacles. Traditional robotic systems mostly rely on a single sensor for environmental perception, which makes it difficult to fully capture the multi-dimensional risk factors in complex scenarios. This is especially true in areas with complex geographical environments and changeable climates, where the limitations of single-modal data are more obvious.

[0003] Most systems in existing technologies use static risk assessment models and are unable to adapt to changes in dynamic environments. For example, when operating in mountainous areas, sudden fog or rainfall may cause distortion of sensor data, and traditional models find it difficult to adjust risk assessment strategies in a timely manner. Existing methods mostly focus on a single risk type, such as mechanical collision, and lack a comprehensive analysis of multiple risk factors, resulting in insufficient comprehensiveness in risk prediction. Some studies in existing technologies have attempted to introduce multi-sensor fusion technology, but usually use a fixed-weight data fusion method, which cannot dynamically adjust the contribution of each sensor according to the environment, resulting in insufficient robustness in complex scenarios. For example, in humid environments, lidar is easily interfered by water mist, and traditional fusion algorithms find it difficult to reduce its weight in a timely manner, thereby increasing the possibility of misjudgment.

[0004] Therefore, in response to the above-mentioned problems, the present invention proposes a robot dynamic risk assessment and decision-making system and method based on multimodal perception. Through multi-source sensor fusion, hierarchical risk modeling and cloud-based collaborative optimization, real-time quantification and trend prediction of risks in complex environments are achieved. The system improves the autonomy and reliability of robots in dynamic environments. Summary of the Invention

[0005] In order to overcome the problem of incomplete perception of dynamic environments in the existing technology, the present invention proposes a robot dynamic risk assessment and decision-making system and method based on multimodal perception.

[0006] The technical solution of the present invention is: a robot dynamic risk assessment and decision-making system based on multimodal perception, including: Multimodal sensor module for real-time collection of environmental visual, acoustic, mechanical and positional data; Edge computing unit, used for spatiotemporal alignment and feature fusion of sensor data; A dynamic risk assessment model that integrates the environmental uncertainty quantification module and the robot state prediction module based on a reinforcement learning framework; Decision execution interface, used to output risk level, obstacle avoidance, speed reduction and shutdown instructions.

[0007] Preferably, the multimodal sensor module includes a binocular depth camera, a lidar, an inertial measurement unit and a voiceprint sensor. The binocular depth camera is used for three-dimensional obstacle recognition, the lidar is used for high-precision distance mapping, the inertial measurement unit is used for robot motion posture monitoring, and the voiceprint sensor is used for abnormal mechanical noise detection.

[0008] Preferably, the dynamic risk assessment model adopts a hierarchical architecture, which is divided into bottom layer, middle layer and high layer. The bottom layer calculates risk factors in real time through Bayesian network and outputs quantitative risk values; the middle layer predicts risk trends based on historical operation database and industry knowledge graph, and analyzes the law of risk evolution; the high layer is used to generate the optimal decision path and adjust the robot behavior strategy.

[0009] Preferably, the industry knowledge graph includes a typical failure mode library of industrial robots and a dynamic environment risk case library. By mining the association rules between risk factors, the system can quickly match historical risk patterns based on current multimodal perception data, and output probabilistic risk evolution paths and preventive decision-making recommendations.

[0010] Preferably, the system adopts a cloud-based collaborative architecture to achieve distributed risk processing. The local end completes real-time risk assessment and executes emergency decisions through a lightweight model, while the cloud uses federated learning technology to aggregate the operating data of multiple robots to optimize the global risk assessment model.

[0011] Preferably, a visual dashboard is deployed on the cloud, which displays regional risk distribution in the form of a heat map and draws a health decline warning curve based on the real-time status data of the equipment.

[0012] Preferably, the decision execution interface includes a human-computer collaborative interaction mode. When the system detects that the risk level exceeds a set threshold, it automatically pushes an alarm signal to the operator, and can also receive manual decision override instructions and record audit logs.

[0013] Preferably, the robot dynamic risk assessment and decision-making method based on multimodal perception includes the following steps: S1, obtains multimodal data of the robot's working environment through multi-source sensors; S2, uses attention mechanism to perform weighted fusion of heterogeneous sensor data; S3, building real-time risk simulation scenarios based on digital twin technology; S4, outputs a decision plan including economic evaluation.

[0014] Preferably, step S2 adopts a heterogeneous data fusion method based on a multi-head attention mechanism. First, the multimodal data of visual, acoustic, mechanical and position data are preprocessed, and then the confidence score of each sensor in the current environment is calculated through the sensor data quality evaluation index. The quality evaluation index includes signal-to-noise ratio, data integrity and environmental adaptability parameters. The contribution of each sensor data is calculated based on the confidence score through a learnable attention weight matrix, and the weight distribution of each sensor data is adjusted according to the contribution. The weight ratio of the interfering sensor is reduced in real time, and the weight ratio of the sensor with the largest contribution weight is increased. Finally, the unified environment representation vector after fusion is output; the calculation of the contribution specifically includes: Data quality assessment, calculates quality assessment indicators for each sensor input data, including: Signal-to-noise ratio (SNR), which separates noise components through frequency domain filtering and calculates the ratio of signal power to noise power; Data integrity, detecting sensor data packet loss rate and timestamp continuity; Environmental adaptability parameters match the preset sensor reliability coefficient based on the environment type (such as humidity > 80% is defined as a humid environment).

[0015] Confidence score generation: input the above indicators into the fully connected layer neural network and output a confidence score of 0-1; Attention weight allocation: input the confidence score into the attention weight matrix and calculate the contribution weight of each sensor to the fusion result to meet the following requirements: Contribution weight ; in, represents the exponential processing of the confidence score of the i-th sensor; Represents the exponential sum of all sensor confidence scores.

[0016] Preferably, the construction of the risk simulation scenario in step S3 includes: constructing a typical industrial scenario model by importing geographic information and industrial data, including geographic information and industrial data, and then simulating equipment loss costs and task delay costs under different decisions.

[0017] Beneficial effects of the present invention:

[0018] 1. By integrating multi-source sensor data such as vision, acoustics, mechanics, and position, the system can comprehensively capture various risk factors in complex dynamic environments, overcoming the shortcomings of traditional single-sensor systems with insufficient perception dimensions. It is particularly suitable for regions with diverse terrain and climatic conditions.

[0019] 2. The risk assessment model based on the reinforcement learning framework can quantify environmental uncertainty in real time and predict robot state changes, solving the problem that static models cannot respond to dynamic environments, thereby ensuring that accurate risk levels can still be output under sudden interference such as fog or vibration.

[0020] 3. By implementing real-time fusion and feature extraction of sensor data through the edge computing unit and combining it with the rapid response mechanism of the decision execution interface, the system can complete closed-loop control from perception to decision-making in milliseconds, significantly improving the robot's risk avoidance capabilities in high-risk scenarios.

[0021] 4. Through local real-time processing and cloud-based model updates, it not only meets the need for low-latency decision-making, but also optimizes the global risk model through continuous learning, avoiding system misjudgments caused by sensor failure or sudden environmental changes, thereby enhancing the stability of long-term operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 What is shown is a schematic diagram of the workflow of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0024] The present invention provides an embodiment: a robot dynamic risk assessment and decision-making system based on multimodal perception, comprising: Multimodal sensor module for real-time collection of environmental visual, acoustic, mechanical and positional data; Edge computing unit, used for spatiotemporal alignment and feature fusion of sensor data; A dynamic risk assessment model that integrates the environmental uncertainty quantification module and the robot state prediction module based on a reinforcement learning framework; Decision execution interface, used to output risk level, obstacle avoidance, speed reduction and shutdown instructions.

[0025] The multimodal sensor module first collects the visual, acoustic, mechanical and position data of the robot's working environment in real time, and performs spatiotemporal alignment and feature fusion processing on the heterogeneous sensor data through the edge computing unit. The dynamic risk assessment model is based on the reinforcement learning framework, and synchronously runs the environmental uncertainty quantification module and the robot state prediction module. Then, the fused multimodal data is hierarchically analyzed, the real-time risk value is calculated and the risk evolution trend is predicted. Finally, the decision execution interface generates control instructions of different levels according to the risk assessment results, and transmits the instructions to the robot actuator. The present invention significantly improves the comprehensiveness and accuracy of environmental information acquisition through the collaborative perception of multi-source sensors, the edge computing unit ensures the real-time performance of data processing, the reinforcement learning framework enables the system to dynamically adapt to changes in complex environments, and the hierarchical decision-making mechanism realizes a rapid response from risk perception to execution control, which overall solves the problems of traditional systems in dynamic environments with single perception dimensions, delayed risk assessment and low decision-making efficiency.

[0026] The multimodal sensor module includes a binocular depth camera, a lidar, an inertial measurement unit and a voiceprint sensor. The binocular depth camera is used for three-dimensional obstacle recognition, the lidar is used for high-precision distance mapping, the inertial measurement unit is used for robot motion posture monitoring, and the voiceprint sensor is used for abnormal mechanical noise detection.

[0027] Furthermore, the binocular depth camera uses a stereo vision algorithm to construct three-dimensional environmental point cloud data in real time, accurately identifying the shape and distance of obstacles in the workspace. The laser radar simultaneously performs 360-degree high-speed scanning to generate a high-precision two-dimensional distance map to supplement the visual blind spot data. The inertial measurement unit continuously monitors the robot's motion parameters such as acceleration and angular velocity, and outputs the body posture information in real time through a posture solution algorithm. The voiceprint sensor collects the acoustic characteristic spectrum of mechanical operation, compares it with a preset fault voiceprint library, and identifies potential equipment anomalies. The data of each sensor is synchronized through hardware timestamps and input into the edge computing unit. The collaboration between the binocular camera and the laser radar of the present invention significantly improves the obstacle recognition rate in complex environments. The precise posture data provided by the inertial measurement unit effectively suppresses sensor data drift caused by robot movement. The voiceprint detection enables early warning of mechanical failures that are difficult to detect with traditional vision / laser systems. The multi-sensor spatiotemporal alignment technology solves the problem of information asynchrony when heterogeneous data is fused. Overall, a comprehensive and highly reliable environmental perception system is constructed, providing an accurate input foundation for subsequent risk assessment.

[0028] Furthermore, the underlying Bayesian network in the dynamic risk assessment model receives multimodal sensor fusion data in real time and calculates the joint probability distribution of various risk events using a probabilistic graphical model. It then outputs core indicators such as collision risk value and equipment failure probability. The middle layer utilizes time series data from the historical operation database and, combined with the semantic relationships of the industry knowledge graph, uses a time series prediction algorithm to deduce the development trajectory of the current risk situation. The high-level decision engine, based on the Q-learning reinforcement learning framework, uses the underlying risk quantification results and the middle-level trend prediction as state inputs. It uses task completion, energy consumption cost, and safety threshold as reward functions and generates the optimal decision strategy through value iteration. The technical benefits of this model are as follows: the Bayesian network enables parallel probabilistic reasoning of multiple risk factors in uncertain environments; the introduction of the middle-level knowledge graph enables the system to predict situations based on industry experience; and the reinforcement learning framework optimizes the risk-efficiency balance strategy through autonomous exploration. The collaborative work of the three-layer architecture ensures millisecond-level real-time risk assessment and enables intelligent prediction of long-term risk evolution. Ultimately, this enables the robot to dynamically adjust its behavior in complex operating environments, significantly improving the accuracy and adaptability of decision-making compared to traditional single-layer assessment models.

[0029] The industry knowledge graph includes a typical failure mode library of industrial robots and a dynamic environment risk case library. By mining the association rules between risk factors, the system can quickly match historical risk patterns based on current multimodal perception data, and output probabilistic risk evolution paths and preventive decision-making recommendations.

[0030] Furthermore, the industry knowledge graph is manifested as follows: the system first extracts typical failure modes from the operation and maintenance logs of industrial robots, such as structured data such as the robot arm overload code E201 and the motor overheating alarm threshold of 85°C, and integrates environmental risk cases unique to the region. A topological network consisting of nodes (risk factors) and edges (causal relationships) is constructed through a graph database. When multimodal sensors input real-time data, the graph neural network automatically activates the associated sub-graph, compares the current sensor readings (such as IMU detecting continuous high-frequency vibration) with the similarity of historical cases, and outputs the risk transmission path (vibration → loose screws → robot arm positioning deviation) and the probability of occurrence; the present invention converts fragmented industry experience into computable graph relationships, thereby realizing early tracing of faults and cross-dimensional risk association analysis, so that the system can not only identify explicit risks, but also predict potential derivative risks. Compared with traditional rule-based diagnostic methods, it reduces the false alarm rate and improves the timeliness of early warning.

[0031] Furthermore, the system deploys a lightweight risk assessment model locally, processes the data of multimodal sensors in real time through the edge computing unit, and immediately triggers obstacle avoidance or emergency stop instructions when an emergency risk is detected. At the same time, the running data is uploaded to the cloud server. The cloud uses a federated learning framework to aggregate data from different regions, and updates the global risk assessment model parameters through gradient exchange. The updated model parameters are regularly sent to each terminal device, and the local model achieves dynamic evolution through incremental learning. The edge computing of the present invention ensures the real-time nature of risk response, and the federated learning mechanism solves the data island problem. At the same time, the iteration of the model enables the system to adapt to the needs of different scenarios.

[0032] A visual dashboard is deployed on the cloud, which displays regional risk distribution in the form of a heat map and draws a health decline warning curve based on the real-time status data of the equipment.

[0033] Furthermore, when the risk level output by the dynamic risk assessment model exceeds the preset threshold, for example, the collision probability is greater than 60% or the equipment temperature is greater than the warning value, the system automatically triggers a three-level alarm protocol. Among them, the primary risk (30%-60% probability) starts the operation interface with a flashing yellow warning and a buzzer alarm, the intermediate risk (60%-80%) superimposes a red border on the screen and intermittent vibration reminders, and the high risk (>80%) activates a full-screen red overlay and continuous buzzing. The operator can make decision interventions through the physical emergency stop button or the touch screen confirmation box. All operation records are stored in the audit log, which includes the manual decision time, operator ID, override reason and the system's original recommended plan.

[0034] See also Figure 1 Furthermore, the present invention provides an embodiment of a robot dynamic risk assessment and decision-making method based on multimodal perception, comprising the following steps: The system synchronously acquires multi-dimensional data of the environment through a distributed sensor array. The binocular stereo camera collects RGB-D images and generates a depth point cloud. The lidar performs a 360° scan and outputs a two-dimensional polar coordinate distance matrix. The inertial measurement unit captures the three-axis acceleration and angular velocity data of the robot's motion. The acoustic sensor collects mechanical vibration audio through a microphone array. After all sensor data are aligned by timestamp, a sliding window mechanism is used to segment the data to ensure timing consistency.

[0035] After converting the data of each sensor into a unified feature vector, a heterogeneous data fusion method based on a multi-head attention mechanism is adopted. First, the multimodal data of visual, acoustic, mechanical and position data are temporally aligned and feature encoded. Then, the confidence score of each sensor in the current environment is calculated using the sensor data quality assessment index. This quality assessment index includes signal-to-noise ratio, data integrity and environmental adaptability parameters. The contribution of each sensor data is calculated based on the confidence score through a learnable attention weight matrix. The weight distribution of each sensor data is adjusted according to the contribution, reducing the weight of the interfering sensor in real time and increasing the weight of the sensor with the largest contribution weight. For example, in a humid environment, the weight of the lidar is automatically reduced by 30%, and the weight of the visual data is increased. Finally, the unified environmental representation vector after fusion is output. The calculation of the contribution specifically includes: Data quality assessment, calculates quality assessment indicators for each sensor input data, including: Signal-to-noise ratio (SNR), which separates noise components through frequency domain filtering and calculates the ratio of signal power to noise power; Data integrity, detecting sensor data packet loss rate and timestamp continuity; Environmental adaptability parameters match the preset sensor reliability coefficient based on the environment type (such as humidity > 80% is defined as a humid environment).

[0036] Confidence score generation: input the above indicators into the fully connected layer neural network and output a confidence score of 0-1; Attention weight allocation: input the confidence score into the attention weight matrix and calculate the contribution weight of each sensor to the fusion result to meet the following requirements: Contribution weight ; in, represents the exponential processing of the confidence score of the i-th sensor; Represents the exponential sum of all sensor confidence scores.

[0037] Furthermore, the present invention provides an embodiment: Applied to the material transportation scenario in the western Sichuan Plateau, the robot uses the dynamic risk assessment model of the present invention, in which the underlying Bayesian network calculates the risk of steep slope slippage in real time, the middle-level knowledge graph matches historical failure cases caused by plateau permafrost, and the high-level decision-making module selects the optimal solution of installing anti-skid chains, thereby reducing the interruption rate of transportation tasks by 67%.

[0038] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A robot dynamic risk assessment and decision-making system based on multimodal perception, characterized by: Includes: Multimodal sensor module for real-time collection of environmental visual, acoustic, mechanical and positional data; Edge computing unit, used for spatiotemporal alignment and feature fusion of sensor data; A dynamic risk assessment model that integrates the environmental uncertainty quantification module and the robot state prediction module based on a reinforcement learning framework; Decision execution interface, used to output risk level, obstacle avoidance, speed reduction and shutdown instructions.

2. The multimodal perception-based robot dynamic risk assessment and decision-making system according to claim 1, characterized in that: The multimodal sensor module includes a binocular depth camera, a lidar, an inertial measurement unit and a voiceprint sensor. The binocular depth camera is used for three-dimensional obstacle recognition, the lidar is used for high-precision distance mapping, the inertial measurement unit is used for robot motion posture monitoring, and the voiceprint sensor is used for abnormal mechanical noise detection.

3. The multimodal perception-based robot dynamic risk assessment and decision-making system according to claim 1, characterized in that: The dynamic risk assessment model adopts a hierarchical architecture, which is divided into the bottom layer, the middle layer and the top layer. The bottom layer calculates the risk factor in real time through the Bayesian network and outputs the quantitative risk value; The middle layer predicts risk trends and analyzes risk evolution patterns based on historical operation databases and industry knowledge graphs; The high-level layers are used to generate optimal decision paths and adjust the robot's behavior strategies.

4. The multimodal perception-based robot dynamic risk assessment and decision-making system according to claim 3 is characterized by: The industry knowledge graph includes a typical failure mode library of industrial robots and a dynamic environment risk case library. By mining the association rules between risk factors, the system can quickly match historical risk patterns based on current multimodal perception data, and output probabilistic risk evolution paths and preventive decision-making recommendations.

5. The multimodal perception-based robot dynamic risk assessment and decision-making system according to claim 1, characterized in that: The system adopts a cloud-based collaborative architecture to achieve distributed risk processing. The local end uses a lightweight model to complete real-time risk assessment and execute emergency decisions, while the cloud uses federated learning technology to aggregate the operating data of multiple robots to optimize the global risk assessment model.

6. The multimodal perception-based robot dynamic risk assessment and decision-making system according to claim 5, characterized in that: A visual dashboard is deployed on the cloud, which displays regional risk distribution in the form of a heat map and draws a health decline warning curve based on the real-time status data of the equipment.

7. The multimodal perception-based robot dynamic risk assessment and decision-making system according to claim 1, characterized in that: The decision execution interface includes a human-machine collaborative interaction mode. When the system detects that the risk level exceeds the set threshold, it automatically pushes an alarm signal to the operator, and can also receive manual decision override instructions and record audit logs.

8. A robot dynamic risk assessment and decision-making method based on multimodal perception, comprising: a robot dynamic risk assessment and decision-making system based on multimodal perception according to any one of claims 1 to 7, characterized in that: The following steps are included: S1, obtains multimodal data of the robot's working environment through multi-source sensors; S2, uses attention mechanism to perform weighted fusion of heterogeneous sensor data; S3, building real-time risk simulation scenarios based on digital twin technology; S4, outputs a decision plan including economic evaluation.

9. The robot dynamic risk assessment and decision-making method based on multimodal perception according to claim 8 is characterized by: Step S2 adopts a heterogeneous data fusion method based on a multi-head attention mechanism. First, the multimodal data of visual, acoustic, mechanical and position data are preprocessed. Then, the confidence score of each sensor in the current environment is calculated using the sensor data quality assessment index. The quality assessment index includes signal-to-noise ratio, data integrity and environmental adaptability parameters. The contribution of each sensor data is calculated based on the confidence score through a learnable attention weight matrix. The weight distribution of each sensor data is adjusted according to the contribution, and the weight proportion of the interfering sensor is reduced in real time, and the weight proportion of the sensor with the largest contribution weight is increased. Finally, the fused unified environment representation vector is output; The calculation of contribution specifically includes: Data quality assessment: Calculate quality assessment indicators for each sensor input data. Quality assessment indicators include: signal-to-noise ratio, which separates noise components through frequency domain filtering and calculates the ratio of signal power to noise power; data integrity, which detects sensor data packet loss rate and timestamp continuity; environmental adaptability parameters, which match preset sensor reliability coefficients based on environmental type; Confidence score generation: input the quality assessment index into the fully connected layer neural network and output a confidence score of 0-1; Attention weight allocation: input the confidence score into the attention weight matrix and calculate the contribution weight of each sensor to the fusion result to meet the following requirements: Contribution weight ; in, represents the exponential processing of the confidence score of the i-th sensor; represents the exponential sum of all sensor confidence scores; Quality assessment indicators include: Signal-to-noise ratio, which separates the noise components by frequency domain filtering and calculates the ratio of signal power to noise power; Data integrity, detecting sensor data packet loss rate and timestamp continuity; Environmental adaptability parameters, based on the preset sensor reliability coefficients matching the environment type.

10. The robot dynamic risk assessment and decision-making method based on multimodal perception according to claim 8 is characterized in that: The construction of the risk simulation scenario in step S3 includes: constructing a typical industrial scenario model by importing geographic information and industrial data, and then simulating the equipment loss cost and task delay cost under different decisions.

Images