Robot real-time potential safety hazard identification system based on multi-modal sensor fusion

Abstract

The invention discloses a robot real-time potential safety hazard recognition system based on multi-modal sensor fusion, and particularly relates to the technical field of intelligent inspection and safety monitoring, the system comprises five parts of data acquisition, information fusion, behavior response, trajectory analysis and risk output, and the potential safety hazard recognition system is used for recognizing potential safety hazards through image acquisition, thermal imaging, gas concentration and temperature and humidity information. Carrying out numerical value normalization and feature extraction, identifying potential abnormity and generating early warning; triggering data enhanced acquisition and track recording in the target area, and constructing a space-time path model to analyze an abnormal evolution trend; and finally, outputting a potential safety hazard assessment result according to a risk level classification rule by combining the enhanced information and the trajectory features. According to the method, high-precision early warning is realized through multi-source data acquisition and normalization fusion, the local recognition capability is improved based on dynamic enhanced acquisition of a behavior response mechanism, an abnormal development trend is tracked by combining track evolution modeling, risk level assessment is output according to the abnormal development trend, and accurate recognition and dynamic management and control of hidden dangers are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent inspection and safety monitoring, and more specifically, to a robot real-time safety hazard identification system based on multimodal sensor fusion. Background Art

[0002] As the scale of various industrial infrastructure systems continues to grow, the operational complexity of equipment continues to increase, significantly increasing the pressure on safety management. This is especially true in locations with complex spatial structures and variable environmental conditions, such as power substations, pipeline corridors in petrochemical plants, underground distribution shafts, enclosed tunnels, and high-temperature and high-humidity industrial chambers. Traditional methods that rely on manual inspections are not only inefficient and pose safety risks, but also have significant limitations in identifying early signs of faults.

[0003] Currently, automated inspection robots are gradually being used in these scenarios to replace manual labor in some routine monitoring tasks. Some commercial solutions already have the ability to perceive image changes, thermal anomalies, or single environmental factors. However, since most solutions only integrate single-modal sensors or lack effective data fusion mechanisms, their ability to accurately identify common problems under complex working conditions, such as "lack of significant early abnormal signals," "cross-overlapping of multiple types of changes," and "the spatial expansion of hidden danger distribution," remains insufficient. Therefore, the present invention proposes a real-time safety hazard identification system for robots based on multimodal sensor fusion to address these issues. Summary of the Invention

[0004] To achieve the above object, the present invention provides the following technical solutions: The robot real-time safety hazard identification system based on multimodal sensor fusion includes a multi-source data acquisition module, an information fusion processing module, an identification behavior response module, a trajectory joint analysis module, and a risk output control module. Its composition and collaborative relationship are as follows: The multi-source data acquisition module collects image information, thermal imaging information, ambient gas concentration, and temperature and humidity change signals respectively, and synchronously marks the acquisition position and timestamp while in motion; The information fusion processing module normalizes the values ​​of various sensor information according to a preset sequence, constructs a correlation modeling structure between signals through cross-feature extraction, uses a rule-driven approach to identify potential abnormal change characteristics in continuous signals, and forms an initial warning value based on behavioral association logic; The recognition behavior response module receives the initial warning value and determines whether to activate the enhanced collection mechanism according to the threshold comparison rule. If the conditions are met, it will correspondingly enhance the sampling frequency of the relevant information source and synchronously activate the trajectory recording behavior of the target area. The newly generated feature parameters during the response process are used as the enhanced results. The target area is determined by the information fusion processing module based on the spatial location of the abnormal signal when generating the initial warning value, and is used as the starting position for triggering trajectory recording in the recognition behavior response module. This area is mapped to the spatial division grid in the trajectory joint analysis module, and the corresponding grid unit is used as the basis for identifying the active unit. The trajectory joint analysis module accesses the real-time location data of the target area, responds to the updated information and its temporal changes after collection, and constructs a space-time evolution path model to identify the continuous growth of anomalies, changes in movement distribution, or aggregation and diffusion patterns, in order to track the dynamic development process of anomalies. The risk output control module generates corresponding safety hazard assessment conclusions based on the trajectory joint analysis results and the enhanced collection information obtained from the recognition behavior response, combined with the preset risk level classification conditions.

[0005] In a preferred embodiment, image information acquisition is performed using a continuous frame sequence, and inter-frame differences are used to identify image target changes in a background subtraction manner, and combined with the gradient changes of pixel values ​​in thermal imaging to form joint features for locating image change areas; The collection of environmental gas concentration and temperature and humidity change signals is carried out at fixed time intervals. Each collection result is attached with a time stamp and the current position of the robot. The current position is output by inertial and visual fusion. All perception data are numerically normalized before entering the information fusion processing module. The processing method is: determine the first extreme value and the second extreme value from the historical operation records, perform the standardized calculation of (current value minus the first extreme value) divided by (second extreme value minus the first extreme value), and the normalized result is used as a feature for subsequent modeling.

[0006] In a preferred embodiment, the cross-feature extraction method in the information fusion processing module constructs a multi-feature association model by calculating the linear correlation between the image change frequency and the average temperature rise value of the thermal imaging; The image change frequency is the average number of times the changed area appears per unit time, and the average temperature rise value is the average value of the temperature changes of all pixels in the unit area. The two are fused in the rule-driven framework with the first coefficient and the second coefficient as weights. The multi-feature correlation model is updated regularly, and the updated result is used to generate the initial warning value. The correlation weight is derived from the prior analysis and the system operation feedback mechanism and is stored in the adjustable structure cache for subsequent reference.

[0007] In a preferred embodiment, after receiving the initial warning value, the recognition behavior response module increases the sampling frequency of the image, thermal imaging, and gas concentration information from the initial frequency to the first frequency value, and activates the data focus acquisition mechanism of the target area. The mechanism performs regional focus processing during the image acquisition process to enhance the local image resolution, and at the same time sets the duration of the response behavior to the first time period value; after the time limit expires, the sampling frequency and acquisition strategy are restored to the initial state, and all parameter changes of the sampling strategy are controlled by the recognition behavior response function through the task instruction stack, and the execution time label and strategy priority are added to the instruction stack.

[0008] In a preferred embodiment, the robot position data in the trajectory recording behavior is obtained by fusing the feature point matching results between the visual image frames and the inertial measurement acceleration value. The specific calculation process is as follows: Between two adjacent frames of image, a first preset number of feature points are extracted using a scale-invariant feature transformation, the feature points are paired using a matching algorithm, and the pixel coordinate difference vector between each pair of matching points is calculated. The difference vector is converted into a three-dimensional relative displacement vector using the camera intrinsic parameter matrix; Obtaining an acceleration signal from an inertial measurement device, integrating it to obtain a velocity vector, and integrating it again to obtain a second relative displacement vector; The two sets of relative displacement vectors are linearly weighted using the matching confidence value and the preset fusion coefficient as weights. The matching confidence value is the inverse of the ratio of the average Euclidean distance of all paired points in the image matching to the maximum distance. The higher the confidence value, the higher the registration accuracy, and the range is between zero and one. The fusion coefficient is set to a fixed ratio based on the system deployment phase to balance image matching accuracy and inertial signal noise. The final fusion result uses the robot's geometric center as the reference coordinate system for position offset correction and combines it with the timestamp to form a complete trajectory sequence. This is used by the trajectory joint analysis module to construct a space-time evolution path model and track abnormal path dynamic changes.

[0009] In a preferred embodiment, the space-time evolution path model is constructed using a regional segmentation method. The target area is divided into multiple equal cells and marked as active cells and inactive cells. The active cells are connected in timestamp order to form an abnormal path chain. The abnormal trend is judged by calculating the following three parameters: First, the trend of path length change is calculated by calculating the total Euclidean length difference of the active unit coordinate set in the path chain in two consecutive time windows and recording the difference sequence. If the difference in three consecutive time windows is positive, it is judged that the path is growing; Second, the continuous density of activity units is defined as the number of consecutive adjacent activity unit pairs in each path chain divided by the total length of the path chain. If the density value is greater than the first preset density threshold, it is judged as concentrated distribution; Third, directional consistency: Calculate the unit direction of each vector in the path chain and the angles between adjacent vectors. If all angles are less than the first angle threshold, the path direction is considered stable. When at least two of the above three parameters meet the preset judgment conditions, the risk output control module is triggered to execute the high-level safety hazard identification process.

[0010] In a preferred embodiment, when generating a safety hazard assessment conclusion, the risk output control function is not only based on the enhanced collection information obtained in the identification behavior response function, but also on the spatial path change characteristics output in the trajectory joint analysis function, combined with the following three judgment factors: first, the total length of the abnormal path chain; second, the enhancement amplitude of the abnormal signal, that is, the difference between the enhanced collection information and the previous data of the same area; third, the average movement rate of the target area within the specified time.

[0011] In a preferred embodiment, the risk output control function automatically initiates an intervention execution mechanism after generating a safety hazard assessment conclusion, which includes sending a path adjustment command to the robot body scheduling control system and simultaneously generating a risk information package; The path adjustment command includes the current position retreat instruction and the next detection point coordinate positioning rule. The rule adjusts the angle based on the current abnormal path chain direction and locates the next target point in the vertical direction safe area. The generated risk information package includes abnormal signal type, location information, assessment level and time tag.

[0012] The technical effects and advantages of the present invention are as follows: The present invention uses a multi-source data acquisition module to simultaneously acquire image information, thermal imaging information, ambient gas concentration, and temperature and humidity change signals while in motion, and accurately marks the acquisition location and timestamp, so that the acquired data has synchronization and a spatial reference basis. The information fusion processing module normalizes different types of sensor data in a preset order, and uses a cross-feature extraction method to construct a modeling structure between various types of signals, thereby enhancing the response correlation between multidimensional data. This processing method not only improves the ability to identify subtle anomalies, but also realizes the dynamic extraction of potential abnormal features in continuously changing signals through a rule-driven method, forming an initial warning value with a behavioral logic basis, thereby providing a reliable basis for subsequent response and judgment processes.

[0013] After receiving the initial warning value, the identification behavior response module of the present invention decides whether to start the enhanced collection mechanism based on the threshold comparison result. When the starting conditions are met, the system will increase the sampling frequency of the corresponding information source and synchronously activate the trajectory recording behavior in the target area to achieve centralized monitoring of suspicious areas. At the same time, the newly added feature parameters generated during the response process will be used as enhanced results for subsequent analysis. This behavioral response process forms a closed-loop logic from warning judgment to target focusing and then to feature enhancement, which not only improves the data density in high-risk areas, but also avoids redundant consumption of resources in low-risk areas, significantly improving the response efficiency and recognition accuracy of the system.

[0014] The present invention accesses the real-time location data of the target area and its updated time series information through the trajectory joint analysis module, constructs a space-time evolution path model, and identifies possible trends such as continued abnormal growth, changes in mobile distribution, or aggregation and diffusion, thereby establishing a tracking mechanism for the evolution of abnormal behavior. The risk output control module further generates a safety hazard assessment conclusion based on the analysis results and the enhanced collection information obtained in the recognition behavior response, combined with the preset risk level classification conditions, so that the system has a complete operation link of "identification-response-tracking-assessment". This process helps to achieve dynamic positioning, hierarchical judgment and accurate output of risks, and significantly improves the ability to respond to hidden safety issues in complex inspection scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings; Figure 1 This is a schematic diagram of the robot real-time safety hazard identification system based on multimodal sensor fusion in the present invention. DETAILED DESCRIPTION

[0016] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0017] Reference Figure 1 The following examples were obtained: Example 1: The robot real-time safety hazard identification system based on multimodal sensor fusion includes a multi-source data acquisition module, an information fusion processing module, an identification behavior response module, a trajectory joint analysis module, and a risk output control module. Its composition and collaborative relationship are as follows: The multi-source data acquisition module provides the system with a variety of basic perception data sources, including image information, thermal imaging information, ambient gas concentrations, and temperature and humidity change signals. By fusing the raw data obtained from multiple different types of sensors, it can more comprehensively and three-dimensionally portray the current environmental status. During the acquisition process, the module not only obtains the data signal itself, but also synchronously marks the spatial position and timestamp corresponding to the data to ensure that all collected information has spatiotemporal attributes. This data acquisition method with location information and time tags in a mobile state provides a high-quality input foundation for subsequent anomaly identification and evolution modeling based on space-time features, effectively enhancing the system's responsiveness and tracking accuracy to changes in hidden dangers in dynamic scenes.

[0018] The information fusion processing module uniformly processes various sensory information from the multi-source data acquisition module. By normalizing it in a preset order, it makes sensory data of different dimensions and units comparable and fusible. This module uses cross-feature extraction to analyze the coupling relationships between various sensory signals in terms of numerical trends, change frequencies, and temporal evolution, establishing a correlation modeling structure between signals to uncover potential abnormal behaviors hidden beneath surface changes. Using a rule-driven approach, the module can identify change segments that do not conform to normal patterns from continuously changing signals, extract them as abnormal change features, and further combine behavioral association logic to perform causal judgments on these abnormal features, thereby generating initial warning values. This initial warning value, as a preliminary quantitative description of the abnormal event, is the key basis for the entire system to advance to the response and decision-making stages.

[0019] The identification behavior response module receives and judges the initial warning value generated by the information fusion processing module in real time, and decides whether to activate the subsequent enhanced collection process by matching it with the threshold comparison rules set in the system. When the initial warning value meets the preset abnormal judgment conditions, the module immediately increases the sampling frequency of the relevant information source to obtain higher density and higher resolution data, ensuring clearer and more accurate observation of the details of the potential risk area. At the same time, the module also synchronously activates the trajectory recording behavior of the target area, and uses the new feature parameters generated in the sampling response process as the enhancement result to provide support for subsequent spatiotemporal evolution analysis. The identification behavior response module plays a transitional role in the system from static identification to dynamic tracking, enabling the system to have the ability to conduct in-depth observation and continuous monitoring of the target area.

[0020] The joint trajectory analysis module identifies the target area's trajectory records activated by the behavioral response module, acquires its real-time location data, updates after the response, and its evolution over time. This data is then modeled to construct a spatial-temporal evolution path model. This model reflects the dynamic expansion, migration, and aggregation of abnormal areas in both time and space. The target area is determined by the information fusion processing module based on the spatial location of the abnormal signal during the initial warning generation phase. This module then maps the target area to a spatial grid structure, forming a series of basic units for determining abnormal activity, known as activity units. By comprehensively analyzing the temporal evolution patterns of these units, the joint trajectory analysis module can identify typical abnormal evolution patterns, such as sustained growth, changes in movement distribution, or clustering and diffusion, providing spatial dynamic support for the system's risk level assessment.

[0021] The risk output control module receives and integrates the path analysis results output by the trajectory joint analysis module and the enhanced collection information formed in the identification behavior response module, and conducts a comprehensive assessment of the overall impact of the current abnormal event based on the risk level classification conditions pre-set in the system. This module forms a risk identification judgment model by comparing the data features in the enhanced results with the spatial evolution trend of the abnormal path, and generates corresponding safety hazard assessment conclusions based on this. The risk output control module not only bears the responsibility of the final classification and quantification of abnormal events within the system, but also serves as the trigger point for subsequent actions such as executing intervention strategies, pushing alarm information, and triggering path adjustment commands. Its output results have a direct impact on the adjustment of the entire system operation strategy, and are the key terminal link in realizing the "identification-judgment-response" closed-loop logic.

[0022] Image information refers to a sequence of image frames captured by the robot's onboard visible light camera. This image sequence is continuously captured at fixed intervals, forming a continuous frame sequence. During this process, the system uses inter-frame difference analysis to identify dynamically changing areas in the image, which may represent safety risks such as changes in the device's appearance, the intrusion of foreign objects, and the proximity of people. The inter-frame differences are calculated using background subtraction, which involves comparing the current frame with a static background model at the pixel level to identify locations where significant changes in pixel values ​​have occurred. These locations are considered to be areas of image target change.

[0023] Thermal imaging information refers to infrared images captured by a thermal infrared sensor, in which the grayscale value of each pixel corresponds to a temperature value. To enhance the accuracy of identifying areas where an image target has changed, the pixel temperature values ​​of the corresponding area in the thermal image are extracted, and the temperature gradient within that area is calculated. The gradient of pixel values ​​refers to the spatial rate of change of the temperature difference between adjacent pixels and can be used to determine whether an area has abnormal temperature rise, hotspots, or other conditions. The image target change area and the thermal imaging area are spatially aligned and matched to form a joint feature. This combines and analyzes the image changes identified through background subtraction with the thermal changes calculated through gradient calculation. This joint feature is used to further locate the image change area, thereby enhancing the robustness of abnormal target recognition.

[0024] The ambient gas concentration signal refers to the concentration value of combustible gas, toxic gas, or other abnormal industrial gas collected by the gas detection device installed on the robot body. The temperature and humidity change signal refers to the air temperature and relative humidity values ​​in the environment, which are collected in real time by the integrated temperature and humidity sensor. The collection method of these two types of signals is to collect them at a fixed time interval, and the time interval is set to a preset value during the deployment phase. For example, it can be set to collect once every 5 seconds. After each collection is completed, the system will use the current time point as a time stamp and bind it to the current position of the robot body at the time of collection.

[0025] The current position is determined by a fusion of inertial and visual methods. The inertial method uses acceleration and angular velocity information measured by the gyroscope and accelerometer to estimate displacement over a short period of time through integration. The visual method uses feature point matching between consecutive image frames to calculate image displacement and, combined with camera parameters, to determine relative position changes. The resulting positioning result from this fusion of the two methods surpasses the accuracy of pose estimation from a single source. The time stamp and position data recorded in the acquisition results together constitute the spatiotemporal labels of the sampled data, providing spatiotemporal support for subsequent path modeling and dynamic analysis.

[0026] Before all sensor data is fed into the information fusion processing module, it must first undergo numerical normalization. Sensor data, including the number of pixels in image regions with varying dimensions and numerical ranges, includes thermal imaging temperature differences, gas concentrations, temperature, and humidity. These data vary in dimension and numerical range, making joint analysis impossible. Therefore, a unified normalization strategy is employed. This normalization process involves extracting the minimum observed value (defined as the first extreme value) and the maximum observed value (defined as the second extreme value) for each sensor signal type within a given time period (e.g., the past 24 hours) from historical operation records. The currently acquired value is then normalized using the formula (current value minus the first extreme value) divided by (second extreme value minus the first extreme value). The result is a dimensionless value between zero and one, representing the current value's position within the historical range. This is used to eliminate magnitude variations between signals due to physical units and sensor types. The normalized result serves as a unified feature vector for subsequent modeling.

[0027] After completing numerical normalization, the information fusion processing module constructs a correlation modeling structure between signals through cross-feature extraction. This cross-feature extraction approach does not analyze the changing trends of each sensor signal individually, but instead simultaneously focuses on the interactions between different signal types. For example, whether the frequency of regional changes in the image information and the degree of temperature rise in that area in the thermal image are synchronized can be used as a joint feature to indicate potential anomalies.

[0028] Specifically, the cross-feature extraction method is implemented by calculating the linear correlation between the image change frequency and the average temperature rise value of thermal imaging. The image change frequency is defined as the average number of times a change region is identified in the same spatial area per unit time, for example, the frequency of identifying moving objects or structural changes in a video sampled per second. The average temperature rise value of thermal imaging is defined as the average temperature rise of all pixels in a unit area over a certain period of time. It is calculated by recording the temperature change of each pixel in the area and calculating the total temperature rise divided by the number of pixels.

[0029] The image change frequency and average temperature rise are constructed into a multi-feature correlation model in the information fusion processing module. This model is constructed using a linear weighting approach: the image change frequency is multiplied by a first coefficient, and the thermal image average temperature rise is multiplied by a second coefficient. The two coefficients are summed to form a comprehensive assessment factor. The first and second coefficients are weighting parameters set during the system initialization phase. They reflect the relative importance of the two features in anomaly detection based on empirical rules or data-driven optimization results. For example, if image changes typically occur before temperature rise in certain types of hidden dangers, the first coefficient can be set higher than the second coefficient to enhance the influence of the image signal. This multi-feature correlation model is not constructed once and fixed; instead, it is updated at regular intervals, known as timed updates. For example, every ten minutes, the correlation between the image change frequency and thermal image temperature rise is recalculated based on the latest sensor data, and the weighting coefficients are recalculated and revised. The updated model is used to generate initial warning values. In a rule-driven framework, the threshold value of the comprehensive factor is used to determine whether an event constitutes an anomaly. A rule-driven approach uses preset numerical rules, such as when the comprehensive factor exceeds a certain warning threshold, to generate an initial anomaly warning value.

[0030] The weights, model update parameters, and execution logic used in the above process are derived from a priori analysis and the system's operational feedback mechanism. A priori analysis refers to offline model training or statistical pattern extraction based on historical inspection data during the initial system deployment. The system's operational feedback mechanism automatically adjusts the original model parameters based on the accuracy and response of the recognition results during actual operation. All generated weight information and model structure are stored in an adjustable structure cache, which supports fast parameter reading and writing and online updates, facilitating subsequent process calls or dynamic adjustments.

[0031] The initial value for anomaly warnings is not an input, but rather an output variable generated by the information fusion processing module based on the characteristics of the perceived data. Its purpose is to trigger the judgment process of the downstream "Identification and Behavior Response Module" and is the first data with logical triggering significance in the anomaly identification process. The initial value for anomaly warnings is a quantitative basis for determining whether the environmental state at the current moment and specific spatial location possesses the characteristics of a potential safety hazard. This initial value is not directly used in the final determination of an anomaly; rather, it serves as an input for the "Identification and Behavior Response Module" to determine whether to perform enhanced data collection and trajectory recording, acting as a pre-warning signal for triggering an alarm.

[0032] The generation of the initial value of abnormal warning consists of the following three parts: Feature fusion factor: It is composed of the weighted sum of the image change frequency and the average temperature rise value of thermal imaging in a multi-feature association model. This factor represents the degree of joint abnormality of visual and thermal signals in the current area.

[0033] Rule-driven judgment threshold: The system sets an early warning trigger threshold. For example, if the comprehensive factor is greater than a certain value, it is considered to have an abnormal tendency. This value is dynamically updated by the prior analysis and operation feedback mechanism.

[0034] Time tag and spatial location: Combined with the timestamp and current location coordinates provided by the data acquisition module, it ensures that each initial warning value has unique positioning in time and space, supporting subsequent behavioral response and trajectory analysis.

[0035] For example, while a robot was inspecting a specific area, the image data showed high-frequency structural changes between consecutive frames (the image change frequency was 0.8). Simultaneously, thermal imaging data indicated a 2.5°C temperature rise in the area over the previous 30 seconds (with an average temperature rise of 0.75). The multi-feature association model weighted these two factors by a first coefficient of 0.6 and a second coefficient of 0.4, generating a fusion factor of (0.8 × 0.6) + (0.75 × 0.4) = 0.48 + 0.30 = 0.78. Given the system's pre-set warning threshold of 0.70, the initial anomaly warning was established. The data was also tagged with the time stamp "2025-05-28 14:21:36" and spatial coordinates (X = 12.6 m, Y = 3.2 m), which together formed the initial anomaly warning.

[0036] After receiving the initial warning value, the identification and action response module first determines whether the current initial warning value meets the triggering conditions according to the threshold comparison rules set in the system. This threshold comparison rule is set during the system design phase and refers to the numerical critical criteria predefined during system operation to determine whether to execute enhanced collection behavior. It is usually based on comparing the value of the fusion factor in the initial warning value with the threshold. If the initial warning value exceeds the threshold, it is considered a potential abnormal state and the next processing flow is entered.

[0037] If the comparison conditions are met, the recognition behavior response module will start the enhanced acquisition mechanism. The enhanced acquisition mechanism refers to the implementation of sampling frequency improvement and data refinement processing on the perception channels related to the target area indicated by the initial warning value to obtain higher density and higher resolution perception information. The "relevant information sources" here include image information, thermal imaging information, and gas concentration information, specifically referring to the three sensor channels provided by the multi-source data acquisition module: visible light image acquisition equipment, infrared thermal imaging sensor, and gas detection sensor. The recognition behavior response module increases the sampling frequency of the above three channels from the initial frequency set by the system to a first frequency value. The first frequency value is set during deployment based on the equipment processing capacity and on-site response timeliness, for example, it can be two to five times the original frequency.

[0038] While increasing execution frequency, the system also activates the data-focused acquisition mechanism for the target area. This target area is determined by the spatial location of the abnormal signal when the information fusion processing module generates the initial warning value. The recognition behavior response module uses this as a reference to perform regional focus processing during image acquisition. Regional focus processing involves controlling the image acquisition device to increase the frame rate, improve image resolution, or perform digital zoom within the warning area to improve image clarity and detail feature recognition within that area, thereby facilitating subsequent modeling and path tracking.

[0039] The entire enhanced data collection process has a time limit. The identification behavior response module sets the duration of the response behavior to a first time period. This means that the collection strategy will not continue indefinitely, but will automatically exit the enhanced mode after a preset time period. The first time period is set by the system operating environment and can be, for example, 60 seconds or 300 seconds. The specific value is stored in the system parameter configuration table.

[0040] When the time limit expires, the identification behavior response module automatically restores the sampling frequency of images, thermal imaging, and gas concentration to the initial frequency, cancels the focusing mechanism, and returns to normal inspection status. All parameter changes of the sampling strategy (including increasing and restoring the sampling frequency and enabling and disabling the focusing mechanism) are controlled by the identification behavior response module through the task instruction stack. The task instruction stack refers to an operation sequence scheduling structure built into the system, in which each control instruction is attached with two elements when it is generated: an execution time tag and a policy priority. The execution time tag is used to accurately schedule the start and end times of the sampling behavior; the policy priority is used to rationally schedule sampling resources when multiple warning areas are triggered simultaneously, ensuring priority response to high-risk areas.

[0041] During this response, the system aggregates newly acquired feature data from sources like images, thermal imaging, and gas concentrations, and combines this data with the parameters of the sampling strategy used in the response to generate an "enhancement result." This enhancement result, output from the recognition response module, encompasses the current multi-dimensional perception enhancement of the target area and serves as a crucial input for the subsequent trajectory joint analysis module and risk output control module.

[0042] It should be noted that the temperature and humidity change signals do not participate in the activation of the enhanced acquisition mechanism at this time. Although the temperature and humidity change signals participate in information fusion and the formation of initial abnormality warning values ​​in the initial multi-source data acquisition, the sampling frequency of the temperature and humidity signals is not increased when the enhanced acquisition mechanism is activated in the recognition behavior response module. The specific reasons are as follows: Temperature and humidity change slowly over short periods of time. Their sensing characteristics determine that their signal change rate is much lower than that of images, thermal imaging, and gas concentrations. Even if there are safety hazards in the environment, temperature and humidity signals tend not to fluctuate dramatically within tens of seconds, so increasing the sampling frequency is of limited significance. Data bandwidth and processing priority limitations: Increasing the sampling frequency of image and thermal imaging data will significantly increase the system's computing load. In embedded platforms with limited processing resources, high-frequency, time-sensitive data channels should be prioritized. Temperature and humidity data are usually used in information fusion processing as background environmental factors, rather than as the main cause of event triggering. Therefore, they do not need to participate in dynamic response control, but are always collected periodically. Focusing on three data sources, namely images, thermal imaging, and gas concentration, as relevant information sources for response optimization in the recognition behavior response module is a design choice that has been technically optimized and reasonably avoided.

[0043] The robot's position data during trajectory recording is derived by fusing feature point matching results between visual image frames with inertial acceleration measurements. This fusion method improves the accuracy and robustness of position information and serves as the fundamental data input for the joint trajectory analysis module to construct a spatial-temporal evolution path model.

[0044] The image feature point matching acquisition process is as follows: Between two adjacent frames (i.e., image data collected continuously by the robot at two time points), a first preset number of feature points are extracted using a scale-invariant feature transformation (SIT). SIT refers to a feature extraction technique that can stably extract local key points in an image despite changes in image size, rotation, or illumination. For example, algorithms such as SIFT (Scale Invariant Feature Transform) or ORB can be used to ensure robust feature extraction even in complex inspection environments. The first preset number can be set based on image resolution and processing power, for example, 200 highly responsive key points can be extracted.

[0045] The feature point sets of the two images are paired using a matching algorithm. The matching algorithm identifies the optimal pairing based on a similarity metric of the feature descriptors, such as the Euclidean distance or Hamming distance. The pixel coordinate differences between each pair of matching points form a two-dimensional vector, representing the relative displacement of the feature point on the image plane. This difference vector is converted into a three-dimensional relative displacement vector using the camera's intrinsic parameter matrix. The camera's intrinsic parameter matrix is ​​a matrix containing internal parameters of the camera device, such as focal length, principal point position, and pixel size. It is used to map pixel coordinates into three-dimensional displacement values ​​in real-world spatial units (e.g., meters), thereby obtaining a first set of three-dimensional relative displacement vectors.

[0046] The data processing of the inertial measurement part is as follows: the acceleration signal is obtained from the inertial measurement device. The acceleration signal comes from the accelerometer inside the robot and provides linear acceleration values ​​on the three spatial axes. This acceleration value is integrated once to obtain the velocity vector per unit time, and then integrated a second time to obtain the second relative displacement vector of the robot within the time period. This vector can represent the rough motion path of the robot in space without an external reference, but due to the cumulative error of the inertial system, the accuracy needs to be corrected with the assistance of visual matching results. The process of displacement vector fusion is as follows: The two sets of relative displacement vectors (visual and inertial calculations) are linearly weighted and fused using the matching confidence value and a preset fusion coefficient as weights, respectively. The matching confidence value is defined as the inverse of the ratio of the average Euclidean distance of all paired points in the image match to the maximum distance. A higher confidence value indicates better image matching quality, meaning more concentrated paired points and less deviation, indicating more reliable displacement estimates. Its value ranges from zero to one. For example, if the average Euclidean distance is 5 pixels and the maximum distance is 20 pixels, the confidence value is 0.25.

[0047] The preset fusion coefficient is a fixed ratio parameter set during system deployment. It is used to balance the trustworthiness of visual and inertial data based on the application scenario. For example, in scenes with strong vibration or visual obstruction, the inertial data weight can be appropriately increased; in scenes with clear images and large inertial drift, the visual data weight can be increased.

[0048] The final fused displacement results are corrected for positional offset using the robot's geometric center as the reference coordinate system. The geometric center is a predefined central reference point in the robot's structural design, typically the center of the structure or the center of the sensor's axis of symmetry. It serves to unify the reference frames of various data and reduce cumulative errors.

[0049] The fused location data is then combined with timestamps to form trajectory points. This process continues, resulting in a complete trajectory data sequence. This sequence, including location coordinates, time tags, and computational accuracy weights, serves as the core data source for the subsequent trajectory joint analysis module to construct a spatial-temporal evolution path model. This model is used to identify dynamic changes in abnormal path behavior, including sustained growth, shifts in movement distribution, or clustered diffusion patterns.

[0050] After receiving the target area location data, enhanced acquisition information, and timestamps from the recognition and response module, the joint trajectory analysis module begins constructing a spatial-temporal evolution path model. This model is constructed using a regional segmentation approach, specifically dividing the warning target area into multiple uniformly sized grid cells on a two-dimensional plane. These cells are typically rectangular or square in shape, and their coverage area is equal to the warning target area boundary.

[0051] The above-mentioned cell construction is used to map continuous trajectory points into discrete grid units in space, thereby enabling the construction of path chain structures and analysis of evolutionary trends. As trajectory data sequences are continuously input, the system determines the cell number corresponding to the spatial coordinates of each time point based on the robot's actual movement path and timestamp records. It then divides the grid into "active cells" and "inactive cells" based on the number and density of trajectory points.

[0052] An "active cell" is a cell that has a trajectory point hit within a selected time period, indicating that the robot performed sampling or an event occurred at that location. An "inactive cell" is a cell that was not hit. The system connects active cells corresponding to consecutive timestamps in chronological order to form an "abnormal path chain," a spatially discrete representation of the evolutionary path of potential risks.

[0053] Abnormal trend identification is calculated and analyzed based on the following three parameters: The first item is path length trend. This trend is used to determine whether the abnormal path chain is expanding over time. The specific operation is as follows: divide the analysis time into multiple consecutive time windows, for example, 5-minute windows. Within each time window, extract the coordinates of all active units in the path chain within that time period. Calculate the total Euclidean length of these coordinate points connected in chronological order, that is, the sum of the distances between each two points. Calculate the length sequence for multiple time windows and subtract the lengths within two consecutive time windows. If the length differences within three consecutive time windows are all positive (i.e., the path length is continuously increasing), the system determines that the path is growing. This parameter reflects the spatial spread of the abnormal trajectory and is applicable to risk models such as the gradual expansion of high-temperature zones and gas leak sources.

[0054] The second item: the continuous density of active units. This parameter is used to identify whether the trajectory path shows a trend of concentration in spatial distribution. The specific definition is: in each path chain, identify the number of all "continuous adjacent active unit pairs", that is, the number of cell pairs that are continuously hit in space; divide this number by the total length of the path chain to obtain the density of continuous hits within the unit length; compare this density value with the first preset density threshold. If it is higher than the threshold, it is judged to be concentrated in distribution. A higher continuous density means that the abnormal activity is not scattered, but is focused on a specific area, which may be a "hotspot" area of ​​security risks, such as a corner of a wall where image anomalies appear multiple times in a row.

[0055] The third item, directional consistency, is used to assess whether an abnormal path chain has a stable movement trend, that is, to determine whether it continues to move in a certain main direction. The specific operation is as follows: the displacement vector formed by each continuous unit in the path chain is extracted; each segment vector is converted into a unit vector, indicating that the direction is not affected by the displacement length; the angle between each pair of adjacent unit vectors is calculated one by one; if all angles are less than a first angle threshold, such as 30 degrees, the path direction is considered stable. This indicator can identify typical phenomena such as gas leaks extending in a single direction along ventilation ducts and cable overheating spreading along the wiring route.

[0056] When any two or more of the three trend parameters meet the system's pre-defined criteria, the target area is deemed to have a structural risk evolution trend. The system automatically triggers the risk output control module to enter the high-level safety hazard identification process, implementing further intervention mechanisms and outputting results. Suppose a robot inspects a channel and observes abnormal path lengths of 8 meters, 10 meters, and 12.5 meters in three consecutive time windows, with difference sequences of +2 meters and +2.5 meters. Eight of the ten active units in the path chain are consecutively adjacent, with a density of 0.8, exceeding the system's pre-defined density threshold of 0.6. The angles between all path segments are less than 20 degrees, and the direction is stable. The system makes a decision based on the "meeting any two of the three criteria" criterion, triggering a high-level safety hazard assessment and transmitting the result to the risk output control module.

[0057] When generating safety hazard assessment conclusions, the risk output control function relies on input data that includes not only the enhanced information collected by the behavioral response recognition function but also the spatial path change characteristics output by the trajectory joint analysis function. This allows for a quantitative and graded assessment of the current safety risk, thereby determining whether to implement further intervention instructions. This assessment mechanism integrates three clear judgment elements, each of which is generated by a structured processing chain in previous modules.

[0058] The first type of data: Enhanced Collection Information. This refers to the image, thermal, and gas concentration information collected within the target area at an increased sampling frequency by the Recognition Behavior Response Module after the enhanced collection mechanism is activated. During the collection process, this module performs regional focus processing and policy adjustments. The resulting new data is assigned time tags and policy context, representing a locally refined and enhanced version of the original inspection data. This information serves as an input to risk assessment, providing a high-precision observational data foundation for the current state of the target area.

[0059] The second type of data: spatial path change characteristics. These characteristics, derived from the trajectory joint analysis module, represent the modeling results of spatial activity trends within the target area. They include the geometric structure, temporal evolution characteristics, directional characteristics, and density distribution of abnormal path chains. These characteristics are formed through the dynamic evolution of "activity units" within the path chain and serve as the core basis for the system to identify abnormal development trends. Based on this, the risk output control function uses the following three criteria as inputs for generating assessment conclusions: Criterion 1: The total length of the abnormal path chain. An abnormal path chain is a spatial structure formed by connecting activity units in timestamp order, representing the robot's abnormal behavior path within the target area. Its "total length" is defined as the sum of the Euclidean distances between all adjacent activity units in the path, expressed in meters or relative spatial grids. The longer the path, the greater the potential range of the abnormal activity and the higher the risk level. This parameter, output by the trajectory joint analysis module, is directly calculable and verifiable.

[0060] The second judgment factor is the magnitude of the abnormal signal's enhancement. An abnormal signal refers to data items in the enhanced acquisition information that significantly deviate from normal conditions, as determined by calculating the difference between the data and historically recorded standard data for the same target area. The magnitude of this deviation is expressed as the magnitude of this deviation. For example, for image changes, the magnitude may represent the percentage increase in the area of ​​the changed region; for thermal imaging, the magnitude may represent the difference in average temperature rise above the historical mean; for gas concentration, the magnitude may represent the difference between the current value and a stable background concentration over a period of time. This difference is standardized and quantified in the system, and compared against a preset magnitude threshold to determine whether the signal is a high-intensity anomaly.

[0061] It's worth noting that while an "abnormal signal" here isn't directly equivalent to an "active unit," its spatial location mapping inevitably falls within the aforementioned "target region" grid structure. In other words, the spatial projection of an abnormal signal determines its contribution to the path chain, which is the spatial connection between the evolutionary behaviors of multiple abnormal signals.

[0062] Determination factor three: The average movement rate of the target area within a specified time period. This is defined as the Euclidean distance between the center points of the two active units at the beginning and end of the abnormal path chain divided by the time difference between these two points, expressed in meters per second or grids per second. This parameter reflects whether the abnormal signal exhibits a continuous motion trend and can be used to identify dynamic hazards (such as gas leaks moving with airflow or cable overload areas expanding). The system can set a rate threshold. If the abnormal signal movement rate within a certain area consistently exceeds this threshold, the risk is considered to be rapidly evolving, and the assessment level should be increased.

[0063] The risk output control function generates a corresponding safety hazard assessment conclusion based on the combined results of the three aforementioned judgment factors, combined with the system's established grading rules. Assessment levels are typically categorized into multiple levels, such as general, moderate, and high-risk, corresponding to different intervention strategies. After generating the assessment conclusion, the system determines whether to invoke subsequent intervention mechanisms, such as path adjustment, risk broadcasting, and monitoring alerts. These are fully described in the subsequent dependent claims.

[0064] Assume that the total length of an abnormal path chain in a certain area is 12 meters, with an average density of 0.7 and a continuous extension time of 15 minutes; the thermal imaging enhancement amplitude is 1.6 times the standard value, and the gas concentration is enhanced to 30 ppm (20 ppm higher than the background average); the distance between the start and end of the abnormal path is 6 meters, and the time span is 120 seconds, resulting in an average movement rate of 0.05 meters per second. After comparing these three indicators, the system triggers a high risk level, generates an alarm, and activates the intervention mechanism.

[0065] Once the risk output control function completes the safety hazard assessment, it automatically initiates the intervention execution mechanism. This mechanism is a comprehensive response process for on-site execution and information reporting. Its goal is to quickly intervene in the dynamic scheduling of inspection tasks based on risk identification results, mitigate potential safety risks, and ensure the integrity of abnormal event recording and transmission.

[0066] Sending a path adjustment command to the robot's dispatch control system: The path adjustment command is the first action generated by the risk output control module. It directly targets the robot's dispatch control system and redefines the current task path to prevent the robot from approaching or remaining in risky areas. A path adjustment command contains two components: Current Position Retreat Command: This instructs the robot to immediately cease inspections near the current trajectory point and rapidly move a predetermined distance in a predefined safe direction, evacuating the current target area. This direction can be in the opposite direction of the path or calculated from a predefined emergency retreat vector. It is typically 180 degrees opposite the direction of the current abnormal path chain to ensure maximum distance from the risk source. Next Inspection Point Coordinate Positioning Rule: After completing the retreat action, the system replans the next inspection point in the robot's patrol path. This positioning rule adjusts the angle based on the current abnormal path chain, aiming to ensure the next target point falls within a vertically safe zone. "Vertical" here refers to a direction at a 90-degree angle to the main direction of the current path chain to avoid the risk's spread. After the system calculates the main direction through the path chain vector, it rotates to generate the vertical direction. Combined with the position marked as inactive unit in the spatial grid and far away from the abnormal point set, the coordinates of the next target point can be generated.

[0067] Generate a risk information package: After the path adjustment command is issued, the system will simultaneously generate a risk information package to record the core characteristics of the current risk event and submit it to the monitoring platform or storage module. The information package consists of the following fields: Abnormal Signal Type: This identifies the type of signal source that triggered the current risk assessment. This abnormal signal type is derived from the identification field added by the information fusion processing module during the generation of the initial warning value. Specific values ​​include, but are not limited to: abnormal image change: triggered by a sudden increase in the area or excessive frequency of changes between image frames; abnormal thermal imaging: triggered by the average temperature rise of thermal image pixels within the target area exceeding a preset range; abnormal gas concentration: triggered by an abnormal increase or sudden change in the concentration of one or more gas types within the target area compared to the background value. If multiple signals are triggered simultaneously, a combined identifier (e.g., "image + gas") may be used in the information packet to represent this.

[0068] Position information: The spatial coordinates of the latest trajectory point of the current abnormal path chain, determined with reference to the geometric center coordinate system of the robot body. The accuracy is supported by the trajectory recording behavior of image and inertial fusion.

[0069] Assessment Level: This is a rating generated by the risk output control module based on the combined results of three criteria (total length of the abnormal path chain, enhancement amplitude, and average movement rate). It can be divided into three levels: "Low," "Medium," and "High," and corresponds to different system response strategies. Time Tag: This is an absolute timestamp generated for the information packet, consistent with the enhanced collection information and trajectory point data, and is used for system-wide log correlation and event recurrence tracking.

[0070] During one inspection, the robot entered a ventilation duct intersection. The recognition and behavioral response module triggered enhanced image and gas concentration acquisition. The image change frequency increased from once per second to five times per second, the gas concentration instantly rose by 40 ppm, the path length reached 15 meters, and the directional stability assessment passed, with the enhancement rate reaching 2.1 times the standard value. Based on this, the risk output control module generated a high-risk assessment and subsequently issued a retreat command, directing the robot to move 2 meters in the opposite direction of the abnormal path chain at a speed of 0.5 meters per second and locate the next inspection point within a grid area 90 degrees to the left of the path direction. The system also generated a risk information packet with the type labeled "Image + Gas," the location coordinates (x=34.2, y=17.6), the level "High," and the time stamp "2025-05-28 14:13:12." This information was transmitted by the communication system to the upper-level monitoring platform, triggering subsequent remote intervention commands or initiating automated lockdown operations.

[0071] The above model formulas are all dimensionless and numerically calculated. The model formula is a model formula obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the model formula are set by technicians in this field according to actual conditions.

[0072] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its module and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0073] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these modules are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described modules for each specific application, but such implementation should not be considered beyond the scope of this application.

[0074] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0075] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A robot real-time safety hazard identification system based on multimodal sensor fusion, characterized by: It includes a multi-source data acquisition module, an information fusion processing module, an identification behavior response module, a trajectory joint analysis module, and a risk output control module. Its composition and collaborative relationship are as follows: The multi-source data acquisition module collects image information, thermal imaging information, ambient gas concentration, and temperature and humidity change signals respectively, and synchronously marks the acquisition position and timestamp while in motion; The information fusion processing module normalizes the values ​​of various sensor information according to a preset sequence, constructs a correlation modeling structure between signals through cross-feature extraction, uses a rule-driven approach to identify potential abnormal change characteristics in continuous signals, and forms an initial warning value based on behavioral association logic; The recognition behavior response module receives the initial warning value and determines whether to activate the enhanced collection mechanism according to the threshold comparison rule. If the conditions are met, it will correspondingly enhance the sampling frequency of the relevant information source and synchronously activate the trajectory recording behavior in the target area. The newly generated feature parameters in the response process are used as the enhanced results. The target area is determined by the information fusion processing module based on the spatial location of the abnormal signal when generating the initial warning value, and is used as the starting position for triggering trajectory recording in the recognition behavior response module. This area is mapped to the spatial division grid in the trajectory joint analysis module, and the corresponding grid unit is used as the basis for identifying the active unit. The trajectory joint analysis module accesses the real-time location data of the target area, responds to the updated information and its temporal changes after collection, and constructs a space-time evolution path model to identify the continuous growth of anomalies, changes in movement distribution, or aggregation and diffusion patterns, in order to track the dynamic development process of anomalies. The risk output control module generates corresponding safety hazard assessment conclusions based on the trajectory joint analysis results and the enhanced collection information obtained from the recognition behavior response, combined with the preset risk level classification conditions.

2. The robot real-time safety hazard identification system based on multimodal sensor fusion according to claim 1 is characterized in that: Image information is collected using a continuous frame sequence. The inter-frame difference is used to identify image target changes by background subtraction. Combined with the gradient change of pixel values ​​in thermal imaging, a joint feature is formed to locate the image change area. The collection of environmental gas concentration and temperature and humidity change signals is carried out at fixed time intervals. Each collection result is attached with a time stamp and the current position of the robot. The current position is output by inertial and visual fusion. All perception data are numerically normalized before entering the information fusion processing module.

3. The robot real-time safety hazard identification system based on multimodal sensor fusion according to claim 2 is characterized in that: The cross-feature extraction method in the information fusion processing module constructs a multi-feature correlation model by calculating the linear correlation between the image change frequency and the average temperature rise value of thermal imaging; The image change frequency is the average number of times the changed area appears in a unit time, and the average temperature rise value is the average value of the temperature changes of all pixels in the unit area. The two are fused in the rule-driven framework with the first coefficient and the second coefficient as weights. The multi-feature association model is updated regularly, and the updated results are used to generate the initial warning value.

4. The robot real-time safety hazard identification system based on multimodal sensor fusion according to claim 3 is characterized in that: After receiving the initial warning value, the recognition behavior response module increases the sampling frequency of image, thermal imaging, and gas concentration information from the initial frequency to the first frequency value, and activates the data focus acquisition mechanism of the target area. This mechanism performs regional focus processing during the image acquisition process to enhance the local image resolution, and at the same time sets the duration of the response behavior to the first time period value; after the time limit expires, the sampling frequency and acquisition strategy return to the initial state.

5. The robot real-time safety hazard identification system based on multimodal sensor fusion according to claim 4 is characterized in that: The robot position data in the trajectory recording behavior is obtained by fusing the feature point matching results between visual image frames and the inertial measurement acceleration value. The specific calculation process is as follows: Between two adjacent frames of image, a first preset number of feature points are extracted using a scale-invariant feature transformation, the feature points are paired using a matching algorithm, and the pixel coordinate difference vector between each pair of matching points is calculated. The difference vector is converted into a three-dimensional relative displacement vector using the camera intrinsic parameter matrix; Obtaining an acceleration signal from an inertial measurement device, integrating it to obtain a velocity vector, and integrating it again to obtain a second relative displacement vector; The two sets of relative displacement vectors are linearly weighted using the matching confidence value and the preset fusion coefficient as weights. The matching confidence value is the inverse of the ratio of the average Euclidean distance of all paired points in the image matching to the maximum distance. The fusion coefficient is set to a fixed ratio according to the system deployment stage to balance the image matching accuracy and inertial signal noise. The final fusion result uses the robot's geometric center as the reference coordinate system for position offset correction and combines it with the timestamp to form a complete trajectory sequence.

6. The robot real-time safety hazard identification system based on multimodal sensor fusion according to claim 5 is characterized in that: The spatial-temporal evolution path model is constructed using a regional segmentation method. The target area is divided into multiple equal cells, which are marked as active cells and inactive cells. The active cells are connected in timestamp order to form an abnormal path chain. The abnormal trend is judged by calculating the following three parameters: First, the trend of path length change is calculated by calculating the total Euclidean length difference of the active unit coordinate set in the path chain in two consecutive time windows and recording the difference sequence. If the difference in three consecutive time windows is positive, it is judged that the path is growing; Second, the continuous density of activity units is defined as the number of consecutive adjacent activity unit pairs in each path chain divided by the total length of the path chain. If the density value is greater than the first preset density threshold, it is judged as concentrated distribution; Third, directional consistency: Calculate the unit direction of each vector in the path chain and the angles between adjacent vectors. If all angles are less than the first angle threshold, the path direction is considered stable. When at least two of the above three parameters meet the preset judgment conditions, the risk output control module is triggered to execute the high-level safety hazard identification process.

7. The robot real-time safety hazard identification system based on multimodal sensor fusion according to claim 6 is characterized in that: When generating safety hazard assessment conclusions, the risk output control function is based not only on the enhanced collected information obtained from the identification behavior response function, but also on the spatial path change characteristics output from the trajectory joint analysis function, combined with the following three judgment factors: first, the total length of the abnormal path chain; Second, the enhancement amplitude of the abnormal signal, that is, the difference between the enhanced collected information and the previous data of the same area; third, the average movement rate of the target area within a specified time.

8. The robot real-time safety hazard identification system based on multimodal sensor fusion according to claim 7 is characterized in that: After generating a safety hazard assessment conclusion, the risk output control function automatically initiates an intervention execution mechanism, which includes sending a path adjustment command to the robot's dispatch control system and generating a risk information package at the same time. The path adjustment command includes the current position retreat instruction and the next detection point coordinate positioning rule. The rule adjusts the angle based on the current abnormal path chain direction and locates the next target point in the vertical direction safe area. The generated risk information package includes abnormal signal type, location information, assessment level and time tag.

Images