Infrastructure operation scene personnel safety early warning method and system based on multi-modal fusion

Abstract

The invention discloses an infrastructure operation scene personnel safety early warning method and system based on multi-modal fusion, and relates to the field of safety early warning, and the method comprises the steps: analyzing video data to obtain behavior action sequence features, analyzing positioning data to obtain space activity track features, and analyzing sound data to obtain environment event features; matching the behavior action sequence features, the space activity track features and the environment event features with an operation scene feature library, and determining the operation type, the operation area attribute and the operation environment state of the current scene; calling a target security rule set from a security specification library; according to the target security rule set, performing logic correlation analysis on the behavior action sequence features, the spatial activity track features and the environmental event features to obtain an anomaly analysis result; and generating a corresponding early warning signal based on the abnormal analysis result. By implementing the method, the accuracy of safety early warning of the personnel in the capital construction operation scene can be improved.

Description

Technical Field

[0001] This application relates to the field of safety early warning, and in particular to a method and system for personnel safety early warning in infrastructure operation scenarios based on multimodal fusion. Background Technology

[0002] With the continuous expansion of infrastructure construction and the increasing number of large-scale infrastructure projects, personnel safety management in these work environments has become a crucial issue. Infrastructure operations involve numerous high-risk work environments, such as working at heights, in confined spaces, and operating heavy equipment, posing a severe challenge to the safety of workers.

[0003] Currently, safety management in infrastructure construction sites mainly relies on single monitoring methods. Some construction sites use video surveillance systems to monitor the work area in real time; others use wearable positioning devices to track personnel location information; still others rely on noise sensors and other equipment to monitor environmental parameters. These monitoring methods operate independently, obtaining data information from a single dimension.

[0004] However, due to the complexity and variability of work scenarios, a single monitoring method is insufficient to comprehensively perceive and analyze the on-site situation. For example, relying solely on video surveillance can lead to blind spots due to obstructions from buildings or equipment, and simple location information cannot reflect specific personnel work behaviors. Furthermore, the lack of correlation analysis between environmental monitoring data and personnel activity status results in safety hazards not being detected and warned of in a timely manner, increasing the risk of safety accidents. Summary of the Invention

[0005] This application provides a method and system for personnel safety early warning in infrastructure operation scenarios based on multimodal fusion, which can improve the accuracy of personnel safety early warning in infrastructure operation scenarios.

[0006] Firstly, this application provides a method for personnel safety early warning in infrastructure operation scenarios based on multimodal fusion, applied to a safety early warning system. The method includes: collecting personnel video data, personnel positioning data, and sound data from the operation scenario; analyzing the video data to obtain behavioral action sequence features, analyzing the positioning data to obtain spatial activity trajectory features, and analyzing the sound data to obtain environmental event features; matching the behavioral action sequence features, spatial activity trajectory features, and environmental event features with an operation scenario feature library to determine the current operation type, operation area attributes, and operation environment status; retrieving a target safety rule set from a safety specification library based on the operation type, operation area attributes, and operation environment status; performing logical correlation analysis on the behavioral action sequence features, spatial activity trajectory features, and environmental event features according to the target safety rule set to determine whether there are abnormal behavioral intentions, abnormal spatial displacements, or abnormal environmental changes, obtaining anomaly analysis results; and generating corresponding early warning signals based on the anomaly analysis results.

[0007] In the above embodiments, based on multimodal data acquisition and fusion analysis, behavioral action sequence features, spatial activity trajectory features, and environmental event features are extracted from video data, positioning data, and audio data, respectively. These features are matched with a work scenario feature library to determine scenario attributes, thereby retrieving the target safety rule set for logical association analysis. By combining anomaly detection of behavioral intent, spatial displacement, and environmental changes, a comprehensive anomaly analysis result is formed and an early warning signal is generated, achieving accurate personnel safety early warning in infrastructure work scenarios.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the step of matching behavioral action sequence features, spatial activity trajectory features, and environmental event features with a work scenario feature library to determine the work type, work area attribute, and work environment state of the current scenario specifically includes: matching behavioral action sequence features with a preset action feature template to obtain a preliminary work type; comparing spatial activity trajectory features with preset area boundary information to obtain a preliminary area attribute; matching environmental event features with a preset environmental feature template to obtain a preliminary environment state; constructing a scenario feature vector based on the preliminary work type, preliminary area attribute, and preliminary environment state; inputting the scenario feature vector into the work scenario feature library for multi-dimensional matching to obtain a candidate scenario set; scoring each candidate scenario in the candidate scenario set to obtain a matching score; selecting the candidate scenario with the highest matching score; determining the work type corresponding to the candidate scenario as the work type of the current scenario; determining the area attribute corresponding to the candidate scenario as the work area attribute of the current scenario; and determining the environment state corresponding to the candidate scenario as the work environment state of the current scenario.

[0009] In the above embodiments, behavioral action sequence features, spatial activity trajectory features, and environmental event features are initially matched with preset templates to construct scene feature vectors for multi-dimensional matching to obtain a set of candidate scenes. By scoring the candidate scenes and selecting the scene with the highest score, the task type, regional attributes, and environmental state of the current scene are determined, establishing a precise mapping relationship from features to scenes and improving the accuracy of scene recognition.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the step of performing logical correlation analysis on behavioral action sequence features, spatial activity trajectory features, and environmental event features based on the target safety rule set to determine whether there is abnormal behavioral intent, abnormal spatial displacement, or abnormal environmental change, and obtaining anomaly analysis results, specifically includes: obtaining from the target safety rule set a standard action sequence template corresponding to the operation type, a safe activity area boundary corresponding to the operation area attribute, and a normal environmental event range corresponding to the operation environment state; comparing the behavioral action sequence features with the standard action sequence template in time sequence, and determining that there is abnormal behavioral intent when the action sequence does not conform to the action combination relationship or execution order in the standard template; comparing the spatial activity trajectory features with the safe activity area boundary in real time, and determining that there is abnormal spatial displacement when the trajectory point exceeds the safety boundary; matching the environmental event features with the normal environmental event range, and determining that there is abnormal environmental change when the type, intensity, or frequency of the environmental event exceeds a preset threshold range; and generating anomaly analysis results based on the determination results of abnormal behavioral intent, abnormal spatial displacement, and abnormal environmental change.

[0011] In the above embodiments, standard action sequence templates, safety activity area boundaries, and normal environmental event ranges are obtained from the target safety rule set and compared with real-time features. Anomalies are determined based on action combination relationships, execution order, spatial boundaries, and environmental parameter thresholds. The determination results of various anomalies are integrated to generate anomaly analysis results, achieving comprehensive and accurate anomaly state detection.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, the step of scoring each candidate scene in the candidate scene set specifically includes: analyzing the changing trends of behavioral action sequence features, spatial activity trajectory features, and environmental event features in the time dimension, identifying key time segments in each feature, where the key time segment represents the state transition process of the feature undergoing significant changes within a preset time period; retrieving historical scene data similar to the job type from the historical scene sample library, filtering the historical scene data based on the key time segments, and determining the influence weight of each feature on scene recognition; dynamically matching the key time segments with the corresponding features of the candidate scenes, calculating the state transition similarity of each time segment, and determining the weight coefficient of each time segment according to the importance of the state transition; multiplying the state transition similarity of each key time segment by the corresponding weight coefficient and summing them to obtain the matching score of the candidate scene.

[0013] In the above embodiments, the temporal variation trend of multimodal features is analyzed to identify key temporal segments representing the state transition process, and the influence weight of features on scene recognition is determined based on historical scene data. State transition similarity is calculated through dynamic matching and weight coefficients are determined. The weighted state transition similarities are then summed to obtain the candidate scene matching score, thus constructing a scene matching scoring mechanism based on temporal dynamic features, which improves the temporal accuracy of scene recognition.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of generating a corresponding early warning signal based on the anomaly analysis results, the method further includes: acquiring real-time detection data of multiple heterogeneous sensors at the construction site for the same monitoring target; identifying conflict states between the detection results of different sensors in the detection data, wherein the conflict state refers to the mutually exclusive state judgments given by different sensors for the same physical event at the same time; extracting the features of the conflict state and classifying them into predefined conflict modes, wherein the features include the duration of the conflict, the combination of sensors involved in the conflict, and the changing trend of the detection values ​​of each sensor; determining the key anomaly factors corresponding to the conflict state based on the characteristic attributes of the conflict mode and the detection mechanism of each sensor; and associating the key anomaly factors with the anomaly analysis results to obtain the credibility of the early warning signal.

[0015] In the above embodiments, real-time detection data of heterogeneous sensors on the same monitoring target are acquired, conflict states between different sensors are identified, and conflict features including duration, sensor combination, and changing trends are extracted and categorized. Based on the characteristic attributes of conflict modes and sensor detection mechanisms, key anomaly factors are identified, and their correlation with anomaly analysis results yields the credibility of the warning signal, establishing a quantitative evaluation system for the reliability of the warning signal.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, the association of key anomaly factors with anomaly analysis results to obtain the credibility of the warning signal specifically includes: determining the weight coefficient corresponding to the key anomaly factor, which is related to the duration of the conflict state and the number of sensors involved in the conflict; calculating the anomaly similarity between the key anomaly factor and the anomaly analysis results, which is determined based on the correspondence between the location of the anomaly and the degree of the anomaly; and calculating the credibility of the warning signal based on the weight coefficient and the anomaly similarity.

[0017] In the above embodiments, the weighting coefficients of the anomalous factors are determined based on the duration of the conflict and the number of sensors involved in the conflict, and the anomalous similarity is calculated based on the correspondence between the location of the anomalous occurrence and the degree of the anomalousness. The credibility of the warning signal is obtained through the comprehensive calculation of the weighting coefficients and the anomalous similarity, forming a quantitative mapping relationship from anomalous features to the credibility of the warning, and realizing an accurate assessment of the reliability of the warning signal.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of calculating the credibility of the warning signal based on the weighting coefficient and the anomaly similarity, the method further includes: determining the handling priority of the abnormal event based on the credibility and the type of the warning signal, and generating an emergency handling instruction based on the handling priority.

[0019] In the above embodiments, an abnormal event handling priority assessment matrix is ​​constructed based on the credibility and type of the early warning signal. Correction factors based on time urgency, impact scope, and cascading effects are applied to determine the handling priority. Emergency response instructions containing specific steps and resource allocation are generated according to the handling priority, establishing a complete response chain from early warning to handling, and achieving efficient allocation and timely response of abnormal event handling resources.

[0020] In a second aspect, embodiments of this application provide a security early warning system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the security early warning system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a security warning system, cause the security warning system to execute the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a security warning system, cause the security warning system to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the security early warning system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. This application, through multimodal data acquisition and fusion analysis, extracts behavioral action sequence features, spatial activity trajectory features, and environmental event features from video data, positioning data, and audio data, respectively. These features are then matched with a work scenario feature library to determine scenario attributes, thereby retrieving a target safety rule set for logical correlation analysis. Combined with anomaly detection of behavioral intent, spatial displacement, and environmental changes, a comprehensive anomaly analysis result is generated, and an early warning signal is produced, thus achieving accurate personnel safety early warning in infrastructure work scenarios.

[0025] 2. This application constructs a scene feature vector by initially matching behavioral action sequence features, spatial activity trajectory features, and environmental event features with preset templates to obtain a candidate scene set through multi-dimensional matching. By scoring the candidate scenes and selecting the scene with the highest score, the operation type, regional attributes, and environmental state of the current scene are determined, establishing a precise mapping relationship from features to scenes and improving the accuracy of scene recognition.

[0026] 3. This application obtains standard action sequence templates, safety activity area boundaries, and normal environmental event ranges from the target safety rule set, and compares and analyzes them with real-time features. Anomaly detection is performed based on action combination relationships, execution order, spatial boundaries, and environmental parameter thresholds. The results of various anomaly detections are then integrated to generate anomaly analysis results, achieving comprehensive and accurate anomaly state detection. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a method for personnel safety early warning in infrastructure operation scenarios based on multimodal fusion, as described in this application. Figure 2 This is another flowchart illustrating the personnel safety early warning method for infrastructure operation scenarios based on multimodal fusion in the embodiments of this application; Figure 3 This is a schematic diagram of the physical device structure of a safety early warning system in the embodiments of this application. Detailed Implementation

[0028] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0029] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0030] To facilitate understanding, the application scenarios of the embodiments of this application are described below.

[0031] In a large-scale power transmission line construction project, the construction area spanned mountains and plains, encompassing multiple high-altitude work sites and underground work spaces. The construction process involved various high-risk operations, including overhead power line stringing, underground cable laying, and transformer installation. Due to the complex and dispersed work environment, workers needed to frequently move between different areas and switch between different work types. In one accident, an electrician, after completing overhead power line stringing in a high-altitude work area, failed to return via the designated route, mistakenly entered a live area without wearing protective equipment, resulting in an electric shock. Subsequent analysis revealed that due to limited visibility in the work area, incomplete monitoring of personnel behavior, and insufficient environmental awareness, on-site management personnel were unable to promptly detect and warn of abnormal situations. This reflects that in complex infrastructure construction scenarios, single-dimensional safety monitoring methods are insufficient to meet the comprehensive and real-time safety management needs, necessitating the establishment of a multimodal integrated early warning mechanism.

[0032] A substation expansion project employed a traditional safety monitoring solution, installing fixed cameras in the work area for video surveillance and requiring workers to wear location tags to track their positions. Environmental monitoring sensors were also deployed. However, these devices operated independently, and the data was not integrated or analyzed. In one incident, while workers were repairing high-voltage equipment, although video surveillance showed the worker entering the work area and their location information was normal, blind spots created by equipment obstructions prevented observation of their actions. Simultaneously, environmental sensors detected abnormal noise, but due to a lack of correlation analysis with personnel behavior data, it was impossible to determine whether the noise was due to equipment malfunction or improper operation. Ultimately, this resulted in a short circuit caused by the worker's violation of regulations. This incident exposes the problems of isolated information and poor correlation in single-monitoring methods, failing to achieve comprehensive risk identification and early warning.

[0033] A newly constructed power transmission station project deployed a safety early warning system based on multimodal fusion. The system uses distributed high-definition cameras, positioning base stations, and sound sensors to collect video, location, and sound data of the work area in real time. When an electrician enters the high-voltage equipment maintenance area, the system first identifies the worker's protective equipment and operational sequence based on video analysis, while simultaneously tracking their movement trajectory using positioning data and monitoring equipment operating status via sound sensors. The system inputs this multimodal data into a deep learning model for fusion analysis, automatically identifying the current scenario as "high-voltage equipment maintenance work." The system then invokes the corresponding safety rule set to compare the worker's behavior with standard operating procedures in real time. When it detects that the worker's action sequence deviates from the specifications and their movement trajectory is about to enter a danger zone, the system immediately generates an early warning signal, issuing alarms through on-site warning equipment and management personnel's mobile terminals, promptly stopping potential violations. This demonstrates the comprehensive perception and intelligent early warning capabilities brought about by multimodal data fusion.

[0034] To facilitate understanding, the method provided in this implementation will be described in detail below, using the above scenario as an example. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating a method for early warning of personnel safety in infrastructure operation scenarios based on multimodal fusion, as described in this application.

[0035] S101. Collect video data, personnel positioning data, and audio data of personnel in the work scene.

[0036] Personnel video data refers to the sequence of dynamic images of workers acquired through video capture equipment, used to record and analyze their appearance, posture, and behavioral patterns. Personnel positioning data refers to the spatial location information of workers at a specific point in time, collected through positioning equipment, including coordinate data, movement speed, and direction. Audio data represents various sound signals recorded by audio capture equipment within the work scene, including audio information such as personnel conversations, equipment operation sounds, and environmental noise. The work scene refers to a specific work area during the construction process of an infrastructure project, involving elements such as construction equipment, the work environment, and personnel activity space.

[0037] This step begins after the system starts up and enters monitoring mode, continuously collecting multimodal data from the work site. Specifically, the system first collects video streams using multiple high-definition cameras distributed throughout the work area. The camera placement and angles are optimized to ensure coverage of key work areas and minimize blind spots. Simultaneously, workers are required to wear electronic tags with positioning capabilities, and the location changes of each tag are tracked in real time through a positioning network composed of multiple positioning base stations. In addition, sound acquisition devices are installed at key locations in the work area to continuously collect ambient sound. These devices transmit the collected data to the data processing center in real time via fieldbus or wireless network.

[0038] In some embodiments, multimodal data acquisition of work scenarios can be achieved in multiple ways: Optionally, firstly, sensor deployment planning is carried out based on the spatial layout and risk distribution of the work area; then, video surveillance equipment is installed and debugged to ensure image clarity and field of view meet requirements; subsequently, positioning base stations are deployed and signal strength tests are conducted to ensure positioning accuracy; finally, sound acquisition equipment is installed and noise calibration is performed. Optionally, a 3D model of the work scenario is first constructed and simulation analysis is performed to determine the optimal sensor deployment scheme; then, various sensor devices are deployed according to the scheme and joint debugging tests are conducted; subsequently, a data acquisition network is established and sampling parameters are configured; finally, field verification is performed to ensure data quality. It is understood that other methods can also be used to achieve the multimodal acquisition process of work scenario data. The specific acquisition method can be adjusted according to the actual scenario requirements and is not limited here.

[0039] S102. Analyze the video data to obtain behavioral action sequence features, analyze the positioning data to obtain spatial activity trajectory features, and analyze the sound data to obtain environmental event features.

[0040] Among these, behavioral action sequence features represent the process of personnel movement changes extracted from video data, including temporal features such as action type, duration, and rate of change. Spatial activity trajectory features refer to the personnel movement paths reconstructed based on positioning data, including information such as location coordinate sequences, movement speed curves, and activity range. Environmental event features represent scene acoustic events identified through sound data analysis, including sound pattern features such as equipment operation sounds, abnormal noises, and personnel dialogues. Feature extraction refers to the process of obtaining key information from raw data using technologies such as computer vision and signal processing.

[0041] This step is executed immediately after data acquisition and aims to extract feature representations from various types of raw data that can be used for subsequent analysis. Specifically, the system first preprocesses the video data to eliminate the effects of noise and lighting variations. Then, it uses a deep learning model to identify and track human targets in the scene, extracting the motion posture sequence of each target. For location data, the system removes positioning errors using filtering algorithms and connects discrete location points into continuous motion trajectories based on timestamps. For audio data, the system performs spectral analysis and voiceprint recognition, segmenting the continuous audio stream into independent sound events and extracting feature vectors.

[0042] In some embodiments, feature extraction from multimodal data can be achieved in several ways: Optionally, firstly, object detection algorithms are applied to video data to locate people, then a skeletal keypoint detection model is used to extract human posture, followed by analysis of the action change process based on a temporal model, and finally, standardized action sequence features are generated; Optionally, the original data is first denoised and normalized, then multi-level features are extracted using a pre-trained deep neural network, followed by a feature selection algorithm to select the most discriminative feature subset, and finally, the features of different modalities are transformed into a unified feature space. It is understood that other methods can also be used to achieve the feature extraction process for multimodal data; the specific extraction method can be selected according to the data characteristics and application requirements, and is not limited here.

[0043] S103. Match the behavioral action sequence features, spatial activity trajectory features, and environmental event features with the work scene feature database to determine the current scene's work type, work area attributes, and work environment status.

[0044] The "Work Scenario Feature Library" is a database storing feature templates for various typical work scenarios, containing standard feature descriptions for different work types. Work type refers to a specific construction project in infrastructure engineering, such as high-altitude work, equipment installation, and pipeline welding. Work area attributes represent the spatial characteristics and safety level of the construction area, including high-altitude areas, confined spaces, and hazardous material areas. Work environment status refers to the current state of environmental factors affecting construction safety, such as temperature, humidity, lighting, and noise. Feature matching refers to the process of calculating and comparing the similarity between real-time extracted features and pre-stored feature templates.

[0045] This step is executed after feature extraction is completed, aiming to accurately identify the current work scenario through comprehensive analysis of multimodal features. Specifically, the system first matches the extracted behavioral action sequence features with preset standard work action templates to preliminarily determine the possible work type. Simultaneously, it compares spatial activity trajectory features with regional boundary information to determine the attributes of the work area. Furthermore, it analyzes the degree of matching between environmental event features and environmental feature templates to assess the current environmental state. Through joint matching and weighted fusion of multidimensional features, the system ultimately determines the most matching scenario type.

[0046] In some embodiments, feature matching and scene recognition can be implemented in multiple ways: Optionally, a feature vector space is first established to uniformly represent multimodal features. Then, the similarity to template features is calculated using metrics such as cosine similarity. Next, a multilayer perceptron model is used to fuse the matching results of different features. Finally, a voting mechanism is used to select the optimal matching scene type. Optionally, a hierarchical scene classification model is first constructed. Then, multimodal features are input into their respective sub-models for preliminary classification. Next, the classification results of each sub-model are fused using a confidence-weighted approach. Finally, the results are optimized based on the statistical distribution of historical data. It is understood that other methods can also be used to implement the feature matching and scene recognition process. The specific implementation method can be selected according to the application scenario and performance requirements, and is not limited here.

[0047] S104. Based on the job type, job area attributes, and job environment status, retrieve the target safety rule set from the safety specification library.

[0048] The safety specification library represents a knowledge base storing safe operating procedures and risk control requirements for various work scenarios. The target safety rule set refers to the collection of safety specifications and control rules related to the current work scenario, including operating procedure specifications, safety protection requirements, and environmental limit standards. Rule retrieval refers to the process of searching and filtering relevant rules from the specification library based on scenario characteristics. A rule set refers to a group of interrelated and synergistic safety management rules.

[0049] This step is executed after the characteristics of the work scenario are determined, with the aim of obtaining safety management rules that match the current scenario. Specifically, the system first retrieves the corresponding work instructions and safe operating procedures from the safety specification library based on the identified work type. Then, based on the attributes of the work area, it supplements this with safety protection requirements and emergency response rules specific to that area. Simultaneously, considering the current environmental conditions, it adds safety limits and warning thresholds related to environmental factors. The system integrates these rules and eliminates conflicts to form a complete rule set for the current scenario.

[0050] In some embodiments, the retrieval and integration of security rules can be achieved in several ways: Optionally, a rule retrieval index is first constructed to map job type, regional attributes, and environmental status to corresponding rules. Then, multi-dimensional retrieval is performed based on scenario features to obtain candidate rules. Next, the correlation and priority relationships between rules are analyzed. Finally, a consistent rule set is generated through rule conflict detection and resolution. Optionally, a correlation model between scenario features and security rules is first established. Then, knowledge graph technology is used to conduct rule reasoning and correlation analysis. Afterward, rules are filtered based on their timeliness and applicability. Finally, a rule completion mechanism ensures the completeness of the rule set. It is understood that other methods can also be used to achieve the retrieval and integration of security rules. The specific implementation method can be selected according to the rule system and application requirements, and is not limited here.

[0051] S105. Based on the target safety rule set, perform logical correlation analysis on the characteristics of behavioral action sequence, spatial activity trajectory, and environmental event characteristics to determine whether there are abnormal behavioral intentions, abnormal spatial displacements, or abnormal environmental changes, and obtain the abnormality analysis results.

[0052] Logical association analysis refers to the process of comprehensively evaluating and logically reasoning about multimodal features based on a rule set. Abnormal behavioral intent refers to situations where the sequence of actions performed by workers deviates from standard operating procedures or involves unsafe behavior. Abnormal spatial displacement indicates that personnel's movement trajectory exceeds the boundary of the safe area or enters a restricted area. Abnormal environmental change refers to situations where environmental parameters exceed safety thresholds or abnormal events occur. The anomaly analysis result refers to the comprehensive assessment of various abnormal states and their severity obtained through rule-based judgment.

[0053] This step is executed after acquiring the target safety rule set, and its purpose is to perform anomaly detection based on the multi-dimensional features of the current scenario. Specifically, the system first extracts standard action templates corresponding to the job type from the rule set, compares them temporally with real-time behavioral sequence features, and detects whether there are any violations. Simultaneously, it determines whether the spatial trajectory conforms to activity boundary restrictions based on regional safety rules. Furthermore, it assesses whether current environmental events exceed normal limits by combining environmental safety standards. The system uses a rule reasoning engine to perform correlation analysis on these abnormal states, considering the causal relationships and combined effects between anomalies, and finally arrives at a comprehensive anomaly assessment result.

[0054] In some embodiments, anomaly detection and correlation analysis can be implemented in several ways: Optionally, a behavior analysis model based on temporal logic is first established, then the deviation between the behavior sequence and the standard template is calculated using a dynamic time warping algorithm, followed by setting multi-level anomaly thresholds for hierarchical judgment, and finally, the anomaly level and type are derived through rule-based reasoning; Optionally, a multi-dimensional anomaly detection framework is first constructed, then specialized anomaly detection algorithms are applied to the behavior, trajectory, and environmental dimensions respectively, then the probabilistic dependencies between anomalies are modeled using Bayesian networks, and finally, multi-dimensional anomaly information is fused based on evidence theory. It is understood that other methods can also be used to implement the anomaly detection and correlation analysis process; the specific implementation method can be selected according to the anomaly type and detection requirements, and is not limited here.

[0055] S106. Generate corresponding early warning signals based on the anomaly analysis results.

[0056] The warning signal represents a safety risk alert generated by the system based on anomaly analysis results. The warning level refers to the alert grade determined according to the severity and urgency of the anomaly. Warning methods include various information delivery methods such as audible and visual alarms, SMS notifications, and system push notifications. The warning recipients are relevant personnel who need to receive the warning information, including on-site workers, safety management personnel, and emergency response personnel.

[0057] This step is executed after anomaly analysis is completed, and its purpose is to promptly notify relevant personnel of any detected anomalies so that they can take appropriate countermeasures. Specifically, the system first determines the level and type of alert based on the anomaly analysis results, including different levels such as alert, warning, and emergency. Then, it selects the appropriate alert method based on the alert level; for example, multiple alert methods may be activated simultaneously for an emergency level. The system also determines the scope of recipients of the alert information based on the alert level and generates alert content that includes an anomaly description, location information, and handling suggestions.

[0058] In some embodiments, the generation and transmission of early warning signals can be achieved in multiple ways: Optionally, firstly, a mapping relationship between anomaly levels and early warning strategies is established; then, standardized early warning information is generated based on the type and severity of the anomaly; subsequently, an appropriate early warning channel is selected for information push; and finally, the early warning process is recorded and the handling status is tracked. Optionally, firstly, an intelligent early warning decision-making model is constructed; then, the early warning strategy is optimized based on historical early warning effects; subsequently, the early warning method and content are dynamically adjusted in conjunction with the scenario context; and finally, a reliable transmission mechanism for early warning information ensures information delivery. It is understood that other methods can also be used to achieve the generation and transmission of early warning signals; the specific implementation method can be selected according to the early warning requirements and system conditions, and is not limited here.

[0059] The following provides a more detailed description of the process of the method provided in this implementation. Please refer to [link / reference]. Figure 2This is another flowchart illustrating the personnel safety early warning method for infrastructure operation scenarios based on multimodal fusion in this application embodiment.

[0060] S201. Collect video data, personnel positioning data, and audio data of personnel in the work scenario.

[0061] The work scenario refers to a specific work area during the construction of an infrastructure project, including elements such as construction equipment, the work environment, and personnel activity space. Personnel video data is a sequence of dynamic images of workers acquired through video equipment. Personnel positioning data refers to the spatial location information of workers acquired through positioning equipment. Audio data represents various sound signals within the work scenario recorded by audio acquisition equipment.

[0062] This step involves data acquisition via a multimodal data acquisition network deployed at the work site. Video data acquisition utilizes a high-definition network camera array, deploying multiple fixed-view cameras and adjustable pan-tilt cameras in the construction area. Camera resolution is no less than 1080P, and the frame rate is maintained at 25fps or higher. The video stream is transmitted to the data center in real-time via the network. Positioning data acquisition requires personnel to wear RFID electronic tags or UWB positioning terminals. Multiple positioning base stations are deployed in the construction area according to signal coverage requirements, with a sampling frequency set to 1Hz, recording tag IDs and 3D coordinate information in real-time. Audio data acquisition uses an omnidirectional microphone array with a microphone sensitivity no less than -38dB and a sampling rate set to 44.1kHz. The acquired audio data undergoes preliminary noise reduction processing before being transmitted to the data center. All acquisition devices are networked via fieldbus or industrial Ethernet to ensure real-time data transmission and reliability.

[0063] S202. Analyze the video data to obtain behavioral action sequence features, analyze the positioning data to obtain spatial activity trajectory features, and analyze the sound data to obtain environmental event features.

[0064] Among these, behavioral sequence features refer to descriptive information about the changes in human movements extracted from videos. Spatial activity trajectory features represent the human movement path information reconstructed based on positioning data. Environmental event features are the acoustic event features of a scene obtained through analysis and identification of sound data. Feature extraction refers to the process of extracting key information from raw data using computer vision and signal processing techniques.

[0065] This step involves feature extraction and analysis of the collected raw data. For video data processing, target detection and personnel localization are performed first. YOLOv5 and other target detection algorithms are used to extract personnel bounding boxes. Then, a human pose estimation model is applied to extract the spatial coordinates of 17 key points. Based on the key point coordinate sequence, joint angles and motion parameters are calculated, ultimately generating standardized motion feature vectors. For localization data processing, a Kalman filter algorithm is used to eliminate localization noise. Discrete coordinate points are connected into a trajectory line through temporal correlation. Kinematic parameters such as velocity, acceleration, and direction of the trajectory are calculated to construct a trajectory feature descriptor. For audio data processing, a short-time Fourier transform is first performed to obtain the spectrogram, and acoustic features such as Mel frequency cepstral coefficients are extracted. A sound event detection model is used to segment continuous audio into independent events, and the acoustic feature vector of each event is extracted. All features are normalized before being used for subsequent analysis.

[0066] S203. Match the behavioral action sequence features with the preset action feature template to obtain the preliminary operation type; compare the spatial activity trajectory features with the preset area boundary information to obtain the preliminary area attributes; and match the environmental event features with the preset environmental feature template to obtain the preliminary environmental state.

[0067] Among them, the action feature template refers to the feature description of predefined standard operation actions. The area boundary information represents the spatial range definition of various operation areas. The environmental feature template refers to the standard acoustic feature description of different working environments. The preliminary operation type, area attribute, and environmental state represent the preliminary scene judgment results obtained through single-modal feature matching.

[0068] This step determines the initial attributes of the scene through feature matching. Action feature matching uses a dynamic time warping algorithm to calculate the similarity between real-time action sequences and various standard actions in a template library. The similarity calculation considers the spatial shape and temporal variation characteristics of keypoint trajectories, selecting the action type with the highest similarity as the initial task type. Spatial trajectory matching calculates the spatial relationship between real-time trajectory points and predefined region boundary polygons, determining whether trajectory points fall within a specific region and calculating the trajectory's dwell time in each region to comprehensively determine the initial region attributes. Environmental feature matching uses a support vector machine classifier to classify acoustic features, matching real-time sound events with preset environmental acoustic templates and selecting the category with the highest confidence as the initial environmental state. During the matching process, a sliding time window is used for feature aggregation, with a window size of 10 seconds and a step size of 1 second to ensure the temporal continuity of features.

[0069] S204. Construct a scene feature vector based on the preliminary operation type, preliminary area attributes, and preliminary environmental status. Input the scene feature vector into the operation scene feature library for multi-dimensional matching to obtain a set of candidate scenes.

[0070] Here, scene feature vectors refer to a unified numerical representation that encodes the preliminary analysis results. The task scene feature library represents a database storing predefined scene feature templates. Multi-dimensional matching refers to the process of calculating vector similarity in the feature space. The candidate scene set represents several scene types with high matching degrees and their corresponding matching scores.

[0071] This step converts the preliminary analysis results into standard feature vectors and performs scene matching. Feature vector construction uses one-hot encoding, encoding job type, region attribute, and environmental state into fixed-dimensional binary vectors (e.g., job type vector with dimension 20, region attribute vector with dimension 15, and environmental state vector with dimension 10). These sub-vectors are then concatenated to form a 45-dimensional scene feature vector. The feature matching process first calculates the cosine similarity between the input vector and all scene templates in the feature library. The similarity calculation formula is the dot product of the two vectors divided by the product of their magnitudes. For each dimension, a weight coefficient is introduced to adjust its importance in the similarity calculation; the weight value is set based on prior knowledge. The calculated similarities are sorted in descending order, and scenes with a similarity greater than a preset threshold of 0.8 are selected as the candidate set. Each candidate scene includes a scene ID, scene type description, and similarity score.

[0072] S205. Analyze the changing trends of behavioral action sequence features, spatial activity trajectory features, and environmental event features in the time dimension, identify key time segments in each feature, and these key time segments represent the state transition process in which the features change significantly within a preset time period.

[0073] Here, the trend of change refers to the evolution of the feature over time. A key time segment refers to the time interval during which the feature undergoes significant changes. The state transition process represents the dynamic process by which the feature value changes from one stable state to another. The preset time period is the size of the time window for trend analysis.

[0074] This step identifies key intervals of feature changes through time-series analysis. For behavioral action sequence features, sliding variance is used to calculate the local fluctuation of attitude parameters, with a variance calculation window set to 2 seconds. When the variance exceeds a threshold, it is marked as an action transition point. The displacement and angle change rates of adjacent key points are calculated, and intervals with change rates exceeding a preset threshold are marked as key action segments. For spatial trajectory features, the instantaneous velocity and acceleration of trajectory points are calculated, and piecewise linear regression is used to fit the motion parameter curves. Trajectory segments are segmented at points of abrupt change in fitting error, and segments with significant changes in velocity or direction are marked as key trajectory segments. For environmental event features, the Z-score method is used to detect outliers in acoustic features, and the first-order difference sequence of features is calculated. Time periods with significant difference values ​​are marked as key environmental segments. The start and end timestamps and amplitude values ​​of state changes are recorded for key segments of various features.

[0075] S206. Retrieve historical scene data similar to the job type from the historical scene sample library, filter the historical scene data based on key time segments, and determine the influence weight of each feature on scene recognition.

[0076] The historical scene sample library refers to a database storing labeled scene data. Similar historical scenes refer to historical records with the same or similar attributes as the current task type. Influence weights represent the importance of different features in scene recognition. Feature filtering refers to the process of filtering historical data based on time-series features.

[0077] This step determines feature weights through historical data analysis. First, a scene similarity calculation function is constructed, combining semantic similarity of task types and attribute overlap to calculate the similarity between scenes, selecting historical scene data with a similarity exceeding 0.7. For each historical data point, time window data corresponding to the current key time segment is extracted, and the dynamic time warping distance of the feature sequence is calculated. Samples with a distance less than a threshold are considered valid samples. The discriminative power of each feature in the valid samples is statistically analyzed, and the feature weights are calculated using the information gain ratio method, calculated as the feature's information gain divided by its intrinsic information value. Finally, the weight coefficients for behavioral features, trajectory features, and environmental features are obtained, with weight values ​​ranging from [0, 1], and the sum of the weights of the three types of features is 1.

[0078] S207. Dynamically match key time segments with the corresponding features of candidate scenes, calculate the state transition similarity of each time segment, and determine the weight coefficient of each time segment based on the importance of the state transition.

[0079] Dynamic matching refers to the process of aligning and comparing time-series data using a dynamic programming algorithm. State transition similarity represents the degree of similarity in the state change processes of two time-series segments. Importance refers to the significance of the state transition's impact on scene recognition. Weighting coefficients are numerical parameters used to adjust the importance of different time-series segments in the overall matching process.

[0080] This step calculates the similarity of time-series segments using a dynamic time warping algorithm. For each candidate scene, state transition segments are extracted from its feature sequence, establishing a correspondence with the current key time-series segment. When calculating state transition similarity, the time-series data is first normalized, and then a cumulative distance matrix is ​​constructed, where each element represents the Euclidean distance between corresponding points in the two sequences. A dynamic programming algorithm is used to calculate the optimal path, and the path cost is the state transition similarity. Importance is calculated based on the magnitude and duration of the state transition. The magnitude of the transition is calculated using the change in eigenvalues, and the duration is the time span of the state transition. The importance is mapped to the [0, 1] interval to obtain weight coefficients, using the Sigmoid function. For each time-series segment, its state transition similarity and weight coefficients are recorded.

[0081] S208. Multiply the state transition similarity of each key time segment by the corresponding weight coefficient and add them together to obtain the matching score of the candidate scene.

[0082] The matching score represents the overall similarity between the candidate scene and the current scene. Weight coefficients are importance parameters for each time segment. State transition similarity is the degree of matching between time segments. Weighted summation is the calculation process of adding multiple weighted terms together.

[0083] This step calculates the final matching score through weighted fusion. For each candidate scene, the state transition similarity and weight coefficients of all its key temporal segments are obtained. The state transition similarity is standardized to the [0, 1] interval to ensure the comparability of similarities between different features. When calculating the weighted score, the state transition similarity of each temporal segment is multiplied by the corresponding weight coefficient to obtain the weighted term. All weighted terms are summed to obtain the total matching score of the candidate scene, calculated as Σ(wi×si), where wi is the weight coefficient of the i-th temporal segment and si is the corresponding state transition similarity. This calculation process is repeated for all candidate scenes to obtain the matching score for each candidate scene.

[0084] S209. Select the candidate scenario with the highest matching score, determine the job type corresponding to the candidate scenario as the job type of the current scenario, determine the area attribute corresponding to the candidate scenario as the job area attribute of the current scenario, and determine the environmental state corresponding to the candidate scenario as the job environment state of the current scenario.

[0085] Among them, the candidate scenario with the highest matching score refers to the scenario with the highest overall similarity among all candidates. The job type is a classification identifier for the nature of the current construction operation. The job area attribute represents the spatial characteristics and safety level of the construction area. The job environment status refers to the state of environmental parameters affecting construction safety.

[0086] This step determines the attributes of the current scenario through optimal matching. First, the matching scores of candidate scenarios are sorted in descending order, and the scenario with the highest score is selected as the optimal matching result. From the attribute information of the optimal matching scenario, a job type identifier is extracted, which includes two levels of information: job category and sub-category. Job area attribute information is extracted, including parameters such as area type, safety level, and spatial range. Environmental status information is extracted, including the current values ​​and trends of environmental parameters. These attribute information are assigned to the corresponding fields of the current scenario, completing the determination of the scenario attributes. For each determined attribute, its corresponding confidence value is also recorded; the confidence value is calculated based on the matching score of the optimal matching scenario.

[0087] S210. Based on the job type, job area attributes, and job environment status, retrieve the target safety rule set from the safety specification library.

[0088] The safety specification library refers to a knowledge base storing safety procedures and management requirements for various work scenarios. The target safety rule set represents a collection of safety specifications and control rules relevant to the current scenario. Retrieval refers to the process of retrieving and extracting rules from the specification library based on scenario characteristics. A rule set refers to a group of interrelated and synergistic safety management rules.

[0089] This step retrieves safety rules from the specification library using a multi-dimensional index. First, rule retrieval criteria are constructed, converting job types into rule classification codes and regional attributes and environmental states into rule filtering parameters. A multi-level index structure is used to locate rules in the specification library, with index levels including job category index, regional type index, and environmental condition index. Rule retrieval employs a combination of exact matching and fuzzy matching; exact matching is used for rule location based on job type, while fuzzy matching is used for supplementary retrieval of rules related to region and environment. The retrieved rules are then integrated, duplicate rules are removed, and rule conflicts are resolved to form a complete rule set specific to the current scenario. The rule set includes multiple rule subsets such as operating procedures, safety limits, and emergency plans. Each rule contains fields such as rule ID, applicable conditions, constraints, and execution requirements.

[0090] S211. Obtain from the target safety rule set the standard action sequence template corresponding to the operation type, the safety activity area boundary corresponding to the operation area attribute, and the normal environmental event range corresponding to the operation environment status.

[0091] Here, the standard action sequence template refers to the standard procedure for work actions specified in the standard. The safe activity area boundary represents the defined safe activity range of the work area. The normal environmental event range refers to the allowable variation range of environmental parameters. Acquisition refers to the process of extracting specific constraint parameters from the rule set.

[0092] This step extracts specific constraint parameters through rule parsing. Rules related to job types are parsed to extract standard action sequence templates. These templates describe the required action combinations and execution order, including mandatory actions, prohibited actions, and action timing constraints. Rules related to area attributes are parsed to extract safe activity area boundaries. These boundaries are described by spatial coordinate point sequences or geometric shapes, including permitted activity areas and prohibited entry areas. Rules related to environmental status are parsed to extract the range of normal environmental events. This range is described by upper and lower limits of parameters or statistical distributions, including warning and critical values ​​for each environmental indicator. All extracted constraint parameters are stored in a standard format for easy use in subsequent anomaly detection.

[0093] S212. Compare the behavioral action sequence features with the standard action sequence template in time. When the action sequence is found to be inconsistent with the action combination relationship or execution order in the standard template, it is determined that there is an abnormal behavioral intention.

[0094] Among these, temporal comparison refers to comparing real-time action sequences with standard templates over time. Action combination relationships represent the rules for combining multiple actions. Execution order refers to the required sequence of actions. Abnormal behavioral intent indicates that the operator's action sequence violates safety regulations.

[0095] This step identifies abnormal behavior through sequence comparison. First, the real-time action sequence and a standard template are temporally aligned. A sliding window approach is used for local comparison, with the window size set according to the action duration. The comparison process checks three aspects: action type matching degree (calculating the feature similarity between the actual action and the standard action); combination relationship conformity (checking whether the action combination meets the rule constraints); and sequence consistency (verifying whether the action execution order conforms to regulations). Anomalies are determined when the following conditions are detected: action feature similarity is below the threshold of 0.75, indicating non-standard action execution; prohibited action combinations appear, indicating a violation of operating procedures; and the action order does not conform to the standard process, indicating a chaotic work process. For each detected anomaly, the anomaly type, occurrence time, and degree of violation are recorded.

[0096] S213. The spatial activity trajectory characteristics are compared with the boundary of the safe activity area in real time. When the trajectory point is detected to exceed the safe boundary, it is determined that there is an abnormal spatial displacement.

[0097] Real-time comparison refers to the continuous calculation of the spatial relationship between trajectory points and boundaries. The safety boundary represents the boundary line of the safe activity range of the work area. Trajectory points are the spatial coordinate points from which positioning data is collected. Spatial displacement anomalies refer to situations where personnel movement trajectories violate spatial activity constraints.

[0098] This step detects trajectory anomalies through spatial calculation. For each newly acquired trajectory point, its spatial relationship with the boundary of the safe activity area is calculated. The spatial relationship calculation uses the ray casting method to determine whether a point is inside a polygon. First, the boundary of the safe area is represented as a polygon. Then, rays are drawn from the trajectory point in any direction, and the number of intersections between the rays and the polygon boundary is counted. An odd number of intersections indicates the point is inside the polygon, while an even number indicates the point is outside. For complex areas, they are decomposed into multiple simple polygons for evaluation. When a trajectory point is located inside the restricted area or outside the safe area, a spatial displacement anomaly is triggered. The anomaly record includes the boundary coordinates, boundary time, and boundary distance. The boundary distance is obtained by calculating the Euclidean distance from the trajectory point to the nearest boundary.

[0099] S214. Match the characteristics of environmental events with the range of normal environmental events. When the type, intensity or frequency of an environmental event is detected to exceed the preset threshold range, it is determined that there is an abnormal environmental change.

[0100] Among them, environmental event characteristics represent the environmental state description obtained from sound data analysis. The normal environmental event range refers to the permissible range of environmental parameters as defined by regulations. Preset thresholds are the warning and critical values ​​for environmental parameters. Abnormal environmental changes refer to situations where the environmental state exceeds the safe range.

[0101] This step detects environmental anomalies through parameter matching. Environmental event features are categorized and matched, calculating the similarity between event types and preset types. Intensity parameters of environmental events are calculated, including sound pressure level, frequency characteristics, and duration. The frequency of events occurring per unit time is statistically analyzed. These feature values ​​are compared with normal ranges: an event type matching degree below 0.8 is considered an anomaly; an event intensity exceeding the upper or lower limit is considered an intensity anomaly; and an event frequency exceeding a preset frequency is considered a frequency anomaly. For continuous parameters, the Z-score method is used to detect outliers, while for discrete parameters, a rule-based judgment method is used. Anomaly records include the anomaly type, parameter value, and degree of exceeding limits.

[0102] S215. Based on the judgment results of abnormal behavioral intentions, abnormal spatial displacement, and abnormal environmental changes, generate anomaly analysis results.

[0103] The anomaly analysis results represent a comprehensive description of the various detected anomalies. The judgment results include anomaly type, anomaly severity, and spatiotemporal information. Generation refers to the process of integrating multiple anomaly judgment results into a unified description. The result is an assessment output of the current scene's security status.

[0104] This step generates analysis results through anomaly integration. First, anomalies are categorized into three levels: minor, moderate, and severe, based on their severity. The severity is calculated based on the following indicators: behavioral intent anomalies are assessed based on the number and duration of violations; spatial displacement anomalies are assessed based on the distance and duration of boundary violations; and environmental change anomalies are assessed based on the magnitude and scope of the exceedance. Then, anomaly correlation analysis is performed to identify causal relationships and combined effects between anomalies, such as how certain environmental anomalies may lead to behavioral anomalies. Finally, structured anomaly analysis results are generated, including fields such as anomaly overview, detailed description, risk level, occurrence time, duration, and scope of impact, and the results are sorted according to the severity of the anomalies.

[0105] S216. Generate corresponding early warning signals based on the anomaly analysis results.

[0106] The warning signal represents a security risk alert generated based on anomaly analysis results. The warning level refers to the alert level determined by the severity of the anomaly. The warning information includes an anomaly description, location information, and handling suggestions. The warning method refers to the specific form of information delivery, such as audible and visual alarms, or push notifications.

[0107] This step generates early warning information through rule mapping. The early warning level is determined based on the risk level of the anomaly analysis results, with three levels: Level 1 corresponds to severe anomalies and triggers a red warning signal; Level 2 corresponds to general anomalies and triggers a yellow warning signal; and Level 3 corresponds to minor anomalies and triggers a blue warning signal. Early warning information generation uses a template-based approach, with each template containing the following elements: warning time, warning area, anomaly type, anomaly description, impact range, and handling recommendations. The early warning method triggers different early warning devices based on the warning level: Level 1 warnings simultaneously activate audible and visual alarms, SMS notifications, and system push notifications; Level 2 warnings activate audible and visual alarms and system push notifications; and Level 3 warnings only send system push notifications. For complex anomalies, the early warning information is determined according to the highest level of anomaly. Early warnings are sent using a tiered push strategy: Level 1 warnings are pushed to all relevant personnel; Level 2 warnings are pushed to on-site management personnel and operators; and Level 3 warnings are pushed to operators. The time of warning transmission, recipients, and confirmation status are recorded. For persistent anomalies, warnings are repeatedly sent at preset time intervals until the anomaly is resolved. When an early warning is resolved, a warning resolution notification is generated, including the anomaly resolution time and handling results.

[0108] In some embodiments, after step S216, the method further includes: The system acquires real-time detection data of the same monitoring target from multiple heterogeneous sensors at the construction site, and identifies conflicting states between the detection results of different sensors in the detection data. These conflicting states refer to the mutually exclusive state judgments given by different sensors for the same physical event at the same time.

[0109] Specifically, heterogeneous sensors refer to sensing devices employing different detection principles and methods. The monitoring target represents the physical object or event requiring state monitoring. Real-time detection data consists of state parameters continuously collected by the sensors. Conflicting states refer to situations where there are logical contradictions between the detection results of different sensors. Mutually exclusive states mean that the detection results cannot be simultaneously valid in a physical sense. This step identifies conflicts through data synchronization and logical comparison. First, the detection data from multiple sensors are synchronized in time, using timestamp alignment to organize data from different sensors at the same moment. Data synchronization employs a sliding time window mechanism with a window size of 100ms to ensure the capture of instantaneous state changes. Then, logical relationship analysis of the sensor data is performed to establish a constraint relationship model for the detection results. The constraints include numerical range constraints, trend constraints, and state combination constraints. A conflict is determined to exist when the following situations are detected: the numerical differences between different sensors exceed the allowable error range; the trends of sensor detection values ​​are contradictory; and the sensor state combination violates physical laws. For each detected conflict, the time of occurrence, the involved sensors, and the specific numerical values ​​are recorded.

[0110] Features of the conflict state are extracted and categorized into predefined conflict patterns. These features include the duration of the conflict, the combination of sensors involved in the conflict, and the changing trends of the detection values ​​of each sensor.

[0111] Specifically, the conflict duration refers to the duration of the conflict state. The sensor combination represents the set of sensors involved in the conflict. The trend refers to the pattern of change in the detected values ​​over time. The conflict pattern is a predefined typical conflict type. Feature extraction refers to the process of extracting key descriptive information from the conflict state. This step classifies conflicts through feature analysis. First, the temporal characteristics of the conflict are calculated, including the start time, end time, and duration. The sensor combinations involved in the conflict are statistically analyzed, recording the type and number of each sensor in the combination. Trend analysis is performed on the detection sequence of each sensor, using moving linear regression to calculate the slope of data change and extracting abrupt change points and trend inflection points. The extracted features are matched with predefined conflict patterns, including: instantaneous conflict, continuous conflict, periodic conflict, etc. The matching process considers the similarity of features and the constraints of the pattern, selecting the best-matching pattern as the conflict type. For conflicts that cannot be classified into existing patterns, their features are recorded as new conflict patterns.

[0112] Based on the characteristic attributes of the conflict mode and the detection mechanism of each sensor, the key anomaly factors corresponding to the conflict state are determined.

[0113] Specifically, the detection mechanism refers to the working principle and detection method of the sensor. Characteristic attributes represent the typical features of the conflict mode. Key anomaly factors refer to the main causes of conflicting detection results. Determination refers to the process of inferring the cause of the anomaly through analysis. This step determines the cause of the anomaly through mechanism analysis. First, a sensor detection mechanism model is established, describing the sensor's measurement principle, influencing factors, and error sources. The characteristic attributes of the conflict mode are analyzed, including the temporal, spatial, and numerical characteristics of the conflict. Combining the detection mechanism and conflict characteristics, possible anomaly factors are inferred. Anomaly factor analysis considers the following aspects: sensor malfunctions, such as sensitivity drift and zero-point offset; environmental interference factors, such as temperature changes and electromagnetic interference; and anomalies in the measured object, such as sudden changes in state or multiple state superpositions. Verification analysis is performed on each anomaly factor, confirming the cause of the anomaly through historical data comparison and on-site verification. Finally, an anomaly factor analysis report is output, including the anomaly type, degree of impact, and handling recommendations.

[0114] Determine the weighting coefficients corresponding to key anomalies. These weighting coefficients are related to the duration of the conflict and the number of sensors involved in the conflict.

[0115] Specifically, the weighting coefficient represents the importance of key anomalies in early warning judgment. Duration refers to the length of time the conflict state persists. Number of sensors is the number of sensors involved in the conflict. Correlation refers to the functional relationship between weight, duration, and number. This step determines the weight value through quantitative calculation. The weight calculation uses a composite function model, including a time factor and a quantity factor. The time factor is calculated based on the duration of the conflict, using an exponential growth function f(t) = 1 - e^(-t / T), where t is the duration and T is the characteristic time constant. The quantity factor is calculated based on the number of sensors, using a logarithmic function g(n) = log(n+1) / log(N+1), where n is the actual number of sensors and N is the total number of sensors in the system. The final weighting coefficient is obtained through a weighted combination of the two factors: w = αf(t) + βg(n), where α and β are balance coefficients, and α + β = 1. The weighting coefficient ranges from [0, 1], with larger values ​​indicating higher importance of the anomaly.

[0116] The anomaly similarity between key anomaly factors and anomaly analysis results is calculated. This anomaly similarity is determined based on the correspondence between the location and degree of anomaly occurrence.

[0117] Specifically, anomaly similarity refers to the degree of matching between anomaly factors and analysis results. The location of the anomaly indicates its spatial distribution characteristics. The degree of anomaly refers to the severity of the anomaly state. The correspondence is a matching relationship between the two dimensions of location and degree. This step evaluates the anomaly matching degree through similarity calculation. First, a spatial location similarity calculation model is established, using a spatial distance metric. When the location distance between the anomaly factor and the analysis result is less than a threshold, they are considered similar in location. Spatial distance is calculated using Euclidean distance: d = sqrt((x1-x2)^2 + (y1-y2)^2 + (z1-z2)^2). Then, the degree of anomaly similarity is calculated, standardizing the degree of anomaly to the [0, 1] interval, and using cosine similarity to calculate the degree matching value: s = cos(θ) = (a·b) / (|a|·|b|), where a and b are the feature vectors of the anomaly factor and the analysis result, respectively. The final anomaly similarity is obtained by a weighted average of location similarity and degree similarity.

[0118] The credibility of the early warning signal is calculated based on the weighting coefficient and the anomaly similarity.

[0119] Specifically, the warning signal represents an abnormal risk alert generated by the system. Credibility refers to the accuracy and reliability of the warning signal. Calculation refers to the process of evaluating credibility through a mathematical model. Weighting coefficients and anomaly similarity are the input parameters for credibility calculation. This step determines the credibility of the warning through comprehensive evaluation. Credibility calculation uses a weighted product model: C = w × s × k, where w is the weighting coefficient, s is the anomaly similarity, and k is the correction coefficient. The correction coefficient considers historical warning accuracy and environmental influence factors and is dynamically adjusted using a Bayesian update method. The credibility value ranges from [0, 1]. When the credibility exceeds a preset threshold (e.g., 0.8), a high-credibility warning is triggered; when the credibility is in the middle range (e.g., 0.5-0.8), a medium-credibility warning is generated; when the credibility is low (e.g., less than 0.5), it is only recorded as reference information. The system also records the intermediate parameters and evaluation basis of the credibility calculation for the interpretation and verification of the warning results.

[0120] The handling priority of abnormal events is determined based on the credibility and type of early warning signal, and emergency response instructions are generated according to the handling priority.

[0121] Specifically, the handling priority indicates the urgency and importance of handling abnormal events. Warning signal type refers to different categories of abnormal warnings. Emergency response instructions are the specific handling measures required for abnormal events. Generation refers to the process of constructing response instructions based on rules and templates.

[0122] This step generates a response plan through priority assessment, comprising two main stages: priority determination and emergency response instruction generation. Priority determination begins with constructing an assessment matrix. Rows represent the type of warning signal (e.g., safety, equipment, environmental), and columns represent confidence intervals (high confidence [0.8, 1], medium confidence [0.5, 0.8], low confidence [0, 0.5]). Matrix elements define priority scores from 1 to 9. Priority calculation combines a base score with a correction factor. The base score is obtained from the assessment matrix, while the correction factor considers time urgency (remaining time), impact scope (affected areas and number of personnel), and cascading effects (probability of triggering other anomalies). The final priority score is calculated by multiplying the base score by the correction factor. Emergency response instruction generation begins by retrieving a basic instruction template from a pre-set template library. Based on the anomaly type and priority, specific parameters and requirements are filled in to generate a complete instruction containing an instruction number, timestamp, anomaly description, location information, response priority, time limit, response steps, precautions, resource allocation, contact information, confirmation method, and expected results. Instruction distribution employs a tiered push strategy: high-priority instructions (score ≥ 7) are immediately pushed to all staff requiring immediate action; medium-priority instructions (4 ≤ score < 7) are pushed to the responsible person requiring timely action; and low-priority instructions (score < 4) are logged and processed periodically. The system continuously monitors instruction execution. When abnormal conditions change, the handling effect is unsatisfactory, new risks arise, or resource conditions change, instructions are dynamically adjusted. Adjusted instructions must be re-evaluated for priority and ensure consistency with the original handling process. Simultaneously, the entire execution process, the reasons for adjustment, and the handling effect are recorded for subsequent optimization.

[0123] The security early warning system in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference]. Figure 3 This is a schematic diagram of the physical device structure of a safety early warning system in an embodiment of this application.

[0124] It should be noted that, Figure 3 The structure of the safety warning system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0125] like Figure 3As shown, the security early warning system includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 302 or programs loaded from storage section 308 into Random Access Memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0126] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0127] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0128] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0130] Specifically, the safety warning system in this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the personnel safety warning method for infrastructure operation scenarios based on multimodal fusion provided in the above embodiment.

[0131] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the safety warning system described in the above embodiments; or it may exist independently and not be assembled into the safety warning system. The storage medium carries one or more computer programs, which, when executed by a processor of the safety warning system, enable the safety warning system to implement the multimodal fusion-based personnel safety warning method for infrastructure operation scenarios provided in the above embodiments.

[0132] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0133] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for personnel safety early warning in a capital construction operation scene based on multi-modal fusion, characterized in that, The method is applied to a safety warning system, and comprises the following steps: Collecting video data, personnel positioning data and sound data of personnel in a work scene; Analyzing the video data to obtain behavior action sequence features, analyzing the positioning data to obtain spatial activity trajectory features, and analyzing the sound data to obtain environmental event features; Matching the behavior action sequence features, the spatial activity trajectory features and the environmental event features with a work scene feature library to determine a work type, a work area attribute and a work environment state of a current scene; Based on the work type, the work area attribute and the work environment state, a target safety rule set is called from a safety specification library; According to the target safety rule set, logical correlation analysis is performed on the behavior action sequence features, the spatial activity trajectory features and the environmental event features to determine whether there is behavior intention abnormality, spatial displacement abnormality or environmental change abnormality, and an abnormality analysis result is obtained; Based on the abnormality analysis result, a corresponding warning signal is generated.

2. The method of claim 1, wherein, The step of matching the behavior action sequence features, the spatial activity trajectory features and the environmental event features with the work scene feature library to determine the work type, the work area attribute and the work environment state of the current scene specifically comprises the following steps: Matching the behavior action sequence features with a preset action feature template to obtain a preliminary work type, comparing the spatial activity trajectory features with preset region boundary information to obtain a preliminary region attribute, and matching the environmental event features with a preset environmental feature template to obtain a preliminary environmental state; Based on the preliminary work type, the preliminary region attribute and the preliminary environmental state, a scene feature vector is constructed, and the scene feature vector is input into the work scene feature library for multi-dimensional matching to obtain a candidate scene set; Scoring each candidate scene in the candidate scene set to obtain a matching score; The candidate scene with the highest matching score is selected, the work type corresponding to the candidate scene is determined as the work type of the current scene, the region attribute corresponding to the candidate scene is determined as the work area attribute of the current scene, and the environmental state corresponding to the candidate scene is determined as the work environment state of the current scene.

3. The method of claim 1, wherein, The step of performing logical correlation analysis on the behavior action sequence features, the spatial activity trajectory features and the environmental event features according to the target safety rule set to determine whether there is behavior intention abnormality, spatial displacement abnormality or environmental change abnormality, and obtaining an abnormality analysis result specifically comprises the following steps: From the target safety rule set, a standard action sequence template corresponding to the work type, a safe activity region boundary corresponding to the work area attribute, and a normal environmental event range corresponding to the work environment state are obtained; The behavior action sequence features are compared with the standard action sequence template in time sequence, and when it is detected that the action sequence does not conform to the action combination relationship or the execution order in the standard template, it is determined that there is behavior intention abnormality. The spatial activity track feature is compared with the safety activity area boundary in real time, and when it is detected that a track point exceeds the safety boundary, it is determined that there is a spatial displacement anomaly; The environmental event feature is matched with the normal environmental event range, and when it is detected that the type, intensity or frequency of the environmental event exceeds the preset threshold range, it is determined that there is an environmental change anomaly; According to the determination results of the behavior intention anomaly, the spatial displacement anomaly and the environmental change anomaly, the abnormal analysis result is generated.

4. The method of claim 2, wherein, The step of scoring each candidate scene in the candidate scene set specifically includes: Analyze the change trend of the behavior action sequence feature, the spatial activity track feature and the environmental event feature in the time dimension, identify the key time sequence segments in each feature, and the key time sequence segments represent the state transition process in which the feature changes significantly within a preset time period; Retrieving historical scene data similar to the work type in a historical scene sample library, filtering the historical scene data based on the key time sequence segments, and determining the influence weight of each feature on scene recognition; The key time sequence segments are dynamically matched with the corresponding features of the candidate scene, the state transition similarity of each time sequence segment is calculated, and the weight coefficient of each time sequence segment is determined according to the importance of the state transition; The state transition similarity of each key time sequence segment is multiplied by the corresponding weight coefficient and then added to obtain the matching score of the candidate scene.

5. The method of claim 1, wherein, After the step of generating a corresponding early warning signal based on the abnormal analysis result, the method further includes: Obtaining real-time detection data of a plurality of heterogeneous sensors on the same monitoring target on a construction site, identifying the conflict state between the detection results of different sensors in the detection data, and the conflict state refers to the state judgment of different sensors on the same physical event at the same time. Extracting the features of the conflict state and classifying them into predefined conflict patterns, the features including conflict duration, sensor combination in conflict and change trend of each sensor detection value; According to the feature attributes of the conflict pattern and the detection mechanism of each sensor, determine the key abnormal factors corresponding to the conflict state; Correlate the key abnormal factors with the abnormal analysis result to obtain the credibility of the early warning signal.

6. The method of claim 5, wherein, The step of correlating the key abnormal factors with the abnormal analysis result to obtain the credibility of the early warning signal specifically includes: Determine the weight coefficient corresponding to the key abnormal factor, the weight coefficient is related to the duration of the conflict state and the number of sensors involved in the conflict; Calculate the abnormal similarity between the key abnormal factor and the abnormal analysis result, the abnormal similarity is determined based on the corresponding relationship between the abnormal occurrence position and the abnormal degree; According to the weight coefficient and the abnormal similarity, the credibility of the early warning signal is calculated.

7. The method of claim 6, wherein, After the step of calculating the credibility of the early warning signal according to the weight coefficient and the abnormal similarity, the method further includes: Determine the handling priority of the abnormal event based on the credibility and the type of the early warning signal, and generate emergency handling instructions according to the handling priority.

8. A safety warning system, characterized in that The safety warning system comprises one or more processors and a memory; the memory is coupled with the one or more processors, the memory is configured to store computer program codes, the computer program codes comprise computer instructions, and the one or more processors invoke the computer instructions to enable the safety warning system to perform the method according to any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, The instructions, when running on the safety warning system, enable the safety warning system to perform the method according to any one of claims 1-7.

10. A computer program product, characterised in that, The computer program product, when running on the safety warning system, enables the safety warning system to perform the method according to any one of claims 1-7.

Images