Multifunctional safety helmet background management method and system based on multi-source heterogeneous data fusion

Abstract

The invention discloses a multifunctional safety helmet background management method and system based on multi-source heterogeneous data fusion, and the method comprises the steps: receiving data uploaded by at least one multifunctional safety helmet terminal, the method further comprises the following steps: performing preprocessing including format conversion, data cleaning, abnormal value elimination and standardization on multi-source heterogeneous data uploaded by the multifunctional safety helmet terminal, wherein the multi-source heterogeneous data comprises two or more of position data, physiological sign data, environmental perception data and audio and video data; carrying out feature layer fusion on the preprocessed multi-source heterogeneous data by adopting a multi-modal fusion algorithm based on an attention mechanism to obtain a fusion data feature vector; executing at least one of the following management operations based on the fusion data feature vector: monitoring a security risk index in the fusion data, and triggering early warning through multi-dimensional analysis; analyzing the behavior pattern of the wearer, and identifying illegal work behaviors; and carrying out full-link tracing and audio and video playback in the operation process based on the fused data.

Description

Technical Field

[0001] This invention relates to the field of industrial operation safety management technology, specifically to a multifunctional safety helmet back-end management method and system based on multi-source heterogeneous data fusion. Background Technology

[0002] With the development of smart wearable devices and IoT technology, multi-functional safety helmets have been widely used in high-risk industrial scenarios such as construction, mining, and chemical industries. Their terminals are equipped with positioning modules, physiological sensors, environmental sensors, and audio / video acquisition units, enabling real-time collection of the wearer's location data (GPS / BeiDou latitude and longitude), physiological data (heart rate, blood oxygen), environmental perception data (temperature, gas concentration), and audio / video data, providing multi-dimensional support for workplace safety management. However, existing back-end management systems still have the following significant shortcomings in processing multi-source heterogeneous data and implementing intelligent management functions: (1) Insufficient processing capability for multi-source heterogeneous data The data collected by the multi-functional safety helmet terminal suffers from problems such as scattered sources, heterogeneous formats, and inconsistent quality; the preprocessing module of the existing system lacks targeted cleaning, anomaly removal, and standardization strategies.

[0003] (2) The data fusion method is not effective. Existing systems often employ simple splicing and shallow feature-level fusion methods to fuse multi-source data, failing to fully capture the spatiotemporal correlations and scene dependencies between modalities. For example, traditional methods do not utilize attention mechanisms to identify the correlation between physiological abnormalities and hazardous environmental areas, or the matching relationship between location changes and audio-visual scenes. This results in fused feature vectors that cannot accurately reflect the complex state of real-world work scenarios, directly impacting the accuracy of subsequent intelligent analysis.

[0004] (3) The core application functions have limitations. Low accuracy of safety warnings: Relying on single-dimensional data (such as only physiological abnormalities or environmental exceedances) to trigger warnings is prone to producing invalid warnings and cannot identify complex risks with multiple factors (such as high-altitude work area + sudden increase in heart rate + ambient temperature exceeding the standard). Insufficient reliability of behavior recognition: The recognition of violations (such as not wearing a safety helmet or entering a prohibited area) relies heavily on single-modal data (such as visual recognition alone), which has blind spots or misjudgments and lacks multi-source collaborative verification; The efficiency of job tracing is low: multi-source data is stored in a scattered manner, and a unified time correlation mechanism has not been established. It is impossible to quickly locate the audio and video segments and multi-dimensional data corresponding to the event, and there is a lack of data integrity verification, which makes it difficult to guarantee the authenticity of the tracing results. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the present invention aims to provide a multi-functional safety helmet backend management method and system based on multi-source heterogeneous data fusion. Through targeted multi-source data preprocessing (format conversion, cleaning, anomaly removal, standardization), feature layer fusion based on attention mechanism, and multi-dimensional intelligent applications (early warning, behavior recognition, full-link traceability), the method improves the accuracy and efficiency of industrial operation safety management and meets the reliable monitoring needs for operation safety in high-risk scenarios.

[0006] To solve the above problems, the technical solution adopted by the present invention is as follows: A backend management method for a multi-functional safety helmet based on multi-source heterogeneous data fusion includes the step of receiving data uploaded by at least one multi-functional safety helmet terminal, and the method further includes: The multi-source heterogeneous data uploaded by the multi-functional safety helmet terminal is preprocessed, including format conversion, data cleaning, outlier removal and standardization. The multi-source heterogeneous data includes two or more of the following: location data, physiological sign data, environmental perception data and audio and video data. A multimodal fusion algorithm based on an attention mechanism is used to fuse preprocessed multi-source heterogeneous data at the feature layer to obtain a fused data feature vector; Perform at least one of the following management operations based on the feature vector of the fused data: Monitor security risk indicators in fused data and trigger early warnings through multi-dimensional analysis; Analyze wearers' behavioral patterns to identify violations of operating procedures; Full-chain traceability and audio / video playback of the operation process based on fused data.

[0007] Preferably, data cleaning includes: Abnormal jump points in location data are detected and removed based on spatiotemporal continuity rules, and missing data is filled in using linear interpolation. Filter out abnormal values ​​that exceed the normal range threshold of physiological parameters, and use a sliding window weighted mean to smooth and correct continuous abnormal segments; Zero-point drift error is eliminated by using a sensor calibration model, and outliers are removed by combining multi-sensor cross-validation. Static invalid frames are removed using the frame difference method, silent segments are filtered out using the audio energy threshold, and the audio and video streams are time-stamped and synchronized. Heterogeneous data with different sampling frequencies are unified into a preset timestamp sequence, and interpolation and downsampling are used to achieve time dimension consistency alignment.

[0008] Preferably, outlier removal includes: When an abnormal jump in location data is detected, the abnormal jump point is retained or removed by combining the stress response in physiological sign data or the regional characteristics in environmental perception data. When physiological data exceeds the preset normal range threshold, the location data is combined to determine whether the wearer is in a preset high-risk scenario or whether there are extreme conditions in the environmental data, thereby retaining or filtering out the abnormal values. When environmental data is flagged as abnormal, the abnormal value is retained or removed by combining it with similar data from other terminals in the same work area during the same period.

[0009] Preferably, standardization includes: The absolute coordinates are converted to relative coordinates of the work area and then normalized to the [0,1] interval using min-max. The wearer's historical physiological data mean and standard deviation are used as the personal baseline, and the typical physiological range of the group in the current scene is used as the scene baseline. Different baselines are used for z-score transformation according to the scene. The relative deviation rate of single-terminal data is calculated based on the regional terminal collaborative benchmark and normalized to the range of [-1,1]. If the number of terminals in the region is ≤3, the industry preset benchmark value is automatically switched. The extracted cross-modal features are L2 normalized to unit vectors, and then cross-modal joint standardization is performed after timestamp alignment. Perform correlation verification on the standardized data of each type.

[0010] Preferably, when performing format conversion, it includes: The original latitude and longitude coordinates output by the positioning module are converted into coordinate values ​​in the local plane coordinate system specific to the work area, and encapsulated into GeoJSON Feature structured format. At the same time, the continuous location data sequence is converted into trajectory line segment format with time interval. Convert the raw sensor sampling values ​​to the HL7 FHIR Observation medical standard format; Convert the raw sensor signal values ​​into a standardized format of physical quantity units, encapsulate them into JSON key-value pairs, and add an index field for related data from other terminals in the same area; The original audio and video streams are converted into the MP4 unified container format. At the same time, the CNN visual features of the video frames and the MFCC acoustic features of the audio segments are extracted, and after associating the timestamps and scene association tags, they are stored in Protobuf format and an audio and video time synchronization index table is generated.

[0011] Preferably, when obtaining the fused data feature vector, the following steps are included: The preprocessed location trajectory features, physiological sign sequence features, environmental perception vectors, and audio / video multimodal features are mapped to a high-dimensional embedding space. The temporal dependencies of features across different modalities are captured through a self-attention layer; Intermodal correlation weights are calculated through a cross-modal attention layer; A unified fusion data feature vector is generated by weighted summation of features across modalities based on cross-modal weights.

[0012] Preferably, when an alert is triggered through multidimensional analysis, it includes: The system analyzes the correlation between physiological signs, environmental perception, and location dimensions in the fused data feature vector in real time, and triggers an alert when at least two of the following conditions are met and the duration exceeds a preset time: Abnormal physiological signs and characteristics were observed; Environmental perception features indicate that parameters are out of range; Location characteristics indicate that the area is in a pre-defined high-risk work zone.

[0013] Preferably, when identifying non-compliant work behavior, the following are included: By combining CNN visual features extracted from audio and video data, abnormal contact status from physiological sign data, and head exposure parameters from environmental perception data, multi-source correlation verification is performed to determine whether there is any violation of not wearing a safety helmet. When location data shows that the wearer has entered a preset prohibited work area, a collaborative judgment is made by combining environmental perception data, visual recognition results of area markers from audio and video data, and stress response data from physiological signs. For pre-defined violations, a comprehensive judgment is made by integrating visual features of audio and video data, abnormal changes in environmental perception data, and regional confirmation of location data.

[0014] Preferably, when performing full-link tracing and audio / video playback of the operation process based on fused data, it includes: Based on the globally unique timestamps and audio-visual time synchronization index table of the fused data, various events are accurately associated with audio and video segments at corresponding time points, supporting quick location and playback of associated audio and video content by event trigger time; The full-link traceability interface synchronously displays the timeline of key events in the operation process, dynamic location trajectory map, physiological sign trend curve, environmental parameter change chart, and links the audio and video playback window. It supports filtering traceability content by work area ID, equipment ID, event type, or time interval, and automatically aggregates multi-source heterogeneous data and related audio and video clips for the corresponding time period to generate a structured work traceability report. Before audio and video playback, the integrity of the associated multi-source data is verified by checking the original data hash value.

[0015] A multi-functional safety helmet backend management system based on multi-source heterogeneous data fusion includes a backend server communicatively connected to at least one multi-functional safety helmet terminal, a data storage module for storing data, and the system further includes: Multi-source heterogeneous data preprocessing module: used to perform format conversion, data cleaning, outlier removal and standardization on multi-source heterogeneous data uploaded by the multi-functional safety helmet terminal. Multi-source heterogeneous data includes two or more of the following: location data, physiological sign data, environmental perception data and audio and video data. Intelligent data fusion module: It adopts a multimodal fusion algorithm based on attention mechanism to fuse preprocessed multi-source heterogeneous data at the feature layer to generate a unified fused data feature vector; Multifunctional application module: Based on fused data feature vectors, it can achieve at least one of the following functions: real-time early warning of safety risks, compliance analysis of wearer behavior, and full-link traceability and playback of the operation process; The multi-functional safety helmet terminal is equipped with sensor components and wireless communication modules for heterogeneous data acquisition.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Precision and scenario-based design of multi-source heterogeneous data preprocessing Type-specific processing: Based on the inherent characteristics of data such as location, physiological signs, environmental perception, and audio and video, customized cleaning rules are developed (such as spatiotemporal continuity detection for location data, sliding window smoothing for physiological data, multi-sensor cross-validation for environmental data, and invalid frame filtering for audio and video), which solves the problem of inconsistent data quality and improves data reliability.

[0017] Multi-source correlation outlier removal: Breaking away from single-rule judgment, it distinguishes between real anomalies and noise through cross-modal data verification (such as location jump combined with physiological stress response, environmental anomalies combined with regional terminal consistency), avoiding the accidental deletion of valid data or the retention of interfering information, and providing a high-quality foundation for subsequent analysis.

[0018] Scenario-adaptive standardization: Adopting a dual baseline strategy of "personal baseline + scenario baseline", parameters are dynamically updated (such as adjusting the physiological baseline when switching scenarios), unifying the scale of heterogeneous data, adapting to complex work scenarios, and ensuring the accuracy of fused features.

[0019] (2) Multimodal feature layer fusion algorithm based on attention mechanism This algorithm integrates intramodal temporal dependencies and cross-modal associations: It captures dynamic changes in single-modal data (such as location trajectory trends and physiological fluctuations) through a self-attention layer, and calculates intermodal association weights (such as the correlation between physiological abnormalities and environmental risks) through a cross-modal attention layer, generating a high-dimensional fusion vector. Compared to traditional splicing or shallow fusion, this algorithm more accurately reflects the complex states of real-world work scenarios and improves feature representation capabilities.

[0020] (3) Intelligent upgrade of core application functions Multi-dimensional collaborative early warning: It adopts the rule of "at least two dimensions trigger (physiological + environmental / location, etc.) + duration" to solve the problems of misjudgment in single dimensions and omission of compound risks, and significantly improve the accuracy of early warning; Multimodal behavior recognition: By collaboratively verifying violations (such as not wearing a safety helmet or entering a prohibited area) using multi-source data such as visual (audio and video CNN features), physiological (abnormal contact status), and environmental (head exposure parameters), the system reduces missed and false judgments. End-to-end trusted traceability: Based on global timestamps, multi-source data and audio / video clips are associated, and hash verification is combined to ensure that the data is tamper-proof, realizing integrated traceability of events, data and audio / video, supporting accident analysis and responsibility determination.

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the steps of the multifunctional safety helmet backend management method according to an embodiment of the present invention; Figure 2 This is a flowchart of the fusion data feature vector generation process according to an embodiment of the present invention; Figure 3 This is an interactive diagram of the multifunctional safety helmet backend management system module according to an embodiment of the present invention.

[0023] The following are the reference numerals: 20. Multifunctional safety helmet backend management system; 21. Multifunctional safety helmet terminal; 211. Sensor component; 212. Wireless communication module; 22. Backend server; 23. Data storage module; 24. Multi-source heterogeneous data preprocessing module; 25. Intelligent data fusion module; 26. Multifunctional application module. Detailed Implementation

[0024] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0025] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0026] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0027] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0028] Example 1, see Figure 1 The present invention provides a step-by-step diagram of a multifunctional safety helmet back-end management method. Figure 1 The method for backend management of a multi-functional safety helmet based on multi-source heterogeneous data fusion, as shown, includes the step of receiving data uploaded by at least one multi-functional safety helmet terminal 21. The method further includes: S1. Perform preprocessing on the multi-source heterogeneous data uploaded by the multi-functional safety helmet terminal 21, including format conversion, data cleaning, outlier removal and standardization. The multi-source heterogeneous data includes two or more of the following: location data, physiological sign data, environmental perception data and audio and video data. S2. A multimodal fusion algorithm based on an attention mechanism is used to fuse the preprocessed multi-source heterogeneous data at the feature layer to obtain the fused data feature vector. S3. Perform at least one of the following management operations based on the fused data feature vector: S31. Monitor safety risk indicators (such as abnormal heart rate, excessive ambient gas) in the fused data, and trigger early warnings through multi-dimensional analysis; S32. Analyze the wearer's behavior patterns to identify violations of operating procedures (such as not wearing a safety helmet or entering a dangerous area). S33. Full-link traceability and audio / video playback of the operation process based on fused data.

[0029] Background Description: The location, physiological signs, environmental perception, and audio / video data collected by the multi-functional safety helmet terminal 21 suffers from diverse sources, heterogeneous formats, and inconsistent quality (such as sensor interference, signal drift, and sampling frequency differences), directly affecting the accuracy of subsequent data fusion and intelligent analysis. Based on this: In step S1 above, during data cleaning, specific processing is performed for different types of heterogeneous data, including: Location data cleaning: Based on the spatiotemporal continuity rule, abnormal jumps in location data collected by the positioning module (GPS module or Beidou module) are detected (such as the distance between adjacent sampling points exceeding the preset distance threshold T1 and the time interval being less than T2). Abnormal jump points are removed, and missing data is filled in using linear interpolation. Physiological data cleaning: Set normal range thresholds for physiological parameters (e.g., heart rate 60-180 beats / minute), filter out abnormal values ​​that exceed the thresholds, and use sliding window weighted average to smooth and correct continuous abnormal segments; Environmental perception data cleaning: Zero-point drift error is eliminated by sensor calibration model, and outliers are removed by multi-sensor cross-validation (e.g., data deviation between adjacent temperature and humidity sensors exceeding 5% is marked as abnormal). Audio and video data cleaning: Static invalid frames (such as 10 consecutive frames without change) are removed using the frame difference method, silent segments are filtered out by the audio energy threshold, and the audio and video streams are time-stamped and synchronized. Multi-source time alignment: Unify heterogeneous data with different sampling frequencies into a preset timestamp sequence (e.g., once per second), and use interpolation and downsampling to achieve time dimension consistency alignment.

[0030] In this embodiment of the invention, it is necessary to further explain that the specific processing of data cleaning is aimed at the inherent characteristics and collection pain points of various types of data. It improves data quality through targeted technical means, providing a reliable data foundation for the system's core functions (security alerts, behavior recognition, and end-to-end traceability). The following details the purpose, technical principles, and system function of each step in detail: 1. Location data cleaning Objective: To resolve abnormal position jumps (such as instantaneous jumps from area A to area B, which do not conform to normal movement logic) caused by signal obstruction and electromagnetic interference in GPS / BeiDou positioning modules, and to ensure the spatiotemporal continuity of trajectory data.

[0031] Technical principle: Spatiotemporal continuity rule: Detect the distance between adjacent sampling points (exceeding the preset threshold T1) and the time interval (less than T2) to determine whether there is an abnormal jump; Anomaly handling: After removing jump points, use linear interpolation to fill in the missing data (filling based on the coordinate trend of the valid points before and after).

[0032] System function: To provide an accurate location basis for subsequent identification of prohibited area entry and trajectory tracing, and to avoid misjudgment of violations or trajectory breaks due to location errors.

[0033] 2. Physiological data cleaning Objective: To filter out temporary interference from sensors (such as poor contact of heart rate sensors) or outliers that do not reflect real physiological changes, and to retain real physiological response data in high-risk scenarios.

[0034] Technical principle: Threshold filtering: Set the normal range for physiological parameters (such as heart rate 60-180 beats / minute) and directly filter out isolated abnormal values ​​that exceed the threshold; Smoothing correction: For continuous abnormal segments (such as continuous fluctuations in sensor data), a sliding window weighted average (with higher weight for recent data) is used to smooth noise and preserve the trend.

[0035] Function: To provide reliable data for early warning of physiological abnormalities (such as a sudden increase in heart rate during high-altitude work), reduce invalid warnings caused by sensor errors, and improve the accuracy of warnings.

[0036] 3. Environmental perception data cleaning Objective: To eliminate outliers caused by sensor errors (such as zero drift) and environmental interference, and to ensure the accuracy of environmental parameters (temperature, gas concentration, etc.).

[0037] Technical principle: Calibration to eliminate drift: Correct zero-point offset (such as baseline error of temperature and humidity sensors) through sensor calibration model. Cross-validation to remove outliers: Compare similar data from adjacent terminals (e.g., mark as abnormal if the deviation exceeds 5%) to determine if it is local interference.

[0038] Function: To provide accurate basis for early warning of environmental parameters exceeding the standard (such as excessively high concentration of combustible gas), and to support subsequent cross-terminal consistency verification (such as determining whether the regional environment is truly abnormal).

[0039] 4. Audio and video data cleaning Objective: To reduce the storage and computational burden of invalid data, synchronize audio and video timestamps, and ensure the temporal consistency of multimodal data.

[0040] Technical principle: Static frame filtering: Identify and delete consecutive static frames that do not change (e.g., 10 frames without difference) using the frame difference method; Silent segment filtering: Filters out invalid audio based on an audio energy threshold (audio below the threshold is considered silent); Timestamp calibration: unifies the timestamps of audio and video streams to ensure audio and video synchronization.

[0041] Function: To provide efficient and synchronized audio and video data for multimodal behavior recognition (such as visual detection of not wearing a safety helmet, audio and video evidence of violations), while reducing storage costs.

[0042] 5. Multi-source time alignment Objective: To solve the problem of inconsistent sampling frequencies for different types of data (e.g., location once per second, physiological data five times per second) and achieve uniformity of all data in the time dimension.

[0043] Technical principle: Uniform timestamp sequence: Maps all data to a preset timestamp (e.g., once per second). Sampling adjustment: Downsample high-frequency data (such as physiological data) and interpolate low-frequency data (such as environmental data) to ensure consistency in the time dimension.

[0044] Function: To provide a foundation for multi-source association verification (such as combining physiological data to judge when the location is abnormal) and feature layer fusion (based on time-aligned multimodal feature vectors), ensuring that different data types can be analyzed at the same time point.

[0045] In summary, specific data cleaning is a crucial bridge connecting terminal data collection and intelligent analysis: by addressing the quality issues of various types of data in a targeted manner, high-quality and consistent data is output, ensuring that subsequent attention-based multimodal fusion algorithms can accurately extract features, ultimately supporting the reliability and accuracy of the system's three core functions: real-time early warning of safety risks, compliance analysis of wearer behavior, and full-chain traceability of the operation process.

[0046] Background Description: The multi-source heterogeneous data collected by the multi-functional safety helmet terminal 21, including location, physiological signs, and environmental perception, is susceptible to sensor interference (such as GPS signal obstruction), equipment errors (such as poor contact of the heart rate sensor), or environmental fluctuations, resulting in false anomalies. Traditional single-data rules (such as judging location anomalies solely based on spatiotemporal continuity) struggle to distinguish between "real anomalies" and "noise interference," leading to misjudgments in subsequent analysis (such as mistakenly deleting genuine emergency evacuation location data or falsely reporting physiological anomalies caused by temporary sensor interference), affecting the accuracy of system warnings and the reliability of behavior recognition. Therefore, it is necessary to establish a collaborative elimination mechanism through the correlation and verification of multi-source data to overcome the limitations of single rules. Based on this: In step S1 above, outlier removal employs a multi-source correlation verification-based collaborative outlier removal mechanism, including: Contextualized verification of location data anomalies: When anomaly changes in location data are detected through spatiotemporal continuity rules, further verification is performed by combining stress responses in physiological signs data (e.g., heart rate increases by ≥30% compared to the average of the previous 5 seconds) or regional characteristics in environmental perception data (e.g., gas concentration suddenly exceeds the standard). If a corresponding stress response or change in regional characteristics exists, the jump is determined to be caused by actual operational movement, and the data point is retained; otherwise, the abnormal jump point is removed, and the missing data is supplemented using linear interpolation. Scene correlation verification of physiological data: When physiological data exceeds the preset normal range threshold, the location data is combined to determine whether the wearer is in a preset high-risk scene (such as a high-altitude work area or a flammable and explosive area) or whether there are extreme conditions in the environmental data (such as a temperature ≥45℃); if the wearer is in the above scene and / or conditions, the abnormal value is retained as a basis for warning; otherwise, it is determined to be temporary interference of the sensor and the abnormal value is filtered out. Cross-terminal consistency verification of environmental sensing data: When environmental data is marked as abnormal, obtain the same type of data from other terminals in the same work area during the same period; if more than 70% of the terminals show the same abnormality, it is determined that the actual environment of the area has changed and the data is retained; otherwise, the abnormal value is removed.

[0047] In this embodiment of the invention, it is necessary to further explain that this mechanism determines whether outliers are valid data in the real-world scenario, rather than noise, through cross-modal data correlation verification. Specifically, it includes three main verification logics: (1) Scenario-based verification of location data anomalies: When location data shows abnormal jumps (such as jumps detected by spatiotemporal continuity rules), it is verified in conjunction with physiological signs (such as a heart rate increase of ≥30% compared to the average of the previous 5 seconds) or environmental perception (such as a sudden exceedance of gas concentration). If there is a corresponding stress response or change in regional characteristics, it is determined to be a real operational movement (such as emergency evacuation), and the data is retained; otherwise, it is discarded and supplemented.

[0048] (2) Scene correlation verification of physiological signs data: When physiological data exceeds normal thresholds, location data is used to determine whether the location is in a high-risk scenario (such as a high-altitude work area) or under extreme environmental conditions (such as a temperature ≥45℃). If the location is in one of these scenarios, the outlier is retained as a basis for early warning; otherwise, it is determined to be sensor interference, and the data is filtered.

[0049] (3) Cross-terminal consistency verification of environmental perception data: When a single terminal displays abnormal environmental data, it is compared with the concurrent data from other terminals within the same work area. If more than 70% of the terminals show the same abnormality, it is determined that there has been a change in the actual environment of the area, and the data is retained; otherwise, it is discarded (considered as a single sensor error).

[0050] Core logic: Leverage the contextual correlations between multiple data sources to corroborate each other and avoid misjudgments based on a single data dimension.

[0051] The function of this mechanism: (1) Improve data quality: Through collaborative verification, we can effectively distinguish between real anomalies and noise, reduce invalid data from entering subsequent processes, and ensure that the preprocessed data is more reliable.

[0052] (2) Supporting accurate analysis: providing high-quality basic data for multimodal fusion algorithms based on attention mechanism, making the fusion feature vector more accurate, and improving the accuracy of safety warnings (such as triggering warnings only when physiological abnormalities occur in high-risk areas) and behavior recognition (such as judging illegal entry by combining location abnormalities with physiological reactions).

[0053] (3) Reduce invalid operations: Avoid invalid warnings or erroneous tracing caused by false anomalies, reduce system operation and maintenance costs, and enhance user trust.

[0054] This mechanism solves the problem of misjudgment due to single rules through multi-source association verification, providing a solid data foundation for the system's subsequent fusion analysis and application functions. It is an important technical means to improve the reliability of the multi-functional safety helmet backend management system 20.

[0055] Background Description: The location, physiological signs, environmental perception, and audio / video data collected by the multi-functional safety helmet terminal 21 suffers from heterogeneous sources, inconsistent scales, and strong scene dependence. Location data: Differences in coordinate systems (such as WGS84 and custom engineering coordinate systems) between different work areas make it impossible to directly compare absolute coordinates; Physiological data: Individual differences (such as different baseline heart rates) and scenario differences (such as heart rate generally increases when working at heights) coexist, making it difficult to adapt to a single standardized method; Environmental perception data: Errors in terminal sensors or changes in the scene within the region can easily lead to data deviations, and there is a lack of regional collaborative benchmarks; Audio and video features: The audio MFCC coefficients and video HOG vectors have different dimensions, resulting in inconsistent scales during cross-modal fusion; Dynamic scene changes: When workers enter / exit high-risk areas, the data distribution will change significantly, and static standardized parameters cannot be adapted in real time.

[0056] Traditional static standardization methods (such as z-score with a fixed mean / standard deviation) cannot solve the above problems, leading to low input data quality and cross-modal correlation failure in subsequent multimodal fusion algorithms, thus affecting the accuracy of system warnings and behavior recognition. Therefore: In step S1 above, during standardization, a scenario-adaptive multi-source collaborative standardization strategy is adopted, including: Class-specific normalization algorithm: Location data: Convert absolute coordinates to relative coordinates of the work area (with the boundary of the work area as the reference system, such as setting the origin as the center point of the area), and normalize to the [0,1] interval through min-max to eliminate the difference in coordinate systems between different work areas; Physiological data: Dual baseline dynamic standardization is adopted: the mean and standard deviation of the wearer's historical physiological data over the past 30 days are used as the personal baseline, and the typical physiological range of the current scene (such as a high-altitude work area) is used as the scene baseline; if the wearer is in a high-risk scene, the scene baseline is used first for z-score transformation (x'=(x-scene mean) / scene standard deviation), otherwise the personal baseline is used for z-score transformation. Environmental perception data: Based on the regional terminal collaborative benchmark (using the median of the environmental data of all effective terminals in the same working area as the benchmark value), the relative deviation rate of single terminal data is calculated ((x-regional median) / regional median × 100%) and normalized to the range of [-1,1]; if the number of terminals in the area is ≤3, the industry preset benchmark value is automatically switched. Audio and video features: The extracted cross-modal features (such as audio MFCC coefficients and video keyframe HOG vectors) are L2 normalized to unit vectors, and then cross-modal joint normalization is performed after timestamp alignment (global min-max normalization is performed after splicing audio and video features). Dynamic parameter update mechanism: Every 5 minutes or when the scene changes (such as entering / exiting a flammable and explosive area), the scene baseline of physiological data and the regional terminal collaborative benchmark of environmental data are automatically updated to ensure that standardized parameters match the real-time operation status. Multi-source standardized consistency verification: The standardized data of each type are verified for correlation. For example, if the location data shows that the wearer is in a high-risk area, but the standardized value of the physiological data does not fall within the scene warning range, the physiological data standardization process is re-executed (switching to the scene baseline). If the standardized value of the environmental data deviates from that of most terminals in the same area by more than 20%, the environmental sensor calibration prompt of the terminal is triggered.

[0057] In this embodiment of the invention, it is necessary to further explain that the strategy of this embodiment achieves the accuracy and consistency of data standardization through three major mechanisms: type-based customization, dynamic updating, and cross-modal verification. (1) Class-specific standardization algorithm Location data: Convert absolute coordinates to relative coordinates of the work area (with the area boundary as the reference system), and normalize to the [0,1] interval through min-max to eliminate differences in the area coordinate system; Physiological data: Dual baseline dynamic standardization is adopted - the wearer's historical data over the past 30 days is used as the personal baseline, and the group data of the current scenario (such as a high-altitude work area) is used as the scenario baseline; when in a high-risk scenario, the scenario baseline is used first, otherwise the personal baseline is used for z-score transformation; Environmental perception data: Based on the median of the data of all terminals in the same working area during the same period as the collaborative benchmark, the relative deviation rate of the data of a single terminal is calculated and normalized to the range of [-1,1]; if the number of terminals in the area is ≤3, the industry preset benchmark value is switched. Audio and video features: First, L2 normalize the audio MFCC coefficients and video HOG vectors to unit vectors, and then perform cross-modal joint normalization (global min-max normalization after splicing) after timestamp alignment.

[0058] (2) Dynamic parameter update mechanism Every 5 minutes or when switching scenes (such as entering a flammable and explosive area), the scene baseline of physiological data and the regional collaborative benchmark of environmental data are automatically updated to ensure that the parameters match the real-time operation status.

[0059] (3) Multi-source standardization consistency verification Cross-modal verification of data correlation: For example, if location data shows that the wearer is in a high-risk area, but physiological data does not use the scenario baseline, the physiological standardization process is re-executed; if environmental data deviates from most terminals in the same area by more than 20%, a terminal sensor calibration prompt is triggered.

[0060] The effect of this strategy: (1) Unified data scale: Eliminate the scale difference of heterogeneous data, make location, physiological, environmental and audio-visual data comparable in the same dimension, provide high-quality input for attention-based multimodal fusion algorithms, and improve the accuracy of fusion feature vectors; (2) Adapt to dynamic scenarios: Dynamically update the baseline to ensure that standardized parameters are adjusted with changes in the scenario, so that physiological, environmental and other data are more in line with the actual operation scenario requirements and avoid deviations caused by static standardization; (3) Ensure data consistency: Correct standardized errors through cross-modal verification, reduce invalid data entering subsequent processes, and improve the accuracy and reliability of safety warnings (such as accurate triggering of physiological abnormalities in high-risk scenarios), behavior recognition (such as scenario-based judgment of entry into prohibited areas), and end-to-end traceability (data consistency supports reliable event association). (4) Optimize system performance: reduce invalid warnings or misjudgments caused by standardization errors, reduce system operation and maintenance costs, and improve user trust.

[0061] This strategy is a key link connecting data preprocessing and intelligent analysis, directly supporting the effectiveness of the system's core functions and providing a reliable standardized data foundation for subsequent fusion algorithms and application scenarios.

[0062] Background Description: The location, physiological characteristics, environmental perception, and audio / video data collected by the multi-functional safety helmet terminal 21 suffers from problems such as scattered sources, heterogeneous formats, and inconsistent units (e.g., location data uses different coordinate systems, physiological data lacks a standard format, environmental data has inconsistent units, and audio / video formats are diverse). This directly results in the data being unable to be efficiently processed by subsequent preprocessing modules (cleaning, anomaly removal, and standardization), and also makes it difficult to support the input requirements of multi-modal fusion algorithms. Based on this: In step S1 above, the format conversion includes: Location data format conversion: The raw latitude and longitude coordinates output by the positioning module (GPS / BeiDou) (such as WGS84 coordinate system) are converted into coordinate values ​​under the local plane coordinate system (such as UTM partition or custom engineering coordinate system) specific to the work area, and encapsulated into a GeoJSON Feature structured format containing timestamp, device ID, coordinate precision, and work area label; at the same time, continuous location data sequences are converted into trajectory line segment format with time intervals, which facilitates spatiotemporal continuity rule detection during data cleaning, reduces the complexity of location calculation within the work area, and improves trajectory analysis efficiency; Physiological sign data format conversion: Convert the raw sensor sampling values ​​into the HL7 FHIR Observation medical standard format, including physiological parameter type (heart rate / blood oxygen, etc.), value, unit, sampling timestamp and scene association label (such as "high-altitude work area" "flammable and explosive area"); the scene association label is generated based on real-time location data, providing a direct basis for scene association verification when removing outliers and reducing invalid warnings; Environmental sensing data format conversion: Convert the raw sensor signal values ​​into a standardized format of physical quantity units (such as temperature ℃, humidity %RH, gas concentration ppm), and encapsulate them into JSON key-value pairs containing sensor ID, calibration status, region ID, timestamp, and normalized value; add an index field for related data from other terminals in the same region to support cross-terminal consistency verification when removing outliers and improve the efficiency of multi-sensor cross-verification. Audio and video data format conversion: The original audio and video streams are converted into the MP4 unified encapsulation format. At the same time, the CNN visual features of the video frames and the MFCC acoustic features of the audio segments are extracted, and the timestamps and scene-related tags (generated based on location / environment data) are associated and stored in Protobuf format. An audio and video time synchronization index table is generated for multi-source time alignment during data cleaning, ensuring accurate alignment of timestamps with other heterogeneous data, and providing time anchors for audio and video playback with full-link traceability.

[0063] In this embodiment of the invention, it is necessary to further explain that the spherical geographic coordinates (such as WGS84) are mapped to a planar coordinate system specific to the work area through coordinate system transformation, eliminating cross-regional coordinate differences. At the same time, a structured format is used to encapsulate the data, which facilitates subsequent spatiotemporal continuity detection and trajectory analysis. Among them, the GeoJSON Feature must contain complete metadata (such as coordinate accuracy for assessing data reliability); the time interval of the trajectory line segment must match the sampling frequency (such as 1 sampling per second corresponding to a 1-second interval line segment); the selection of the local planar coordinate system must be adapted to the scale of the work area (UTM for large areas, and a custom engineering coordinate system for small areas).

[0064] This embodiment adopts the medical industry standard format (HL7 FHIR) to unify the physiological data structure, and combines scene-related tags with real-time location information to provide a contextual basis for subsequent outlier removal. The original sensor sampling values ​​are mapped to the HL7FHIR Observation format, and scene-related tags (such as "high-altitude work area") are generated based on real-time location data and embedded in the Observation format. Specifically, the HL7 FHIR Observation must strictly adhere to resource specifications (e.g., parameter types use LOINC encoding); the scene-related tag generation logic must be linked to location data in real time (e.g., determined by the geofence of the work area); and numerical units must be standardized (e.g., heart rate uses "beats / minute", blood oxygen uses "%").

[0065] This embodiment converts the raw signal into physical quantity units through sensor calibration, encapsulates the data in structured JSON format, and associates it with the terminal index in the same region, supporting cross-terminal consistency verification. The calibration status needs to record the sensor's most recent calibration time (to ensure data reliability); the normalized value uses the min-max or z-score method (for easy subsequent standardization); the associated data index field is linked to the list of terminals in the same region in the database through the region ID.

[0066] This embodiment achieves data compression and storage, as well as cross-modal feature fusion, through audio and video coding standardization, feature extraction, and efficient serialization. Simultaneously, it generates a time-synchronized index table to ensure alignment of multi-source data. Specifically, the CNN visual feature extraction uses a pre-trained model (e.g., ResNet-50) with a fixed output dimension (e.g., 2048 dimensions); MFCC parameter settings are implemented (e.g., 13 coefficients, 26 Mel filters); the Protobuf format needs to optimize feature storage efficiency (e.g., using compressed encoding); and the accuracy of the time-synchronized index table must ensure time alignment with other data.

[0067] The above conversion steps, through standardized formats, structured encapsulation, and feature extraction, provide a unified and efficient input foundation for subsequent data cleaning, anomaly removal, standardization, and multimodal fusion, ensuring the compatibility of various modules in the system and the efficiency of data processing.

[0068] See Figure 2 The flowchart for generating the fused data feature vector is as follows: In step S2 above, obtaining the fused data feature vector includes: Modal feature embedding: The preprocessed location trajectory features (such as direction / velocity derived features of latitude and longitude conversion), physiological sign sequence features (such as heart rate sliding window statistics), environmental perception vectors (such as temperature / gas concentration standardized values), and audio and video multimodal features (such as CNN visual features and MFCC acoustic feature concatenation vectors) are respectively mapped to a high-dimensional embedding space; Intramodal temporal attention learning: Captures the temporal dependencies of features in each modality (such as the continuous movement trend of position trajectory and the changing trend of physiological signs) through a self-attention layer. Cross-modal association attention learning: Calculate the association weights between modalities (such as the association between physiological abnormalities and environmental features, and the association between location features and audio / video scene features) through a cross-modal attention layer. Global feature fusion: Based on cross-modal weights, the features of each modality are weighted and summed to generate a unified fused data feature vector.

[0069] In the embodiments of the present invention, it is necessary to further explain that (1) modal feature embedding Background: The preprocessed multi-source data (such as relative coordinates of location, physiological standardized heart rate, environmental deviation rate, and feature vectors of audio and video) have significant differences in dimensionality and spatial distribution, and cannot be directly used for cross-modal fusion.

[0070] Function: Maps features of each modality to the same high-dimensional embedding space, eliminates scale differences between modalities, makes different types of data comparable, and lays the foundation for subsequent attention learning.

[0071] (2) Intramodal temporal attention learning Background: Single-modal data (such as continuous sampling points of location trajectory, time series of heart rate) have strong temporal dependencies, and traditional feature extraction (such as static mean) will lose dynamic change information (such as sudden location jumps, sudden increases in heart rate).

[0072] Function: By capturing the temporal correlations within each modality (such as the trend of positional movement and the pattern of changes in physiological signs) through the self-attention layer, the features of key time points (such as positional changes during emergency evacuation and heart rate fluctuations in dangerous scenarios) are highlighted, thereby improving the dynamic expression of features.

[0073] (3) Cross-modal associative attention learning Background: Scene-driven associations exist between different modalities of data (such as the association between physiological abnormalities and dangerous areas in the environment, and the matching of location features with audio and video scenes). Traditional fusion methods (such as simple stitching) cannot quantify the importance of such associations.

[0074] Function: By calculating the correlation weights between modalities through cross-modal attention layers (such as the correlation between physiological abnormalities and excessive ambient gas levels, and the matching degree between location features and audio-visual scenes), strong correlation features are prioritized for retention (such as a sudden increase in heart rate in high-altitude work areas + excessive ambient temperature), while weak correlation noise is filtered out to improve the targeting of fused features.

[0075] (4) Global feature fusion Background: After the processing of the first three steps, each modal feature has a unified space, dynamic temporal sequence and correlation weights, but it is still in a scattered state and cannot be directly used by the application module.

[0076] Function: Based on cross-modal association weights, the features of each modality are weighted and summed to generate a single fusion data feature vector, which condenses multi-source information into a structured feature representation to support subsequent core functions such as safety risk warning (e.g., fusion of physiological + environmental + location features to judge danger), violation behavior identification (e.g., fusion of audio and video + physiological + location features to identify not wearing a safety helmet), and full-link traceability of operations (e.g., fusion of timestamp-related multimodal features to achieve event backtracking).

[0077] Background: Traditional safety early warning mechanisms have significant limitations in industrial operation scenarios: High false alarm rate of single-dimensional warnings: Relying on only one of the following to trigger a warning: physiological signs (such as isolated heart rate abnormalities), environmental parameters (such as gas concentration exceeding the standard at a single terminal), or location data (such as false alarms of entering a high-risk area), the system is prone to a large number of invalid warnings due to temporary sensor interference (such as poor contact of heart rate sensor), local environmental fluctuations (such as false gas alarms at a single terminal), or location jumps, thus reducing the reliability of the system.

[0078] Missing composite risk scenarios: Safety risks in real operations are often the result of multiple factors (such as working at height + sudden increase in heart rate + exceeding the standard ambient temperature). Single-dimensional early warning cannot capture such linked risks, leading to the omission of key hidden dangers.

[0079] Lack of scenario-related validation: The warning logic is not dynamically adjusted based on the work scenario. For example, heart rate fluctuations in normal areas may not require attention, but similar fluctuations in high-risk areas require an emergency response. Static rules are difficult to adapt to complex scenario requirements. Based on this: In step S31 above, when an early warning is triggered through multidimensional analysis, the following steps are included: The system analyzes the correlation between physiological characteristics (such as heart rate and blood oxygen), environmental perception (such as gas concentration and temperature), and location (such as whether the data is in a high-risk area) in the feature vector of the fused data in real time. An alert is triggered when at least two of the following conditions are met and the duration exceeds a preset duration (such as 5 seconds): Abnormal physiological signs (e.g., heart rate exceeding the scene adaptive baseline ±30%) Environmental sensing characteristics indicate that parameters exceed the standard (e.g., combustible gas concentration ≥ 20% of the lower explosive limit); Location characteristics indicate that the work is located in a pre-defined high-risk work area (such as a high-altitude work area or a flammable and explosive area).

[0080] In this embodiment of the invention, it is necessary to further explain that this embodiment integrates physiological, environmental, and location features through multidimensional analysis, and sets triggering rules that "meet at least two conditions and last for a preset duration." This aims to solve the accuracy problem of traditional early warning systems, achieve effective identification of real composite risks, and reduce the interference of invalid early warnings on operations. This mechanism relies on preprocessing of multi-source data (such as scene adaptive standardization and cross-modal time alignment) and feature layer fusion (multimodal association based on attention mechanisms) to ensure the correlation and consistency of data across all dimensions, providing reliable support for accurate early warning.

[0081] In step S32 above, identifying violations of operating procedures includes: multimodal collaborative identification of not wearing a safety helmet, scenario-based verification of entering prohibited areas, and cross-modal identification of violations of operating procedures. Multimodal collaborative recognition of helmet-wearing behavior, including: Multi-source correlation verification is performed by combining CNN visual features extracted from audio and video data (the matching degree of the helmet feature vector in the head area is lower than the preset threshold), abnormal contact status in physiological sign data (among which abnormal contact status is such as continuous absence of heart rate sensor signal for ≥5 seconds or fluctuation amplitude exceeding the baseline value by 40%), and head exposure parameters in environmental perception data (the difference between head temperature and ambient temperature is ≤2℃ and lasts for ≥3 seconds). If the visual features do not detect the helmet and the physiological and / or environmental data meet the conditions for not wearing a helmet, it is determined to be a violation of not wearing a helmet. Contextualized validation of prohibited area entry behavior includes: When location data shows that the wearer has entered a preset prohibited work area (such as a flammable and explosive prohibited area or an unauthorized high-altitude work area), the system combines environmental perception data (the gas concentration / temperature in the area meets the prohibition standards), the visual recognition results of area markers from audio and video data (such as feature matching of prohibition warning signs), and the stress response data from physiological signs (heart rate increases by ≥25% compared to the average of the previous 10 seconds) for a collaborative judgment. If the location data and at least two related data points are consistent and support entry into the prohibited area, a violation warning is triggered; otherwise, it is determined that the location data is misjudged and data completion and correction are performed. Cross-modal identification of unauthorized operations includes: For pre-defined violations (such as using open flames in flammable and explosive areas or failing to wear protective equipment as required), a comprehensive judgment is made by integrating visual features from audio and video data (such as flame detection and feature matching of missing protective equipment), abnormal changes in environmental perception data (such as a sudden decrease in oxygen concentration or an increase in combustible gas concentration), and area confirmation from location data. When at least two modalities of data detect the characteristics of a violation and the duration exceeds a pre-defined threshold (such as 3 seconds), it is determined to be a violation.

[0082] In the embodiments of the present invention, it is necessary to further explain that (1) the multimodal collaborative recognition of the behavior of not wearing a safety helmet. Background: Traditional visual recognition has blind spots (such as when a worker's head is obscured when they look down), requiring verification using multimodal data. This embodiment employs a three-source association approach combining visual, physiological, and environmental data. Visual: Match audio and video CNN features with head helmet features (mark as not worn if below a threshold); Physiological: Abnormal contact of heart rate sensor (signal loss ≥5 seconds or fluctuation exceeding 40%); Environment: The temperature difference between the head and the environment is ≤2℃ (if it lasts for ≥3 seconds, it means that the head is not insulated by a safety helmet).

[0083] Application scenarios: Scenarios where wearing safety helmets is mandatory, such as construction sites, mines, and power maintenance, to solve the problem of missed detection by single visual recognition and improve recognition accuracy.

[0084] (2) Contextualized verification of prohibited area entry behavior Background: Traditional location data is prone to misinterpretation due to signal drift, requiring scenario-based multi-source verification. This embodiment employs a four-source collaborative approach: location, environment, audio / video, and physiological data. Location: GPS / BeiDou data indicates entry into a restricted area; Environment: The gas concentration / temperature in the area meets the prohibition standards (e.g., flammable gas exceeds the standard in flammable and explosive areas). Audio and video: Visual recognition of features of no-entry warning signs; Physiological: Heart rate increases by ≥25% compared to the average of the previous 10 seconds (stress response).

[0085] Application scenarios: Flammable and explosive restricted areas in chemical industrial parks, unauthorized high-altitude work areas, dangerous underground areas in mines, etc., to avoid invalid warnings triggered by misjudgment of location signals and ensure that real violations are accurately captured.

[0086] (3) Cross-modal identification of violations Background: Single-modal analysis cannot handle complex operational scenarios (such as smoke obscuring flames), necessitating the fusion of multi-source features. This embodiment employs a comprehensive three-source judgment system—audio / video, environment, and location—to address pre-defined violations (such as open flames or failure to wear protective equipment). Audio and video: Visual feature matching indicates missing flame / protective equipment; Environment: Oxygen concentration drops sharply, flammable gas concentration rises (caused by open flame); Location: Confirmed to be in a high-risk area (such as a flammable or explosive area).

[0087] Application scenarios: scenarios such as illegal hot work operations in chemical industrial parks, failure to wear protective equipment in mines, and illegal use of dangerous tools on construction sites. Cross-modal association is used to improve the robustness of illegal operation identification.

[0088] The above identification steps depend on the coordination of other steps: Preprocessing includes format conversion (standardizing data formats), data cleaning (removing outliers), and scenario-adaptive standardization (such as dual baselines for physiological data) to ensure data quality. Intelligent fusion: modal feature embedding (unifying high-dimensional space), temporal attention (capturing dynamic changes), cross-modal association (quantifying inter-modal weights) to generate accurate fused feature vectors; Compliance Analysis: Based on fused feature vectors, behavioral compliance analysis is performed, and real-time warnings and traceability evidence are output.

[0089] This embodiment overcomes the limitations of traditional single-modal recognition by fusing multi-source heterogeneous data, providing accurate and real-time behavior monitoring for safe operation in high-risk industries and reducing accident risks.

[0090] Background Description: In industrial operation scenarios, traditional methods of tracing operational processes have significant limitations: Multi-source data is scattered: The location trajectory, physiological signs, environmental parameters and audio and video data collected by the safety helmet terminal are stored in a scattered manner, lacking a unified correlation mechanism, making it impossible to quickly locate the multi-source data corresponding to the event; Data authenticity cannot be guaranteed: Traditional traceability systems do not perform integrity verification on raw data, making the data susceptible to tampering and resulting in unreliable accident analysis results; Low traceability efficiency: Manually matching audio and video clips with event data is time-consuming and labor-intensive, and cannot quickly locate relevant evidence of key events; Unmet scenario-based requirements: Accident analysis necessitates the integration of multi-dimensional data (such as physiological abnormalities and exceeding environmental parameter limits) with audio and video evidence, which traditional traceability systems cannot achieve in a unified manner. Therefore: In step S33 above, when performing full-link tracing and audio / video playback of the operation process based on fused data, the following are included: Based on the globally unique timestamps and audio-visual time synchronization index table of fused data, location trajectory events, abnormal physiological signs events, and environmental parameter exceedance events are accurately associated with audio-visual segments at corresponding time points, supporting quick location and playback of associated audio-visual content by event trigger time; The full-link traceability interface synchronously displays the timeline of key events in the operation process (including safety warning triggers and violation identification results), dynamic location trajectory map, physiological sign trend curves, and environmental parameter change charts, and links with the audio and video playback window to achieve integrated visualization of events, data, and audio and video. It supports filtering traceability content by work area ID, equipment ID, event type (such as not wearing a safety helmet, entering a prohibited area), or time interval, and automatically aggregates multi-source heterogeneous data and related audio and video clips for the corresponding time period to generate a structured work traceability report, including event timeline, abnormal data details, and audio and video evidence links; Before audio and video playback, the integrity of the associated multi-source data is verified by checking the hash value of the original data to ensure that the traceable content is authentic and tamper-proof, providing a reliable basis for accident analysis or liability determination.

[0091] In this embodiment of the invention, it needs to be further explained that the main function of this embodiment is as follows: Data association integration: Through globally unique timestamps and audio-visual synchronization index tables, events such as location trajectories, abnormal physiological signs, and excessive environmental parameters are accurately associated with audio-visual segments, enabling integrated traceability of multi-source data; Contextualized presentation: The system displays key event timelines, dynamic location trajectories, physiological trend curves, and audio-visual playback, providing comprehensive data support for accident analysis. Data authenticity assurance: The original data hash value verification mechanism is adopted to ensure that the traceable content is authentic and tamper-proof, providing a reliable basis for liability determination; Highly efficient traceability: Supports filtering content by region, device, event type, or time interval, automatically aggregates multi-source data to generate structured reports, and significantly improves traceability efficiency.

[0092] It provides reliable data support for accident analysis, liability determination, and operation optimization, and is a key link in achieving a closed loop of safety management.

[0093] Example 2, see Figure 3The multifunctional safety helmet back-end management system module interaction diagram of this invention provides, as shown in the following figure. Figure 3 The illustrated multi-functional safety helmet backend management system 20 based on multi-source heterogeneous data fusion includes a backend server 22 communicatively connected to at least one multi-functional safety helmet terminal 21, and a data storage module 23 for storing data. The system further includes: Multi-source heterogeneous data preprocessing module 24: used to perform format conversion, data cleaning, outlier removal and standardization on the multi-source heterogeneous data uploaded by the multi-functional safety helmet terminal 21. The multi-source heterogeneous data includes two or more of the following: location data, physiological sign data, environmental perception data and audio and video data. Intelligent data fusion module 25: It adopts a multimodal fusion algorithm based on attention mechanism to fuse preprocessed multi-source heterogeneous data at the feature layer to generate a unified fused data feature vector; Multifunctional application module 26: Based on fused data feature vectors, it can achieve at least one of the following functions: real-time early warning of safety risks, wearer behavior compliance analysis, and full-link traceability and playback of the operation process; The multi-functional safety helmet terminal 21 is equipped with a sensor component 211 for heterogeneous data acquisition (such as a positioning module (GPS module or Beidou module), heart rate sensor, temperature and humidity sensor, camera) and a wireless communication module 212. In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A backend management method for a multi-functional safety helmet based on multi-source heterogeneous data fusion, comprising the step of receiving data uploaded by at least one multi-functional safety helmet terminal, characterized in that, The method further includes: The multi-source heterogeneous data uploaded by the multi-functional safety helmet terminal is preprocessed, including format conversion, data cleaning, outlier removal and standardization. The multi-source heterogeneous data includes two or more of the following: location data, physiological sign data, environmental perception data and audio and video data. A multimodal fusion algorithm based on an attention mechanism is used to fuse preprocessed multi-source heterogeneous data at the feature layer to obtain a fused data feature vector; Perform at least one of the following management operations based on the feature vector of the fused data: Monitor security risk indicators in fused data and trigger early warnings through multi-dimensional analysis; Analyze wearers' behavioral patterns to identify violations of operating procedures; Full-chain traceability and audio / video playback of the operation process based on fused data.

2. The method according to claim 1, characterized in that, Data cleaning includes: Abnormal jump points in location data are detected and removed based on spatiotemporal continuity rules, and missing data is filled in using linear interpolation. Filter out abnormal values ​​that exceed the normal range threshold of physiological parameters, and use a sliding window weighted mean to smooth and correct continuous abnormal segments; Zero-point drift error is eliminated by using a sensor calibration model, and outliers are removed by combining multi-sensor cross-validation. Static invalid frames are removed using the frame difference method, silent segments are filtered out using the audio energy threshold, and the audio and video streams are time-stamped and synchronized. Heterogeneous data with different sampling frequencies are unified into a preset timestamp sequence, and interpolation and downsampling are used to achieve time dimension consistency alignment.

3. The method according to claim 2, characterized in that, Outlier removal includes: When an abnormal jump in location data is detected, the abnormal jump point is retained or removed by combining the stress response in physiological sign data or the regional characteristics in environmental perception data. When physiological data exceeds the preset normal range threshold, the location data is combined to determine whether the wearer is in a preset high-risk scenario or whether there are extreme conditions in the environmental data, thereby retaining or filtering out the abnormal values. When environmental data is flagged as abnormal, the abnormal value is retained or removed by combining it with similar data from other terminals in the same work area during the same period.

4. The method according to claim 3, characterized in that, Standardization includes: The absolute coordinates are converted to relative coordinates of the work area and then normalized to the [0,1] interval using min-max. The wearer's historical physiological data mean and standard deviation are used as the personal baseline, and the typical physiological range of the group in the current scene is used as the scene baseline. Different baselines are used for z-score transformation according to the scene. The relative deviation rate of single-terminal data is calculated based on the regional terminal collaborative benchmark and normalized to the range of [-1,1]. If the number of terminals in the region is ≤3, the industry preset benchmark value is automatically switched. The extracted cross-modal features are L2 normalized to unit vectors, and then cross-modal joint standardization is performed after timestamp alignment. Perform correlation verification on the standardized data of each type.

5. The method according to claim 4, characterized in that, When performing format conversion, the following are included: The original latitude and longitude coordinates output by the positioning module are converted into coordinate values ​​in the local plane coordinate system specific to the work area, and encapsulated into GeoJSON Feature structured format. At the same time, the continuous location data sequence is converted into trajectory line segment format with time interval. Convert the raw sensor sampling values ​​to the HL7 FHIR Observation medical standard format; Convert the raw sensor signal values ​​into a standardized format of physical quantity units, encapsulate them into JSON key-value pairs, and add an index field for related data from other terminals in the same area; The original audio and video streams are converted into the MP4 unified container format. At the same time, the CNN visual features of the video frames and the MFCC acoustic features of the audio segments are extracted, and after associating the timestamps and scene association tags, they are stored in Protobuf format and an audio and video time synchronization index table is generated.

6. The method according to claim 5, characterized in that, When obtaining the feature vector of the fused data, the following are included: The preprocessed location trajectory features, physiological sign sequence features, environmental perception vectors, and audio / video multimodal features are mapped to a high-dimensional embedding space. The temporal dependencies of features across different modalities are captured through a self-attention layer; Intermodal correlation weights are calculated through a cross-modal attention layer; A unified fusion data feature vector is generated by weighted summation of features across modalities based on cross-modal weights.

7. The method according to claim 6, characterized in that, When an alert is triggered by multidimensional analysis, it includes: The system analyzes the correlation between physiological signs, environmental perception, and location dimensions in the fused data feature vector in real time, and triggers an alert when at least two of the following conditions are met and the duration exceeds a preset time: Abnormal physiological signs and characteristics were observed; Environmental perception features indicate that parameters are out of range; Location characteristics indicate that the area is in a pre-defined high-risk work zone.

8. The method according to claim 6, characterized in that, When identifying violations of operating procedures, the following should be included: By combining CNN visual features extracted from audio and video data, abnormal contact status from physiological sign data, and head exposure parameters from environmental perception data, multi-source correlation verification is performed to determine whether there is any violation of not wearing a safety helmet. When location data shows that the wearer has entered a preset prohibited work area, a collaborative judgment is made by combining environmental perception data, visual recognition results of area markers from audio and video data, and stress response data from physiological signs. For pre-defined violations, a comprehensive judgment is made by integrating visual features of audio and video data, abnormal changes in environmental perception data, and regional confirmation of location data.

9. The method according to claim 6, characterized in that, When performing end-to-end traceability and audio / video playback of the operation process based on fused data, it includes: Based on the globally unique timestamps and audio-visual time synchronization index table of the fused data, various events are accurately associated with audio and video segments at corresponding time points, supporting quick location and playback of associated audio and video content by event trigger time; The full-link traceability interface synchronously displays the timeline of key events in the operation process, dynamic location trajectory map, physiological sign trend curve, environmental parameter change chart, and links the audio and video playback window. It supports filtering traceability content by work area ID, equipment ID, event type, or time interval, and automatically aggregates multi-source heterogeneous data and related audio and video clips for the corresponding time period to generate a structured work traceability report. Before audio and video playback, the integrity of the associated multi-source data is verified by checking the original data hash value.

10. A multi-functional safety helmet backend management system based on multi-source heterogeneous data fusion, comprising a backend server communicatively connected to at least one multi-functional safety helmet terminal, and a data storage module for storing data, characterized in that, The system also includes: Multi-source heterogeneous data preprocessing module: used to perform format conversion, data cleaning, outlier removal and standardization on multi-source heterogeneous data uploaded by the multi-functional safety helmet terminal. Multi-source heterogeneous data includes two or more of the following: location data, physiological sign data, environmental perception data and audio and video data. Intelligent data fusion module: It adopts a multimodal fusion algorithm based on attention mechanism to fuse preprocessed multi-source heterogeneous data at the feature layer to generate a unified fused data feature vector; Multifunctional application module: Based on fused data feature vectors, it can achieve at least one of the following functions: real-time early warning of safety risks, compliance analysis of wearer behavior, and full-link traceability and playback of the operation process; The multi-functional safety helmet terminal is equipped with sensor components and wireless communication modules for heterogeneous data acquisition.

Images