Safety helmet climbing operation early warning method and system based on multi-sensor fusion

Abstract

The invention discloses a safety helmet climbing operation early warning method and system based on multi-sensor fusion, and the method comprises the following steps: carrying out the self-inspection and zero calibration of a sensor, and collecting real-time data; on the basis of the collected real-time data, whether a user is in an outdoor environment or not is judged, and if the user is in the outdoor environment, the climbing state is monitored, and height data is collected in real time; the current climbing state is judged according to the collected real-time height data, and if the current climbing state is in the upward climbing state, an image collection module is started for image collection; the image acquisition module acquires a current ground landmark article image in real time, compares image data acquired at a current acquisition point with image data acquired at a previous acquisition point, judges a current climbing direction, enters an early warning state when the climbing direction is upward, and sends climbing early warning information to the cloud platform; when the collected current height value is smaller than the height of the human body, the climbing early warning state is relieved, and climbing early warning relieving information is sent to the cloud platform.

Description

Technical Field

[0001] This invention relates to the field of high-altitude fall accident detection technology, and in particular to a method and system for early warning of safety helmet climbing operations based on multi-sensor fusion. Background Technology

[0002] With the continuous improvement of safety regulations in high-risk industries such as construction, power, and petrochemicals, the demand for safety protection in high-altitude operations is becoming increasingly prominent. Traditional safety helmets only provide basic physical protection and cannot monitor workers' climbing behavior and environmental risks in real time, leading to frequent falls from heights. Existing technologies attempt to determine height changes using single sensors (such as accelerometers, GNSS, or barometers), but are limited by sensor drift, indoor and outdoor signal differences, and complex environmental interference, resulting in high false alarm rates and significant delays, making it difficult to meet the safety management requirements of "early detection and early intervention." Therefore, an intelligent safety helmet that can integrate multi-sensor data, has adaptive indoor and outdoor judgment capabilities, and can proactively issue warnings during climbing has become an urgent industry need. Thus, there is a pressing need for an intelligent safety helmet system that can integrate multi-sensor data, adapt to complex environments, and accurately identify climbing status. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, the present invention provides a method and system for early warning of safety helmet climbing operations based on multi-sensor fusion.

[0004] The technical solution of this invention is implemented as follows: A safety helmet climbing operation early warning method based on multi-sensor fusion, characterized by the following steps: S1: The sensors in the safety helmet perform self-tests and zero-point calibration, and collect real-time data; S2: Based on the real-time data collected in step S1, determine whether the environment is outdoors. If the environment is outdoors, start monitoring the climbing status and collect height data in real time. S3: Determine the current climbing status based on the collected real-time height data. If the current state is upward climbing, start the image acquisition module to acquire images. S4: The image acquisition module acquires images of current ground landmarks in real time, compares the current image data with the image data acquired at the previous acquisition point, determines the current climbing direction, and enters an early warning state when the climbing direction is upward. S5: Sends climbing warning information to the cloud platform via the network module and plays climbing operation prompts; S6: When the ultrasonic sensor detects that the current height value hs1 is less than the human body height, the climbing warning status is lifted, a lifting warning information is sent to the cloud platform, and a new judgment is made.

[0005] Preferably, in step S1, the sensors include a gyroscope sensor, a barometric pressure sensor, a Global Navigation Satellite System (GNSS) sensor, and an ultrasonic sensor. The gyroscope sensor samples at a preset sampling frequency to obtain the vertical acceleration a1 in real time. A preset acceleration threshold a0 is set. When a1 is greater than the acceleration threshold a0, the state jumps to T1, and the judgment continues. The specific sampling method of the gyroscope sensor is as follows: The gyroscope and accelerometer (MIU) continuously sample at preset values, and align four types of raw values ​​at each sampling time t: a_vert(t), a_horiz(t), ω_pitch(t), and P(t), where a_vert(t) is the instantaneous acceleration in the vertical direction, a_horiz(t) is the resultant acceleration in the horizontal direction, ω_pitch(t) is the pitch angular velocity, and P(t) is the instantaneous ambient air pressure; Construct a four-channel tensor and take the latest frame to form matrix X; When X is fed into a convolutional neural network (CNN), the network outputs the probability of starting the process, P_start. If P_start ≥ preset maximum value, set state T1; if P_start < preset minimum value, reduce the sampling rate of gyroscope and accelerometer to save energy; otherwise, maintain state T0 and continue real-time monitoring.

[0006] Preferably, in step S1, the barometric pressure sensor samples at a preset sampling frequency, collecting the current altitude hg1 and the previous altitude hg2, calculating the altitude change Δh, and setting a preset altitude threshold H0. When Δh is greater than the altitude threshold H0, the system enters state T2 to continue subsequent judgments. The specific sampling method of the barometric pressure sensor is as follows: Continuously read the integrated temperature T_b(t) and power supply voltage V_b(t) of the barometer and establish a first-order compensation model; Perform a Discrete Fourier Transform on the most recent sampling point to detect the period L with the highest energy; if L is within the expected value, it is considered a periodic air conditioning fluctuation. Establish a state vector x=[h, b]ᵀ, where h is the true height and b is the slowly drifting baseline, and perform dynamic Kalman estimation through the prediction equation and the observation equation; The ground pressure P_met (kPa) and altitude H_met (m) of the nearest weather station are obtained via 4G, and the theoretical ground pressure of this station is calculated; if the theoretical ground pressure is less than the preset value, an external consistency check is performed. Otherwise, it will be flagged as a meteorological anomaly, and sudden changes in altitude are unreliable; Define a mutation amount Δh_k. If Δh_k ≥ H0 and σ_k has a preset height and external consistency is passed, then output the height mutation as reliable and enter state T2; otherwise, output the height mutation as unreliable and return to state T0.

[0007] Preferably, the specific method for the GNSS sensor to determine whether it is indoors or outdoors is as follows: Read the number of satellites Ns and the horizontal precision factor HDOP output by the GNSS module; define the signal quality index Q_gnss; if Q_gnss ≥ preset value, initially mark it as sufficient GNSS and continue with subsequent judgment; otherwise, directly mark it as insufficient GNSS. Calculate the Pearson correlation coefficient r between the barometer altitude change Δh_bar and the GNSS altitude change Δh_gnss. If r ≥ the preset value, it is determined to be outdoors; otherwise, it is considered that the GNSS is subject to multipath interference. The average brightness L_mean and coefficient of variation CV are statistically analyzed using the image sensor. If L_mean < 400 lux and CV < preset value, the image is marked as likely to be indoors; otherwise, the image is marked as uncertain and further judgment is required. Activate the helmet's miniature microphone to transmit linear frequency modulation, record the reflected sound, and calculate T_20. If T_20 ≥ the preset value, determine that it is indoors; otherwise, determine that it is outdoors.

[0008] Preferably, in step S1, the GNSS sensor samples according to a preset sampling frequency, acquiring the current altitude hg1 and the previous altitude hg2, calculating the altitude change Δhg, and setting a preset altitude threshold H0. When Δhg is greater than the altitude threshold H0, the system enters state T3 for the next step of judgment. The specific sampling method of the GNSS sensor is as follows: The system utilizes the built-in dual-frequency GNSS module in the safety helmet to receive differential data from the surrounding CORS network, generating an RTK floating-point solution. It outputs a preset frequency altitude sequence h_gnss(t) and provides vertical accuracy σ_v. If σ_v > the preset value, the marking accuracy is insufficient. Subsequently, using the determined outdoor air pressure data P(t), the air pressure altitude h_bar(t) is calculated, and a Kalman filter is established to obtain the fused altitude h_fuse(t) and its variance σ_fuse². If σ_fuse < the preset value, the fusion result is accepted; otherwise, the marking fails. Using the identified ground markers, calculate the visual height h_vis(t). If δ ≤ preset value, the GNSS height is considered to be without drift; otherwise, the marking is abnormal and the output fails. Calculate the current fusion height h_fuse(t) and the previous cycle h_fuse(t−Δt). If Δh≥H0, the output verification passes and enters state T3; otherwise, the output fails, the state returns to T0, and the process returns to step S1.

[0009] Preferably, after determining that it is an outdoor environment, the ultrasonic sensor samples at a preset sampling frequency, collects the current height value hs1 and the previous collected height value hs2, calculates the height difference Δhs, and sets a preset height threshold H0. When Δhs is greater than the height threshold H0, it enters state T3 for subsequent sampling and judgment. The specific sampling method of the ultrasonic sensor is as follows: Read the values ​​from the built-in temperature sensor and anemometer on the safety helmet, and calculate the sound speed correction value c′; Control the ultrasonic sensor to emit pulse trains, collect the arrival time t_i of each echo, and calculate the original flight time series; The echo energy attenuation coefficient α is calculated, and a pre-trained convolutional neural network (CNN) is used to map α and the time-of-flight (TOF) distribution to the ground material category g. The corresponding echo confidence weight w_g is selected according to the category g. Using c′, TOF_i is converted into instantaneous distance d_i. Then, d_i is subjected to weighted median filtering to obtain d_med. d_med is used as the observation value z_k and input into the Kalman filter to output the smooth height h_smooth(t) and its variance σ_h². Calculate the smooth height change. If Δhs ≥ H0 and σ_h < preset value, the output verification passes and enters state T4; otherwise, the output fails and the state returns to T0.

[0010] Preferably, the image acquisition sensor identifies ground landmarks in the current image, with pixel width W1 and height H1, and the same landmarks in the previously acquired image have pixel width W2 and height H2. It then calculates the pixel width change ΔW and pixel height change ΔH. If ΔW is greater than a pixel width threshold W0 or ΔH is greater than a pixel height threshold H, the sensor will determine the appropriate threshold. max When entering state T4, the specific method for image acquisition is as follows: Images are acquired, and the angular IMU velocity ω and acceleration a are recorded simultaneously. A soft synchronization algorithm is used to align each frame of image with the most recent data to form a spatiotemporal frame queue with a sliding window. The Transformer-UNet deep learning model combines a lightweight self-attention mechanism with an encoder-decoder architecture for queued inputs, retaining only connected components with confidence values ​​greater than a preset value as stable markers S. A 3D plane assumption is established for the stable marker S. The relative pose of the camera is solved by using IMU pre-integration and visual reprojection error in the sliding window to obtain the vertical displacement Δz_vis and the confidence level σ_vis. The instantaneous scale factor is calculated using the real physical size L_real of the marker and the image pixel size l_pixel; Perform Random Sampling Consensus (RANSAC) fitting on the queue to obtain the height-time curve h(t). If the curve meets the requirements, proceed to subsequent verification; otherwise, output visual verification failure and return to T0. Calculate the final visual height increment. If Δh_final ≥ H and confidence σ_vis < preset value, then output that the visual verification is successful and enter state T4. State T4 proves that we are currently in the climbing state; otherwise, output that the visual verification is unsuccessful.

[0011] As a preferred option, the specific method for issuing the warning in step S5 is as follows: After the safety helmet is powered on, it reads the wearer's work badge UID (Unique Identifier) ​​via near-field RFID (Radio Frequency Identification) and calls the locally cached permission table to obtain the wearer's name, job type, and maximum allowed working height H_max; Calculate the relative over-limit rate using the current fusion height h_now and H_max and map it to the hierarchical interval; The microphone continuously measures the ambient sound pressure level (SPL) and dynamically adjusts the voice broadcast gain (G). The level, name, real-time altitude, and seat belt status are concatenated into a semantic template, which is then synthesized into PCM (Pulse Code Modulation) in real time through the on-device TTS (Text-to-Speech) engine. After the broadcast is completed, the microphone is used to detect whether the wearer has issued a voice confirmation keyword; if confirmed, a positive sample is recorded; if there is no response or a sudden change in environmental noise, a negative sample is marked.

[0012] Preferably, the method for canceling the ascent warning by determining the current altitude in step S6 is as follows: When the ultrasonic measurement height is less than the preset height value, the safety helmet MCU (microcontroller unit) immediately reads the pitch angle θ_pitch and horizontal acceleration a_horiz from the IMU (inertial measurement unit) to verify whether the wearer is in an upright and stationary state. The judgment condition is that |θ_pitch| < preset angle and a_horiz < preset value. If the condition is met, the local confirmation flag is set; otherwise, the warning is maintained and the process returns to step S1. After local confirmation, the safety helmet sends a release request via 4G. Upon receiving the request, the cloud searches all work order records associated with the same task order for that helmet. If there are any unfastened safety belts or leftover tools in the sub-work order, the cloud returns a rejection for release with the ID of the unfinished item; otherwise, it returns an acceptance for release. If the peak height or duration of the ascent is greater than or equal to the preset value, the tiered review process will be triggered. After the review is approved, the cloud performs a hash signature on the complete data of this event and writes it into the immutable audit chain. Then, it generates an event archive number (Archive_ID) and sends it back to the safety helmet. The safety helmet stores the Archive_ID locally and plays a prompt, completing the entire deactivation process.

[0013] A safety helmet climbing operation early warning system based on multi-sensor fusion is characterized by comprising: The physical sensing layer continuously acquires triaxial acceleration, angular velocity, magnetic field, satellite positioning results, meteorological and acoustic parameters, visible light images and ultrasonic echoes at a specified frequency. After temperature compensation, sound velocity correction and time synchronization, the raw data is provided to the edge fusion layer as a distortion-free and time-accurate multimodal input stream. The edge fusion layer completes online calibration, feature extraction, model inference, confidence assessment, state transition decision-making, and real-time audio-visual prompts for sensor data on the device side. At the same time, it uploads key events and logs to the cloud in real time through an encrypted link and has the ability to remotely update parameters and perform differential firmware flashing. The cloud service layer uses a high-throughput streaming processing framework to receive, persistently store, and index data from all terminals in partitions. It uses a group collaborative verification algorithm to perform correlation analysis on data from multiple nodes on the same work surface to suppress false alarms. It also pushes real-time curves and graded prompts to the web screen and mobile terminals through visualization interfaces and alarm APIs. The operations and data layer is responsible for version management, canary release and rollback of algorithm models, collecting and encrypting terminal operation logs for fault prediction and maintenance work order dispatch, and continuously outputting device location, power consumption, sensor status and event information to the digital twin platform to achieve a closed loop of data throughout the entire lifecycle.

[0014] The beneficial effects of this invention are as follows: This invention achieves centimeter-level height estimation by deeply fusing multiple sensors, including IMU, GNSS, barometer, ultrasound, vision, and acoustic sensors, and employs Kalman filtering, monocular-IMU joint optimization, and Transformer-UNet semantic segmentation at the edge, thereby reducing the overall system error.

[0015] This invention introduces a motion-initiated trigger, a scene-adaptive engine, and a group collaborative verifier, which establish dynamic thresholds and correlation constraints for interference sources such as vibration, temperature drift, multipath, and stair walking, respectively. The false alarm rate in field testing is reduced, reaching an industry-leading level.

[0016] This invention utilizes GNSS signal integrity indicators, barometric pressure-satellite consistency, visible light intensity, and acoustic reverberation time to determine multi-dimensional scenarios. It can automatically select the optimal fusion strategy in complex environments such as outdoor tower cranes, indoor warehouses, and underground utility tunnels without manual switching, achieving seamless coverage.

[0017] This invention deploys all edge computing algorithms on the MCU, with CNN inference latency of less than 30 ms and ultrasound-vision-IMU joint computation of less than 50 ms per frame; dynamic sampling rate and Tickless sleep extend the overall battery life to more than 12 hours, meeting the needs of all-day operation.

[0018] This invention automatically generates differentiated prompts based on the wearer's identity, real-time height percentage, environmental noise, and historical behavior, and broadcasts them in real time via the device. Field tests show that the voice confirmation rate is improved and the interruption time of operations is reduced.

[0019] The real-time stream processing engine of this invention supports 30,000 concurrent safety helmets, and pushes events to the supervision terminal within 1 second; the digital twin interface and audit chain ensure that events are traceable and accountable; and the model canary release is realized through OTA (over-the-air download), shortening the average upgrade time. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the workflow of the safety helmet climbing operation early warning method based on multi-sensor fusion of the present invention; Figure 2 This is a flowchart illustrating the early warning method for safety helmet climbing operations based on multi-sensor fusion according to the present invention. Figure 3 This is a system block diagram of the safety helmet climbing operation early warning system based on multi-sensor fusion of the present invention; Figure 4 A schematic diagram showing the result of performing a discrete Fourier transform on this invention; Figure 5 This is a first schematic diagram of data sampling for the barometric pressure sensor of the present invention; Figure 6 This is a second schematic diagram of data sampling for the barometric pressure sensor of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figure 1-2As shown in Figures 4-6, this invention provides a method for early warning of safety helmet climbing operations based on multi-sensor fusion, comprising the following steps: S1: The sensors in the safety helmet perform self-tests and zero-point calibration, and collect real-time data; S2: Based on the real-time data collected in step S1, determine whether the environment is outdoors. If the environment is outdoors, start monitoring the climbing status and collect height data in real time. S3: Determine the current climbing status based on the collected real-time height data. If the current state is upward climbing, start the image acquisition module to acquire images. S4: The image acquisition module acquires images of current ground landmarks in real time, compares the current image data with the image data acquired at the previous acquisition point, determines the current climbing direction, and enters an early warning state when the climbing direction is upward. S5: Sends climbing warning information to the cloud platform via the network module and plays climbing operation prompts; S6: When the ultrasonic sensor detects that the current height value hs1 is less than the human body height, the climbing warning status is lifted, a lifting warning information is sent to the cloud platform, and a new judgment is made.

[0023] Further, in step S1, the sensors include a gyroscope sensor, a barometric pressure sensor, a Global Navigation Satellite System (GNSS) sensor, and an ultrasonic sensor. The gyroscope sensor samples at a sampling frequency of 10Hz to obtain the vertical acceleration a1 in real time, with a preset acceleration threshold a0 = 1m / s². 2 When a1 is greater than the acceleration threshold a0, the state jumps to T1 and continues to make judgments. The specific sampling method of the gyroscope sensor is as follows: The gyroscope and accelerometer (MIU) sampled continuously at 100 Hz, and the barometer sampled at 50 Hz. For each sampling time t, four types of raw values ​​were aligned: a_vert(t), a_horiz(t), ω_pitch(t), and P(t), where a_vert(t) is the instantaneous acceleration in the vertical direction, a_horiz(t) is the resultant acceleration in the horizontal direction, ω_pitch(t) is the pitch angular velocity, and P(t) is the instantaneous ambient air pressure. Construct a four-channel tensor, and take the latest 50 frames to form matrix X, with the following formula: X = [a_vert;a_horiz;ω_pitch;ΔP], where ΔP(i)=P(i)−P(i−1); When X is fed into a convolutional neural network (CNN), the network outputs the starting probability P_start, which is given by the formula: P_start=σ(W3·ReLU(W2·GlobalAvg(CNN(X)))); If P_start ≥ 0.85, set state T1; If P_start < 0.20 and continues for 10 seconds, reduce the IMU sampling rate to save energy. In other cases, the status remains T0, and real-time monitoring continues. Further, in step S1, the barometric pressure sensor samples at a sampling frequency of 1Hz, acquiring the current altitude hg1 and the previous altitude hg2, and calculates the altitude change Δh = hg1 - hg2. The altitude threshold H0 can be selected as 1m. When Δh is greater than the altitude threshold H0, the system enters state T2 to continue subsequent judgments. The specific sampling method of the barometric pressure sensor is as follows: By continuously reading the integrated temperature T_b(t) and supply voltage V_b(t) of the barometer, a first-order compensation model is established: P_corr(t)=P_raw(t)+α·(T_b(t)−T_ref)+β·(V_b(t)−V_ref), Where α = −0.2 hPa / °C, β = 0.05 hPa / mV, and T_ref and V_ref are the average reference values ​​within 10 seconds of power-on; Perform a Discrete Fourier Transform on the most recent 90 sampling points to detect the period L with the highest energy; If L is between 25 and 35, then it is considered to be a periodic air conditioning fluctuation, and the sliding average window length W=L is calculated. P_smooth(t)=1 / W·Σ_{k=t−W+1}^{t}P_corr(k); Establish a state vector x = [h, b]ᵀ, where h is the true altitude and b is the slowly drifting baseline. Perform dynamic Kalman estimation using the prediction equation and the observation equation, where: Prediction equation: x_{k|k-1}=F·x_{k-1|k-1}, where F=[[1,Δt],[0,1]]; Observation equation: z_k = h+b+v_k, v_k~N(0, R_k), R_k is updated in real time by the residual variance from step S302; The updated result yields the optimal estimate h_k and its variance σ_k².

[0024] For example, if h increases from 0m to 1.05m within 10 seconds and σ_k converges to 0.07m, it indicates that the mutation is plausible.

[0025] The theoretical surface air pressure P_met (kPa) and altitude H_met (m) of the nearest 5km weather station were obtained via 4G, and the following data were used to calculate the surface air pressure of this station: P_theory(t)=P_met·exp[g·(H_met−h_k) / (R·T_avg)]; If the theoretical ground air pressure is less than 30 Pa, then the external consistency check will be passed. Otherwise, it will be flagged as a meteorological anomaly, and sudden changes in altitude are unreliable; For example, the weather station gives P_met=1013.25hPa, H_met=5m, T_avg=298K. When h_k=1.05m, P_theory=1012.12 hPa, and the measured P_smooth=1012.10 hPa, the difference is 2 Pa, so it passes.

[0026] Define the mutation amount Δh_k, Δh_k = h_k−h_{kN}, where N is 10s; If Δh_k≥H and σ_k<0.10 m and external consistency is passed, then the output is highly reliable and enters state T2; otherwise, the output is unreliable and the state returns to T0.

[0027] Furthermore, the specific method by which the GNSS sensor determines whether it is indoors or outdoors is as follows: Read the number of satellites Ns and the horizontal precision factor HDOP output by the GNSS module; define the signal quality index Q_gnss=Ns / (HDOP +ε), where ε=0.1 to avoid division by zero; If Q_gnss≥6, it is initially marked as sufficient GNSS, and further judgment is continued; otherwise, it is directly marked as insufficient GNSS. Calculate the Pearson correlation coefficient r between the barometer altitude change Δh_bar and the GNSS altitude change Δh_gnss. If r ≥ 0.8, it is determined to be outdoor; otherwise, it is considered that the GNSS is subject to multipath interference, and subsequent calculations are performed. The average brightness L_mean and coefficient of variation CV were statistically analyzed using an image sensor. CV = σ_L / L_mean; Indoor lighting is typically constant; a threshold value is set. If L_mean < 400 lux and CV < 0.15, then the label is most likely indoors; otherwise, the label is uncertain, and further judgment will be made. Activate the helmet's miniature microphone to emit a 20ms 1kHz linear frequency modulated signal, record the reflected sound and calculate T_20. If T_20 ≥ 0.35 s, determine that it is indoors; otherwise, determine that it is outdoors. Record the reflected sound and calculate T_20. If T_20 ≥ 0.35 s, then determine that it is indoors; otherwise, determine that it is outdoors.

[0028] Further, in step S1, the GNSS sensor samples at a sampling frequency of 1Hz, acquiring the current altitude hg1 and the previously acquired altitude hg2, and calculates the altitude change Δhg = hg1 - hg2. The altitude threshold H0 can be selected as 1m. When Δhg is greater than the altitude threshold H0, the system enters state T3 for the next step of judgment. The specific sampling method of the GNSS sensor is as follows: The safety helmet's built-in dual-frequency GNSS module is used to receive differential data from the surrounding CORS (Continuously Operating Reference Station) network to form an RTK (Real-Time Kinematic Differential Positioning) floating-point solution; Output the 1Hz altitude sequence h_gnss(t) and give the vertical accuracy σ_v. If σ_v > 0.5 m, the marking accuracy is insufficient. Then, use the determined outdoor air pressure data P(t) to calculate the barometric altitude h_bar(t): h_bar(t) = −(RT / g)·ln(P(t) / P0), where R=287 J·kg-1·K-1, T=288.15 K, g=9.806 m·s-2, and P0 is the reference pressure; Establish a Kalman filter to obtain the fusion height h_fuse(t) and its variance σ_fuse². If σ_fuse < 0.3 m, accept the fusion result; otherwise, mark it as a failure. If the accuracy is sufficient, the visual height h_vis(t) can be calculated directly using the ground markers identified in step S4, and the three-source consistency residual can be calculated: δ = |h_fuse(t)−h_vis(t)|; If δ≤0.2m, the GNSS altitude is considered to be without drift, and the subsequent fusion altitude judgment is performed; otherwise, an anomaly is marked and the output fails. Calculate using the current fusion height h_fuse(t) and the previous period's h_fuse(t−Δt): Δh = h_fuse(t)−h_fuse(t−Δt), if Δh≥H0, the output verification passes and enters state T3; otherwise, the output fails, the state returns to T0, and the process returns to step S1.

[0029] Furthermore, after determining that it is an outdoor environment, the ultrasonic sensor samples at a sampling frequency of 1Hz, acquiring the current height value hs1 and the previous acquired height value hs2, calculating the height difference Δhs, and setting a preset height threshold H0. When Δhs is greater than the height threshold H0, it enters state T3. The height threshold H0 can be selected as 1m. The specific sampling method is as follows: Read the values ​​from the built-in temperature sensor and anemometer on the safety helmet, and calculate the sound speed correction value c′: c′ = 331.3√(1+ T / 273.15)+0.6v_w; The ultrasonic sensor is controlled to emit a burst of N = 8 pulses at 40 kHz with a 5 ms interval. The arrival time t_i of each echo is acquired, and the original flight time series is calculated. TOF_i = t_i − t_emit; Calculate the echo energy attenuation coefficient α: α = 20log10(A_emit / A_echo), using a pre-trained CNN to map α and the TOF distribution to ground material category g, and selecting the corresponding echo confidence weight w_g based on category g: w_g={1.0, 0.9, 0.8, 0.6}; Use c′ to convert TOF_i into instantaneous distance d_i: d_i = c′·TOF_i / 2, We obtain d_med by performing a weighted median filter on d_i: d_med = median{w_g · d_i}; Input d_med as the observation value z_k into the Kalman filter, and output the smooth height h_smooth(t) and its variance σ_h²; Calculate the smooth height change. If Δhs≥H0 and σ_h<0.05 m, the output verification passes and enters state T4; otherwise, the output fails and the state returns to T0.

[0030] Furthermore, the image acquisition sensor identifies ground landmarks in the current image, with pixel width W1 and height H1, and the same landmarks in the previously acquired image, with pixel width W2 and height H2. It calculates the pixel width change ΔW and pixel height change ΔH. If ΔW is greater than a pixel width threshold W0 or ΔH is greater than a pixel height threshold H... max When entering state T4, the specific method for image acquisition is as follows: Images are acquired at 30fps, and the angular IMU velocity ω and acceleration a are recorded simultaneously. A soft synchronization algorithm is used to align each frame with the most recent data, forming a spatiotemporal frame queue Q with a 0.5s sliding window. Q = {I_t,ω_t,a_t | t=1…15}; Input queue Q into lightweight Transformer-UNet, output semantic mask M and geometric edge graph E, and retain only connected components with confidence greater than 0.9 as stable markers S; For the stable marker S, a 3D plane assumption is established, and the plane equation is: nᵀX = d; Jointly optimize using IMU pre-integration within a sliding window and visual reprojection error to solve for the relative pose \(T\) of the camera, obtaining the vertical displacement \(\Delta z_{vis}\) and confidence \(\sigma_{vis}\); Use the true physical size \(L_{real}\) of the marker and the image pixel size \(l_{pixel}\) to calculate the instantaneous scale factor: \(s = L_{real} / l_{pixel}\); Convert \(\Delta z_{vis}\) to the physical height change: \(h_{vis}=s\cdot\Delta z_{vis}\); S805: Perform RANSAC fitting on the queue to obtain the height-time curve \(h(t)\), with the requirements: Residual: \(RMS < 0.02\ m\);   Slope: \(\alpha>0.7\ m / s\), ensuring upward movement;   Fusion height difference with air pressure, ultrasonic, and IMU: \(|h_{vis}-h_{fuse}|<0.15\ m\); If all three are satisfied, proceed to subsequent verification; otherwise, output visual verification failure and return to T0; Subsequently, calculate the final visual height increment. If \(\Delta h_{final}\geq H\) and confidence \(\sigma_{vis}<0.03\ m\), output visual verification passed and enter state T4. State T4 indicates that the current state is climbing; otherwise, output visual verification failure.

[0031] Furthermore, in step S5, the specific method for giving an early warning is as follows: After the safety helmet is powered on, it reads the wearer's work permit UID through near-field RFID and calls the locally cached permission table to obtain the wearer's name, job type, and the maximum allowed working height \(H_{max}\); Use the current fusion height \(h_{now}\) and \(H_{max}\) to calculate the relative overlimit rate: \(R=(h_{now} / H_{max})\times100\%\); And map it to the grading interval: \(0\%<R\leq60\%\): Level L1; \(60\%<R\leq85\%\): Level L2; \(85\%<R\leq100\%\): Level L3; \(R > 100\%\): Level L4; For example, \(h_{now}=2.8\ m\), \(H_{max}=3.5\ m\), \(R = 80\%\), corresponding to level L2.

[0032] The microphone continuously measures the ambient sound pressure level SPL and dynamically adjusts the voice broadcast gain G: \(G = min(0\ dB,24 - 0.2\cdot SPL)\), Ensure that the output is 75dB in a quiet 55dB site and increases to 85dB in a noisy 95dB area without exceeding the regulatory limit. The level, name, real-time altitude, and seatbelt status are concatenated into a semantic template, which is then used by the on-device TTS engine to synthesize the PCM in real time. Text = "Zhang San, current height 2.8 meters, 80% of the permitted height has been reached. Please fasten your safety belt and stop climbing." TTS inference time is less than 30 ms, and data is directly written to the DAC (digital-to-analog converter) buffer; After the broadcast is completed, the microphone detects whether the wearer has issued a voice confirmation keyword, such as "received," within 3 seconds. If confirmed, a positive sample is recorded. If there is no response or a sudden change in environmental noise, a negative sample is marked. Positive and negative samples are uploaded to the cloud model repository daily for periodic retraining of TTS speech rate, tone, and trigger thresholds to achieve closed-loop optimization.

[0033] For example, if Zhang San responds "received" within 3 seconds, this data is marked as a positive sample upload, and the TTS speech rate will automatically decrease by 5% in the next time the same scenario is used.

[0034] Furthermore, the method for canceling the ascent warning by determining the current altitude in step S6 is as follows: When the ultrasonic measurement height is less than 2.5 m, the safety helmet MCU immediately reads the IMU pitch angle θ_pitch and horizontal acceleration a_horiz to verify whether the wearer is in an upright and stationary state; the judgment condition is |θ_pitch|<10° and a_horiz<0.2 g; if the condition is met, the local confirmation flag is set; otherwise, the warning is maintained and the process returns to step S1. After local confirmation, the safety helmet sends a release request via 4G. Upon receiving the request, the cloud searches all work order records associated with the same task order for that helmet. If there are any unfastened safety belts or leftover tools in the sub-work order, the cloud returns a rejection for release with the ID of the unfinished item; otherwise, it returns an acceptance for release. If the peak height of this ascent is greater than or equal to 3.5m or the duration is greater than or equal to 300s, the tiered review process will be triggered. After the review is approved, the cloud performs a hash signature on the complete data of this event and writes it into the immutable audit chain. Then, it generates an event archive number (Archive_ID) and sends it back to the safety helmet. The safety helmet stores the Archive_ID locally and plays a "Climbing ended, archived" prompt, completing the entire release process.

[0035] like Figure 3 As shown, the present invention provides a safety helmet climbing operation early warning system based on multi-sensor fusion, comprising: The physical sensing layer continuously acquires triaxial acceleration, angular velocity, magnetic field, satellite positioning results, meteorological and acoustic parameters, visible light images and ultrasonic echoes at a specified frequency. After temperature compensation, sound velocity correction and time synchronization, the raw data is provided to the edge fusion layer as a distortion-free and time-accurate multimodal input stream. The edge fusion layer completes online calibration, feature extraction, model inference, confidence assessment, state transition decision-making, and real-time audio-visual prompts for sensor data on the device side. At the same time, it uploads key events and logs to the cloud in real time through an encrypted link and has the ability to remotely update parameters and perform differential firmware flashing. The cloud service layer uses a high-throughput streaming processing framework to receive, persistently store, and index data from all terminals in partitions. It uses a group collaborative verification algorithm to perform correlation analysis on data from multiple nodes on the same work surface to suppress false alarms. It also pushes real-time curves and graded prompts to the web screen and mobile terminals through visualization interfaces and alarm APIs. The operations and data layer is responsible for version management, canary release and rollback of algorithm models, collecting and encrypting terminal operation logs for fault prediction and maintenance work order dispatch, and continuously outputting device location, power, sensor status and event information to the digital twin platform to achieve a closed loop of data throughout the entire lifecycle. The digital twin platform adopts the self-developed SafeHat-DTwin platform, version v2024.05, which is compatible with Unity 3D 2022.3 LTS and CesiumJS, and provides gRPC (Google Remote Procedure Call) and Shadow JSON v2 protocol interfaces to the outside world, supporting real-time rendering of 6000 safety helmets in concurrent scenes.

[0036] Specifically, the algorithm model includes: motion-triggered model 1-D MobileNet-V3 lightweight CNN, with 8kB parameters, and the current version is v2.1.3; The visual segmentation model Transformer-UNet-Lite has 0.7 MB of parameters and is version v1.4.0. Acoustic ground material classification model 1-D CNN, 3 kB parameters, version v1.2.0; State machine Kalman filter model: Standard linear Kalman, version v3.0.1.

[0037] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

Claims

1. A method for early warning of safety helmet climbing operations based on multi-sensor fusion, characterized in that: Includes the following steps: S1: The sensors in the safety helmet perform self-tests and zero-point calibration, and collect real-time data; S2: Based on the real-time data collected in step S1, determine whether the environment is outdoors. If the environment is outdoors, start monitoring the climbing status and collect height data in real time. S3: Determine the current climbing status based on the collected real-time height data. If the current state is upward climbing, start the image acquisition module to acquire images. S4: The image acquisition module acquires images of current ground landmarks in real time, compares the current image data with the image data acquired at the previous acquisition point, determines the current climbing direction, and enters an early warning state when the climbing direction is upward. S5: Sends climbing warning information to the cloud platform via the network module and plays climbing operation prompts; S6: When the ultrasonic sensor detects that the current height value hs1 is less than the human body height, the climbing warning status is lifted, a lifting warning information is sent to the cloud platform, and a new judgment is made.

2. The safety helmet climbing operation early warning method based on multi-sensor fusion according to claim 1, characterized in that: In step S1, the sensors include a gyroscope sensor, a barometric pressure sensor, a Global Navigation Satellite System (GNSS) sensor, and an ultrasonic sensor. The gyroscope sensor samples at a preset sampling frequency to obtain the vertical acceleration a1 in real time. A preset acceleration threshold a0 is set. When a1 is greater than the acceleration threshold a0, the state jumps to T1, and the judgment continues. The specific sampling method of the gyroscope sensor is as follows: The gyroscope and accelerometer (MIU) continuously sample at preset values, and align four types of raw values ​​at each sampling time t: a_vert(t), a_horiz(t), ω_pitch(t), and P(t), where a_vert(t) is the instantaneous acceleration in the vertical direction, a_horiz(t) is the resultant acceleration in the horizontal direction, ω_pitch(t) is the pitch angular velocity, and P(t) is the instantaneous ambient air pressure; Construct a four-channel tensor and take the latest frame to form matrix X; When X is fed into a convolutional neural network (CNN), the network outputs the probability of starting the process, P_start. If P_start ≥ preset maximum value, set state T1; if P_start < preset minimum value, reduce the sampling rate of gyroscope and accelerometer to save energy; otherwise, maintain state T0 and continue real-time monitoring.

3. The safety helmet climbing operation early warning method based on multi-sensor fusion according to claim 1, characterized in that: In step S1, the barometric pressure sensor samples at a preset sampling frequency, collecting the current altitude hg1 and the previous altitude hg2, calculating the altitude change Δh, and setting a preset altitude threshold H0. When Δh is greater than the altitude threshold H0, the system enters state T2 to continue subsequent judgments. The specific sampling method of the barometric pressure sensor is as follows: Continuously read the integrated temperature T_b(t) and power supply voltage V_b(t) of the barometer and establish a first-order compensation model; Perform a Discrete Fourier Transform on the most recent sampling point to detect the period L with the highest energy; if L is within the expected value, it is identified as a periodic air conditioning fluctuation. Establish a state vector x=[h, b]ᵀ, where h is the true height and b is the slowly drifting baseline, and perform dynamic Kalman estimation through the prediction equation and the observation equation; The ground pressure P_met (kPa) and altitude H_met (m) of the nearest weather station are obtained via 4G, and the theoretical ground pressure of this station is calculated; if the theoretical ground pressure is less than the preset value, an external consistency check is performed. Otherwise, it will be flagged as a meteorological anomaly, and sudden changes in altitude are unreliable; Define a mutation amount Δh_k. If Δh_k ≥ H0 and σ_k has a preset height and external consistency is passed, then output the height mutation as reliable and enter state T2; otherwise, output the height mutation as unreliable and return to state T0.

4. The safety helmet climbing operation early warning method based on multi-sensor fusion according to claim 1, characterized in that: The specific method used by the GNSS sensor to determine indoor and outdoor conditions is as follows: Read the number of satellites Ns and the horizontal precision factor HDOP output by the GNSS module; define the signal quality index Q_gnss; if Q_gnss ≥ preset value, initially mark it as sufficient GNSS and continue with subsequent judgment; otherwise, directly mark it as insufficient GNSS. Calculate the Pearson correlation coefficient r between the barometer altitude change Δh_bar and the GNSS altitude change Δh_gnss. If r ≥ the preset value, it is determined to be outdoors; otherwise, it is considered that the GNSS is subject to multipath interference. The average brightness L_mean and coefficient of variation CV are statistically analyzed using the image sensor. If L_mean < 400 lux and CV < preset value, the image is marked as likely to be indoors; otherwise, the marking is uncertain and further judgment is required. Activate the helmet's miniature microphone to transmit linear frequency modulation, record the reflected sound, and calculate T_20. If T_20 ≥ the preset value, determine that it is indoors; otherwise, determine that it is outdoors.

5. The safety helmet climbing operation early warning method based on multi-sensor fusion according to claim 1, characterized in that: In step S1, the GNSS sensor samples according to a preset sampling frequency, acquiring the current altitude hg1 and the previous altitude hg2, calculating the altitude change Δhg, and setting a preset altitude threshold H0. When Δhg is greater than the altitude threshold H0, the system enters state T3 for the next step of judgment. The specific sampling method of the GNSS sensor is as follows: The system utilizes the built-in dual-frequency GNSS module of the safety helmet to receive differential data from the surrounding CORS network, generating an RTK floating-point solution. It outputs a preset frequency altitude sequence h_gnss(t) and provides vertical accuracy σ_v. If σ_v > the preset value, the marking accuracy is insufficient. Subsequently, using the determined outdoor air pressure data P(t), the air pressure altitude h_bar(t) is calculated, and a Kalman filter is established to obtain the fused altitude h_fuse(t) and its variance σ_fuse². If σ_fuse < the preset value, the fusion result is accepted; otherwise, the marking fails. Using the identified ground markers, calculate the visual height h_vis(t). If δ ≤ preset value, the GNSS height is considered to be without drift; otherwise, the marking is abnormal and the output fails. Calculate the current fusion height h_fuse(t) and the previous cycle h_fuse(t−Δt). If Δh≥H0, the output verification passes and enters state T3; otherwise, the output fails, the state returns to T0, and the process returns to step S1.

6. The safety helmet climbing operation early warning method based on multi-sensor fusion according to claim 1, characterized in that: After determining that it is an outdoor environment, the ultrasonic sensor samples according to a preset sampling frequency, acquiring the current height value hs1 and the previous acquired height value hs2, calculating the height difference Δhs, and a preset height threshold H0. When Δhs is greater than the height threshold H0, it enters state T3 for subsequent sampling and judgment. The specific sampling method of the ultrasonic sensor is as follows: Read the values ​​from the built-in temperature sensor and anemometer on the safety helmet, and calculate the sound speed correction value c′; Control the ultrasonic sensor to emit pulse trains, collect the arrival time t_i of each echo, and calculate the original flight time series; The echo energy attenuation coefficient α is calculated, and a pre-trained convolutional neural network (CNN) is used to map α and the time-of-flight (TOF) distribution to the ground material category g. The corresponding echo confidence weight w_g is selected according to the category g. Using c′, TOF_i is converted into instantaneous distance d_i. Then, d_i is subjected to weighted median filtering to obtain d_med. d_med is used as the observation value z_k and input into the Kalman filter to output the smooth height h_smooth(t) and its variance σ_h². Calculate the smooth height change. If Δhs ≥ H0 and σ_h < preset value, the output verification passes and enters state T3; otherwise, the output fails and the state returns to T0.

7. The safety helmet climbing operation early warning system based on multi-sensor fusion according to claim 1, characterized in that: The image acquisition sensor identifies the pixel width W1 and height H1 of ground landmarks in the current image, and the pixel width W2 and height H2 of the same landmark in the previous image. It then calculates the pixel width change ΔW and the pixel height change ΔH. If ΔW is greater than a pixel width threshold W0 or ΔH is greater than a pixel height threshold H... max When entering state T4, the specific method for image acquisition is as follows: Images are acquired, and the angular IMU velocity ω and acceleration a are recorded simultaneously. A soft synchronization algorithm is used to align each frame of image with the most recent data to form a spatiotemporal frame queue with a sliding window. The Transformer-UNet deep learning model combines a lightweight self-attention mechanism with an encoder-decoder architecture for queued inputs, retaining only connected components with confidence values ​​greater than a preset value as stable markers S. A 3D plane assumption is established for the stable marker S. The relative pose of the camera is solved by jointly optimizing the IMU pre-integration and visual reprojection error within the sliding window, and the vertical displacement Δz_vis and confidence σ_vis are obtained. The instantaneous scale factor is calculated using the real physical size L_real of the marker and the image pixel size l_pixel; Perform Random Sampling Consensus (RANSAC) fitting on the queue to obtain the height-time curve h(t). If it meets the requirements, proceed with subsequent verification; otherwise, output visual verification failure and return to T0. Calculate the final visual height increment. If Δh_final ≥ H and confidence σ_vis < preset value, then output that the visual verification is successful and enter state T4. State T4 proves that we are currently in the climbing state; otherwise, output that the visual verification is unsuccessful.

8. The safety helmet climbing operation early warning system based on multi-sensor fusion according to claim 1, characterized in that: In step S5, the specific method for issuing an early warning is as follows: After the safety helmet is powered on, it reads the wearer's work badge UID (Unique Identifier) ​​via near-field RFID (Radio Frequency Identification) and calls the locally cached permission table to obtain the wearer's name, job type, and maximum allowed working height H_max; Calculate the relative over-limit rate using the current fusion height h_now and H_max and map it to the hierarchical interval; The microphone continuously measures the ambient sound pressure level (SPL) and dynamically adjusts the voice broadcast gain (G). The level, name, real-time altitude, and seat belt status are concatenated into a semantic template, which is then synthesized into PCM (Pulse Code Modulation) in real time through the on-device TTS (Text-to-Speech) engine. After the broadcast is completed, the microphone is used to detect whether the wearer has issued a voice confirmation keyword; if confirmed, a positive sample is recorded; if there is no response or a sudden change in environmental noise, a negative sample is marked.

9. The safety helmet climbing operation early warning system based on multi-sensor fusion according to claim 1, characterized in that: The method for canceling the ascent warning by determining the current altitude in step S6 is as follows: When the ultrasonic measurement height is less than the preset height value, the safety helmet MCU (microcontroller unit) immediately reads the pitch angle θ_pitch and horizontal acceleration a_horiz from the IMU (inertial measurement unit) to verify whether the wearer is in an upright and stationary state. The judgment condition is |θ_pitch| < preset angle and a_horiz < preset value. If the condition is met, the local confirmation flag is set; otherwise, the warning is maintained and the process returns to step S1. After local confirmation, the safety helmet sends a release request via 4G. Upon receiving the request, the cloud searches all work order records associated with the same task order for that helmet. If there are any unfastened safety belts or leftover tools in the sub-work order, the cloud returns a rejection for release with the ID of the unfinished item; otherwise, it returns an acceptance for release. If the peak height or duration of the ascent is greater than or equal to the preset value, the tiered review process will be triggered. After the review is approved, the cloud performs a hash signature on the complete data of this event and writes it into the immutable audit chain. Then, it generates an event archive number (Archive_ID) and sends it back to the safety helmet. The safety helmet stores the Archive_ID locally and plays a prompt, completing the entire deactivation process.

10. A helmet climbing operation early warning system based on multi-sensor fusion, used to implement the helmet climbing operation early warning method based on multi-sensor fusion as described in any one of claims 1-9, characterized in that: include: The physical sensing layer continuously acquires triaxial acceleration, angular velocity, magnetic field, satellite positioning results, meteorological and acoustic parameters, visible light images and ultrasonic echoes at a specified frequency. After temperature compensation, sound velocity correction and time synchronization, the raw data is provided to the edge fusion layer as a distortion-free and time-accurate multimodal input stream. The edge fusion layer completes online calibration, feature extraction, model inference, confidence assessment, state transition decision-making, and real-time audio-visual prompts for sensor data on the device side. At the same time, it uploads key events and logs to the cloud in real time through an encrypted link and has the ability to remotely update parameters and perform differential firmware flashing. The cloud service layer uses a high-throughput streaming processing framework to receive, persistently store, and index data from all terminals in partitions. It uses a group collaborative verification algorithm to perform correlation analysis on data from multiple nodes on the same work surface to suppress false alarms. It also pushes real-time curves and tiered prompts to web screens and mobile terminals through visualization interfaces and alarm APIs (application programming interfaces). The operations and data layer is responsible for version management, canary release and rollback of algorithm models, collecting and encrypting terminal operation logs for fault prediction and maintenance work order dispatch, and continuously outputting device location, power consumption, sensor status and event information to the digital twin platform to achieve a closed loop of data throughout the entire lifecycle.

Images