Method and device for early warning of heat radiation risk of shielding clothes based on posture image, and medium

Abstract

The application provides a shielding clothes heatstroke risk early warning method and device based on posture images, and a medium. The method comprises the following steps: collecting image data of a user, collecting first temperature data inside the shielding clothes, and collecting second temperature data of an electric power environment; identifying posture data of the user in each frame of image data; fusing into original multi-modal time sequence features in a feature fusion module; inputting the original multi-modal time sequence features into a classifier to detect a heatstroke risk value of the user; and performing a warning operation on live working of the user according to the heatstroke risk value. The shielding clothes is avoided to be heavy, thick, and poor in air permeability, and the burden on the working personnel is heavy. The working personnel takes off the shielding clothes to rest every certain time when carrying out live working, and the mode is recycled until the live working is completed. The mode mainly depends on subjective feeling of the working personnel, the subjective feeling of the working personnel is relatively slow, and heatstroke problems are prone to occur.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application belongs to the technical field of deep learning, and particularly relates to a shielding suit heatstroke risk early warning method and device based on posture images and a medium. BACKGROUND

[0002] In summer and other seasons with high temperature, operating personnel often face the "three high" working environment of high voltage, high altitude and high temperature when they carry out live working such as operation and maintenance and repair work on ultra-high voltage transmission lines. In order to protect the personal safety of the operating personnel during live working, the operating personnel wear shielding suits made of special shielding materials to complete the work. The shielding suit is heavy and has poor air permeability, and the operating personnel have a heavy burden. Therefore, the operating personnel take off the shielding suit to rest every certain time when they carry out live working, and the cycle continues until the live working is completed. There are some related shielding technologies on the market at present.

[0003] For example, Chinese invention CN201911057582.6 discloses a power transmission line live working intelligent shielding suit sensing system and method. The system carries out a shielding suit inspection process before working through a sensing module, a self-checking and state evaluation module and the like. The system can give real-time early warning and prompt for data such as safety distance, electric field intensity, temperature and humidity and environmental data. However, the technology cannot judge the shielding effect according to the differences of the operators themselves.

[0004] For another example, patent CN202311083642.8 discloses a test method and system for a heat-resistant shielding suit test electrode. The system includes obtaining an electrode placement guide signal generated after a preset test electrode contact sheet is placed with a test electrode to be tested, placing the test electrode to be tested at a test electrode test point, obtaining a current positioning electrode image, generating an actual test pulling frequency, generating an initial insulation test result, and obtaining a stable image of a sensing area of the test electrode to be tested according to an electrode sensing test indication, so as to accurately test the heat-resistant shielding suit test electrode. However, the existing shielding suit sensing system can only give real-time early warning and prompt for data such as electric field intensity, temperature and humidity and environmental data, and cannot effectively cope with the subjective feelings of the operating personnel. The subjective feelings of the operating personnel are relatively slow, and the operating personnel are prone to heatstroke problems. SUMMARY

[0005] Therefore, the embodiment of the present application provides a shielding suit heatstroke risk early warning method and device based on posture images, so as to give heatstroke risk early warning for operating personnel who wear shielding suits to carry out live working.

[0006] A first aspect of the embodiment of the present application provides a shielding suit heatstroke risk early warning method based on posture images, including:

[0007] When a user wearing a shielding suit performs live-line work in a power environment, image data of the user, first temperature data inside the shielding suit, and second temperature data of the power environment are synchronously collected at each moment;

[0008] Posture data of the user is identified in each frame of the image data.

[0009] A risk detection network is determined; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module, and a classifier.

[0010] Target action time sequence features are extracted by inputting the posture data into the first branch structure.

[0011] First target temperature time sequence features are extracted by inputting the first temperature data into the second branch structure.

[0012] Second target temperature time sequence features are extracted by inputting the second temperature data into the third branch structure.

[0013] The target action time sequence features, the first target temperature time sequence features, and the second target temperature time sequence features are fused into original multi-modal time sequence features by inputting them into the feature fusion module.

[0014] The original multi-modal time sequence features are input into the classifier to detect a heat radiation risk value of the user.

[0015] A warning operation is performed on the live-line work of the user according to the heat radiation risk value.

[0016] A second aspect of an embodiment of the application provides a shielding suit heat radiation risk warning device based on posture images, including:

[0017] A data collection module is configured to, when a user wearing a shielding suit performs live-line work in a power environment, synchronously collect image data of the user, first temperature data inside the shielding suit, and second temperature data of the power environment at each moment.

[0018] A posture data identification module is configured to identify posture data of the user in each frame of the image data.

[0019] A risk detection network determination module is configured to determine a risk detection network; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module, and a classifier.

[0020] An action time sequence feature extraction module is configured to extract target action time sequence features by inputting the posture data into the first branch structure.

[0021] The first temperature time sequence feature extraction module is configured to input the first temperature data into the second branch structure to extract first target temperature time sequence features.

[0022] The second temperature time sequence feature extraction module is configured to input the second temperature data into the third branch structure to extract second target temperature time sequence features.

[0023] The multi-modal time sequence feature fusion module is configured to input the target action time sequence features, the first target temperature time sequence features and the second target temperature time sequence features into the feature fusion module to fuse the features into original multi-modal time sequence features.

[0024] The heat stroke risk value detection module is configured to input the original multi-modal time sequence features into the classifier to detect a heat stroke risk value of the user.

[0025] The early warning operation execution module is configured to perform an early warning operation on the live-line work of the user according to the heat stroke risk value.

[0026] A third aspect of the embodiments of the present application provides a terminal device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the heat stroke risk early warning method based on posture images of the first aspect.

[0027] A fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the heat stroke risk early warning method based on posture images of the first aspect.

[0028] A fifth aspect of the embodiments of the present application provides a computer program product, which, when executed on a computer, causes the computer to perform the heat stroke risk early warning method based on posture images of the first aspect.

[0029] In the embodiment, when a user wears a shielding suit to perform live-line work in an electric power environment, image data of the user, first temperature data inside the shielding suit and second temperature data of the electric power environment are synchronously collected at each moment; posture data of the user is recognized in each frame of image data; a risk detection network is determined; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module and a classifier; the posture data is input into the first branch structure to extract target action time sequence features; the first temperature data is input into the second branch structure to extract first target temperature time sequence features; the second temperature data is input into the third branch structure to extract second target temperature time sequence features; the target action time sequence features, the first target temperature time sequence features and the second target temperature time sequence features are input into the feature fusion module to be fused into original multi-modal time sequence features; the original multi-modal time sequence features are input into the classifier to detect a heatstroke risk value of the user; and a warning operation is performed on the live-line work of the user according to the heatstroke risk value. In the embodiment, multi-modal features related to the body temperature of the user, such as the internal temperature of the shielding suit, the environmental temperature and the action of the user, are selected to detect the heatstroke risk of the user, so that the live-line work of the user wearing the shielding suit is warned in time, the probability of occurrence of the heatstroke problem of the user is effectively reduced, and the safety of the live-line work of the user wearing the shielding suit is improved. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0031] Figure 1 is a schematic diagram of a shielding suit heatstroke risk early warning method based on posture images provided by the embodiment of the present application;

[0032] Figure 2 is a schematic diagram of a risk detection network provided by the embodiment of the present application;

[0033] Figure 3 is a schematic diagram of a shielding suit heatstroke risk early warning device based on posture images provided by the embodiment of the present application;

[0034] Figure 4 is a schematic diagram of a terminal device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0035] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, technologies, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0036] The technical solutions of the present application are described below through specific embodiments.

[0037] Referring to Figure 1 , a schematic diagram of a heatstroke risk early warning method based on posture images of a shielding suit is shown, which can specifically include the following steps:

[0038] Step 101, when a user wears a shielding suit to perform live-line work in a power environment, image data of the user, first temperature data inside the shielding suit, and second temperature data of the power environment are synchronously collected at each time.

[0039] A first temperature sensor is arranged inside the shielding suit, a camera and a second temperature sensor are deployed on site in the power environment (especially in an ultra-high voltage environment), the first temperature sensor and the second temperature sensor are both connected to an edge computing node in a wireless manner (such as Bluetooth, WiFi (Wireless Fidelity), etc.), and the camera is connected to the edge computing node in a wired manner.

[0040] Generally, the edge computing node is a terminal device with strong computing capability, such as a computer, a server, or an embedded device, etc. A graphics processing unit (GPU) or an embedded neural network processor (NPU) is equipped in the edge computing node.

[0041] In the scenario of detecting heatstroke risk of the user, the edge computing node refers to constructing a new service platform close to the live-line work site (i.e. the network edge), providing storage, computing, network, etc. resources, sinking part of the key business applications (i.e. detecting heatstroke risk of the user) to the access network edge, so as to reduce the width and delay loss caused by network transmission and multi-level forwarding. The edge computing node is located between the user and the cloud (server), and is closer to the user (data source) than the traditional cloud, has the characteristics of miniaturization, distribution, and closeness to the user. Massive data (such as audio data) no longer needs to be uploaded to the cloud for processing, and the processing of data is realized on the network edge side, reducing the request response time, reducing the network bandwidth while ensuring the safety and privacy of data.

[0042] When the user wears the shielding suit to perform live-line work in the power environment, the camera is called synchronously at each time to track the user and collect multiple frames of image data, the first temperature sensor is called to collect multiple first temperature data inside the shielding suit, and the second temperature sensor is called to collect multiple second temperature data of the power environment.

[0043] The multiple frames of image data are sorted according to the time of collection and stored in the first cache queue.

[0044] The multiple first temperature data are sorted according to the time of collection and stored in the second cache queue.

[0045] The multiple second temperature data are sorted according to the time of collection and stored in the third cache queue.

[0046] In actual application, the first temperature data inside the shielding suit directly affects the body temperature of the user, the second temperature data of the environment indirectly affects the body temperature of the user, and the posture data of the user to some extent reflects the body temperature of the user, that is, the action made by the user to the environment inside the shielding suit and the environment outside the shielding suit under the established body state (including the body temperature) of the user, and these actions also reversely affect the body temperature of the user and may make the body temperature of the user rise. The data of the three modalities are highly correlated with the body temperature of the user.

[0047] Step 102, identifying the posture data of the user in each frame of image data.

[0048] A general posture detection network such as CPM (Convolutional Pose Machine) and HRNet (High-Resolution Network) is built in the edge computing node, each frame of image data in the image sequence is input into the posture detection network for human pose estimation (Human Pose Estimation), and the posture data of the user at each time is obtained. At this time, the posture data at each time is a multi-dimensional vector.

[0049] Considering that the time requirement for detecting the heat radiation risk of the user is high and the accuracy requirement of subsequent operation on the posture data is low, 15 joint nodes can be selected to represent the posture data of the human body, the operation amount is reduced under the condition of meeting the accuracy, and the real-time performance of detecting the heat radiation risk of the user is improved. Therefore, the posture data at each time is a 15-dimensional vector.

[0050] The multiple frames of posture data are sorted according to the time of collection and stored in the fourth cache queue.

[0051] Step 103, determining a risk detection network.

[0052] In the embodiment, the risk detection network can be constructed and trained in advance in an offline environment.

[0053] As shown in Figure 2 , the risk detection network has a first backbone structure Backbone_1, a second backbone structure Backbone_2, a third backbone structure Backbone_3, a feature fusion module (FFM), and a classifier Classification.

[0054] In training the risk detection network, the first temperature data, the second temperature data, and the posture data of the user when wearing the shielding clothes to perform live-line work in the power environment can be collected as samples, the user's confirmed state (such as comfortable, uncomfortable) as a label Label, and cross entropy as a loss function. The risk detection network is supervised and verified, and when the risk detection network is trained and verified to meet the performance requirements, the risk detection network can be deployed to the edge computing node.

[0055] Further, the physiological information (such as body temperature, degree of consciousness, etc.) of the user can be detected, and the user's confirmed state can be corrected to improve the accuracy of the label.

[0056] When the user wears the shielding clothes to perform live-line work in the power environment, the risk detection network can be started to run.

[0057] Step 104, input the posture data into the first backbone structure to extract the target action time sequence feature.

[0058] In the embodiment, as shown in Figure 2 , part of the posture data Pose can be input into the first backbone structure Backbone_1 to extract features in sequence, which is recorded as the target action time sequence feature.

[0059] In one design, as shown in Figure 2 , the first backbone structure Backbone_1 has a first long short-term memory network LSTM_1, a first self-attention layer Self-Attention, and a first fully connected layer FC_1.

[0060] In this design, a first sliding window is added to the posture data Pose to obtain a posture sequence.

[0061] The posture sequence is input into the first long short-term memory network LSTM_1 to extract the first candidate action time sequence feature.

[0062] The first candidate action time sequence feature is input into the first self-attention layer Self-Attention_1 to convert it into the second candidate action time sequence feature.

[0063] The first self-attention layer, Self-Attention_1, provides a self-attention mechanism to capture important features in the pose sequence. It also helps to alleviate the forgetting problem of the first long short-term memory network, LSTM_1, in long-term prediction. Since the first long short-term memory network, LSTM_1, processes the pose sequence step by step, its positional information is preserved. Therefore, after processing by the first long short-term memory network, LSTM_1, no additional positional information is added, and it can be directly input into the first self-attention layer, Self-Attention_1. This combined structure integrates the complementary functions of long short-term memory network and self-attention mechanism, effectively processing long-period temporal pose sequences.

[0064] Furthermore, adding a self-attention mechanism facilitates the subsequent use of attention mechanisms for feature fusion.

[0065] The temporal features of the second candidate action are input into the first fully connected layer FC_1 and mapped to the temporal features of the target action to achieve dimensional alignment.

[0066] Step 105: Input the first temperature data into the second branch structure to extract the first target temperature time series features.

[0067] In this embodiment, as Figure 2 As shown, a portion of the first temperature data Temperature_1 can be sequentially input into the second branch structure Backbone_2 to extract features, which are denoted as the first target temperature time series features.

[0068] In a design, such as Figure 2 As shown, the second branch structure has a first bidirectional long short-term memory network Bi-LSTM_1, a second long short-term memory network LSTM_2, and a second fully connected layer FC_2.

[0069] In this design, the first temperature data Temperature_1 at each time step is iterated, and the first temperature data Temperature_1 at the previous time step is subtracted from the first temperature data Temperature_1 at the current time step to obtain the first difference data Diff_T1. This process can be expressed as ΔT1 = ΔT 1,t -ΔT 1,t-1 Where ΔT1 is the first difference data, ΔT 1,t The first temperature data at time t is Temperature_1, ΔT 1,t-1 Let Temperature_1 be the first temperature data at time t-1, where t is a positive integer and t≥2.

[0070] Multiple first difference data points Diff_T1 are sorted according to the time of acquisition and stored in the fifth buffer queue.

[0071] Add a sliding second window to the first temperature data Temperature_1 to obtain the first internal temperature sequence.

[0072] A sliding second window is added to the first difference data Diff_T1 to obtain the second internal temperature sequence.

[0073] The first internal temperature sequence is input into the first bidirectional long short-term memory network Bi-LSTM_1 to extract the first internal candidate temperature features.

[0074] The second internal temperature sequence is input into the second long short-term memory network LSTM_2 to extract the second internal candidate temperature features.

[0075] Among them, the first bidirectional long short-term memory network Bi-LSTM_1 can take into account past and future contextual information and mine important trend features in terms of heat radiation from the original first temperature data Temperature_1. The first difference data Diff_T1 itself is one of the trends of the first temperature data Temperature_1, and the features can be mined using a single-layer second long short-term memory network LSTM_2.

[0076] The first internal candidate temperature feature and the second internal candidate temperature feature are concatenated to form the third internal candidate temperature feature.

[0077] The third internal candidate temperature feature is input into the second fully connected layer FC_2 and mapped to the first target temperature time series feature to achieve dimensional alignment.

[0078] Step 106: Input the second temperature data into the third branch structure to extract the second target temperature time series features.

[0079] In this embodiment, as Figure 2 As shown, a portion of the second temperature data Temperature_2 can be sequentially input into the third branch structure Backbone_3 to extract features, which are denoted as the second target temperature time series features.

[0080] In a design, such as Figure 2 As shown, the third branch structure Backbone_3 contains a second bidirectional long short-term memory network Bi-LSTM_2, a third long short-term memory network LSTM_3, and a third fully connected layer FC_3.

[0081] In this design, the second temperature data Temperature_2 at each time step is iterated, and the second temperature data Temperature_2 at the previous time step is subtracted from the second temperature data Temperature_2 at the current time step to obtain the second difference data Diff_T2. This process can be expressed as ΔT2 = ΔT2,t -ΔΔT 2,t-1 wherein, ΔT2 is the second differential data, Δ Δ T 2,t is the second temperature data Temperature_2 at time t, Δ Δ T 2,t-1 is the second temperature data Temperature_2 at time t-1, t is a positive integer, t≥2.

[0082] The plurality of second differential data Diff_T2 are sorted according to the collection time and stored in the sixth cache queue.

[0083] A third window is added to the second temperature data Temperature_2 to obtain a first ambient temperature sequence.

[0084] A third window is added to the second differential data Diff_T2 to obtain a second ambient temperature sequence.

[0085] The first ambient temperature sequence is input into a second bidirectional long short-term memory network Bi-LSTM_2 to extract first ambient candidate temperature features.

[0086] The second ambient temperature sequence is input into a third long short-term memory network LSTM_3 to extract second ambient candidate temperature features.

[0087] wherein, the second bidirectional long short-term memory network Bi-LSTM_2 can consider past and future context information, and mine important trend features in heat radiation from the original second temperature data Temperature_2, and the second differential data Diff_T2 itself is one of the trends of the second temperature data Temperature_2, and a single-layer third long short-term memory network LSTM_3 is used to mine features.

[0088] The first ambient candidate temperature features and the second ambient candidate temperature features are spliced Concat into third ambient candidate temperature features.

[0089] The third ambient candidate temperature features are input into a third fully connected layer FC_3 to be mapped into second target temperature time sequence features.

[0090] Further, considering that the temperature change rate inside the shielding suit is large, the user's action change rate is moderate, and the temperature change rate of the environment is small, therefore, the width of the third window is greater than the width of the first window, and the width of the first window is greater than the width of the second window.

[0091] Step 107, input the target action time sequence features, the first target temperature time sequence features and the second target temperature time sequence features into a feature fusion module to be fused into original multi-modal time sequence features.

[0092] In the embodiment, as shown in Figure 2 The target action time sequence feature, the first target temperature time sequence feature, and the second target temperature time sequence feature can be input into a feature fusion module FFM, and the target action time sequence feature, the first target temperature time sequence feature, and the second target temperature time sequence feature are interacted and fused into original multi-modal time sequence features.

[0093] In one design, the feature fusion module FFM has a second self-attention layer Self-Attention_2, a third self-attention layer Self-Attention_3, a first attention layer Attention_1, a second attention layer Attention_2, a third attention layer Attention_3, and a fourth attention layer Attention_4.

[0094] Generally, the user's action has obvious burstiness, so that the posture data is burst data, and the temperature inside the shielding clothes and the temperature of the environment both have obvious trendiness, so that the first temperature data and the second temperature data are both trend data, which can be learned and predicted using deep learning.

[0095] In addition, the user's action is the user's reaction to the temperature inside the shielding clothes and the temperature of the environment, so that the posture data has the effect of enhancing the first temperature data and the second temperature data.

[0096] In the design, the first bimodal time sequence feature F1 and the second bimodal time sequence feature F2 can be initialized in a random manner.

[0097] The first bimodal time sequence feature is input into the second self-attention layer Self-Attention_2, and is converted into a first candidate modal time sequence feature under the self-attention mechanism.

[0098] The first target temperature time sequence feature and the first candidate modal time sequence feature F1 are input into the first attention layer Attention_1, and the first target temperature time sequence feature and the first candidate modal time sequence feature are fused into a second candidate modal time sequence feature according to the attention weight of the first target temperature time sequence feature on the first candidate modal time sequence feature.

[0099] The target action time sequence feature and the second candidate modal time sequence feature are input into the second attention layer Attention_2, and the target action time sequence feature and the second candidate modal time sequence feature are fused into a third candidate modal time sequence feature according to the attention weight of the target action time sequence feature on the second candidate modal time sequence feature.

[0100] The second bimodal time sequence feature F2 is input into a third self-attention layer Self-Attention_3, and is converted into a fourth candidate modal time sequence feature under the self-attention mechanism.

[0101] The second target temperature time sequence feature and the fourth candidate modal time sequence feature are input into a third attention layer Attention_3, and the second target temperature time sequence feature and the fourth candidate modal time sequence feature are fused into a fifth candidate modal time sequence feature according to the attention weight of the second target temperature time sequence feature relative to the fourth candidate modal time sequence feature.

[0102] The target action time sequence feature and the fifth candidate modal time sequence feature are input into a fourth attention layer Attention_4, and the target action time sequence feature and the fifth candidate modal time sequence feature are fused into a sixth candidate modal time sequence feature according to the attention weight of the target action time sequence feature relative to the fifth candidate modal time sequence feature.

[0103] The third candidate modal time sequence feature and the sixth candidate modal time sequence feature are spliced Concat into the original multi-modal time sequence feature.

[0104] Step 108, input the original multi-modal time sequence feature into the classifier to detect the heat risk value of the user.

[0105] In the embodiment, as shown in Figure 2 , the original multi-modal time sequence feature can be input into the classifier Classification to perform a classification task, and the heat risk value of the user is detected. The heat risk value is the probability that the user has heat.

[0106] In one design, as shown in Figure 2 , the classifier Classification is a Seq2Seq (Sequence to Sequence) architecture, which includes an encoder Encoder, a decoder Decoder, and a head structure Head.

[0107] The encoder Encoder and the decoder Decoder can use structures such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Units), and the head structure Head can use structures such as multi-layer FC (Fully Connected Layer) and Sigmoid (Sigmoid function).

[0108] In the design, the original multi-modal time sequence feature is input into the encoder Encoder to be encoded into a candidate multi-modal time sequence feature.

[0109] The candidate multimodal time sequence feature is input into a decoder Decoder to be decoded into a target multimodal time sequence feature.

[0110] The target multimodal time sequence feature is input into a head structure Head to generate a heat stroke risk value for the user.

[0111] Step 109, performing a pre-warning operation on the live working of the user according to the heat stroke risk value.

[0112] In actual application, the heat stroke risk value can be analyzed by using a threshold method, a trend method, etc., so as to perform a pre-warning operation on the live working of the user, and when the user has a potential risk of heat stroke, the live working of the user is stopped in advance, and the shielding clothes are taken off for rest.

[0113] In an embodiment of the present application, step 109 can include the following steps:

[0114] Step 1091, calculating an average value of the plurality of heat stroke risk values as a sliding risk value.

[0115] In the embodiment, a fourth window that can slide is added to the heat stroke risk value to obtain a risk sequence, and an average value of the plurality of heat stroke risk values in the risk sequence is calculated as a sliding risk value.

[0116] Step 1092, if the sliding risk value is greater than or equal to a preset first risk threshold, a risk monitoring mode is started.

[0117] In the embodiment, the sliding risk value is compared with the preset first risk threshold, and if the sliding risk value is greater than or equal to the first risk threshold, it indicates that the sliding risk value is high and the user has a potential risk of heat stroke, but considering that the risk detection network has a certain error, the risk monitoring mode can be started to further monitor the user and reduce the interference to the live working of the user.

[0118] Step 1093, in the risk monitoring mode, a monitoring time range is set, and the first risk threshold is attenuated according to the length of the monitoring time range to obtain a second risk threshold.

[0119] In the risk monitoring mode, a dynamic monitoring time range can be set starting from the time when the sliding risk value is greater than or equal to the first risk threshold, and the length of the monitoring time range is negatively correlated with the sliding risk value, that is, the lower the sliding risk value, the greater the length of the monitoring time range, and vice versa, the higher the sliding risk value, the smaller the length of the monitoring time range.

[0120] In addition, the first risk threshold is attenuated according to the length of the monitoring time range to obtain a second risk threshold, that is, the second risk threshold is lower than the first risk threshold.

[0121] The dynamic monitoring time range combined with the dynamic second risk threshold can improve the sensitivity of monitoring and ensure the safety of the user.

[0122] Exemplarily, the length of the monitoring time range and the first risk threshold are substituted into the following formula to obtain the second risk threshold:

[0123]

[0124] wherein H1 is the first risk threshold, H2 is the second risk threshold, t1 is the time stamp of the set monitoring time range, t2 is the current time stamp, w is the length of the monitoring time range, and a is an adjustment coefficient for adjusting the degree of attenuation of the first risk threshold.

[0125] Step 1094, if any of the sliding risk values is greater than or equal to the second risk threshold within the monitoring time range, a pre-warning operation is performed on the live working of the user.

[0126] Step 1095, if all the sliding risk values are less than the second risk threshold within the monitoring time range, the risk monitoring mode is cancelled.

[0127] In this embodiment, the heatstroke risk values output by the risk detection network are continuously used, and the plurality of heatstroke risk values are smoothed to obtain the sliding risk values.

[0128] Within the monitoring time range, the sliding risk values are not compared with the original first risk threshold, but compared with the second risk threshold after attenuation.

[0129] If any of the sliding risk values is greater than or equal to the second risk threshold, a pre-warning operation is performed on the live working of the user, prompting the user to stop the live working and take off the shielding clothes for rest.

[0130] If all the sliding risk values are less than the second risk threshold, it indicates that the previous high sliding risk value is a false alarm, and the risk monitoring mode is cancelled to restore the normal monitoring mode, at this time, the sliding risk values are compared with the original first risk threshold.

[0131] In the embodiment, when a user wears a shielding suit to perform live working in a power environment, image data of the user, first temperature data inside the shielding suit and second temperature data of the power environment are synchronously collected at each moment; posture data of the user is identified in each frame of image data; a risk detection network is determined; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module and a classifier; the posture data is input into the first branch structure to extract target action time sequence features; the first temperature data is input into the second branch structure to extract first target temperature time sequence features; the second temperature data is input into the third branch structure to extract second target temperature time sequence features; the target action time sequence features, the first target temperature time sequence features and the second target temperature time sequence features are input into the feature fusion module to be fused into original multi-modal time sequence features; the original multi-modal time sequence features are input into the classifier to detect a heatstroke risk value of the user; and a warning operation is performed on live working of the user according to the heatstroke risk value. In the embodiment, multi-modal features related to body temperature of the user, such as internal temperature of the shielding suit, environmental temperature and action of the user, are selected to detect the heatstroke risk of the user, so that live working of the user wearing the shielding suit is warned in time, the probability of occurrence of heatstroke of the user is effectively reduced, and the safety of live working of the user wearing the shielding suit is improved.

[0132] It should be noted that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0133] Referring to Figure 3 , a schematic diagram of a shielding suit heatstroke risk warning device based on posture images provided by an embodiment of the application is shown, which can specifically include the following modules:

[0134] The data collection module 301 is configured to synchronously collect image data of the user, first temperature data inside the shielding suit and second temperature data of the power environment at each moment when the user wears the shielding suit to perform live working in the power environment.

[0135] The posture data identification module 302 is configured to identify posture data of the user in each frame of image data.

[0136] The risk detection network determination module 303 is configured to determine a risk detection network; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module and a classifier.

[0137] The action time sequence feature extraction module 304 is configured to input the posture data into the first branch structure to extract target action time sequence features.

[0138] The first temperature time sequence feature extraction module 305 is configured to input the first temperature data into the second branch structure to extract first target temperature time sequence features.

[0139] The second temperature time sequence feature extraction module 306 is configured to input the second temperature data into the third branch structure to extract second target temperature time sequence features.

[0140] The multi-modal time sequence feature fusion module 307 is configured to input the target action time sequence features, the first target temperature time sequence features and the second target temperature time sequence features into the feature fusion module to fuse the features into original multi-modal time sequence features.

[0141] The heat stroke risk value detection module 308 is configured to input the original multi-modal time sequence features into the classifier to detect a heat stroke risk value of the user.

[0142] The early warning operation execution module 309 is configured to perform early warning operation on the live working of the user according to the heat stroke risk value.

[0143] In an embodiment of the present application, the first branch structure comprises a first long short-term memory network, a first self-attention layer and a first full connection layer.

[0144] The action time sequence feature extraction module 304 is further configured to:

[0145] add a first window to the posture data to obtain a posture sequence;

[0146] input the posture sequence into the first long short-term memory network to extract first candidate action time sequence features;

[0147] input the first candidate action time sequence features into the first self-attention layer to convert the features into second candidate action time sequence features;

[0148] input the second candidate action time sequence features into the first full connection layer to map the features into target action time sequence features.

[0149] In an embodiment of the present application, the second branch structure comprises a first bidirectional long short-term memory network, a second long short-term memory network and a second full connection layer.

[0150] The first temperature time sequence feature extraction module 305 is further configured to:

[0151] subtract the first temperature data at a previous time from the first temperature data at a current time to obtain first differential data;

[0152] add a second window to the first temperature data to obtain a first internal temperature sequence;

[0153] add a second window to the first difference data to obtain a second internal temperature sequence;

[0154] input the first internal temperature sequence into the first bidirectional long short-term memory network to extract first internal candidate temperature features;

[0155] input the second internal temperature sequence into the second long short-term memory network to extract second internal candidate temperature features;

[0156] concatenate the first internal candidate temperature features and the second internal candidate temperature features into third internal candidate temperature features;

[0157] input the third internal candidate temperature features into the second fully connected layer to map into first target temperature time sequence features.

[0158] In an embodiment of the present application, the third branch structure has a second bidirectional long short-term memory network, a third long short-term memory network and a third fully connected layer;

[0159] The second temperature time sequence feature extraction module 306 is further configured to:

[0160] subtract the second temperature data at the last moment from the second temperature data at the current moment to obtain second difference data;

[0161] add a third window to the second temperature data to obtain a first ambient temperature sequence;

[0162] add a third window to the second difference data to obtain a second ambient temperature sequence;

[0163] input the first ambient temperature sequence into the second bidirectional long short-term memory network to extract first ambient candidate temperature features;

[0164] input the second ambient temperature sequence into the third long short-term memory network to extract second ambient candidate temperature features;

[0165] concatenate the first ambient candidate temperature features and the second ambient candidate temperature features into third ambient candidate temperature features;

[0166] input the third ambient candidate temperature features into the third fully connected layer to map into second target temperature time sequence features;

[0167] wherein the width of the third window is greater than the width of the first window, and the width of the first window is greater than the width of the second window.

[0168] In an embodiment of the present application, the feature fusion module has a second self-attention layer, a third self-attention layer, a first attention layer, a second attention layer, a third attention layer, and a fourth attention layer;

[0169] The multi-modal time-series feature fusion module 307 is further configured to:

[0170] initialize a first dual-modal time-series feature and a second dual-modal time-series feature;

[0171] input the first dual-modal time-series feature into the second self-attention layer to convert the first dual-modal time-series feature into a first candidate modal time-series feature;

[0172] input the first target temperature time-series feature and the first candidate modal time-series feature into the first attention layer to fuse the first target temperature time-series feature and the first candidate modal time-series feature into a second candidate modal time-series feature;

[0173] input the target action time-series feature and the second candidate modal time-series feature into the second attention layer to fuse the target action time-series feature and the second candidate modal time-series feature into a third candidate modal time-series feature;

[0174] input the second dual-modal time-series feature into the third self-attention layer to convert the second dual-modal time-series feature into a fourth candidate modal time-series feature;

[0175] input the second target temperature time-series feature and the fourth candidate modal time-series feature into the third attention layer to fuse the second target temperature time-series feature and the fourth candidate modal time-series feature into a fifth candidate modal time-series feature;

[0176] input the target action time-series feature and the fifth candidate modal time-series feature into the fourth attention layer to fuse the target action time-series feature and the fifth candidate modal time-series feature into a sixth candidate modal time-series feature;

[0177] concatenate the third candidate modal time-series feature and the sixth candidate modal time-series feature into an original multi-modal time-series feature.

[0178] In an embodiment of the present application, the classifier includes an encoder, a decoder, and a head structure;

[0179] The heatstroke risk value detection module 308 includes:

[0180] an encoding module configured to input the original multi-modal time-series feature into the encoder to encode the original multi-modal time-series feature into a candidate multi-modal time-series feature;

[0181] a decoding module configured to input the candidate multi-modal time-series feature into the decoder to decode the candidate multi-modal time-series feature into a target multi-modal time-series feature;

[0182] a classification module configured to input the target multi-modal time-series feature into the head structure to generate a heatstroke risk value for the user.

[0183] In an embodiment of the present application, the pre-warning operation execution module 309 comprises:

[0184] a sliding risk value calculation module, configured to calculate an average of the plurality of heatstroke risk values as a sliding risk value;

[0185] a risk monitoring mode starting module, configured to start a risk monitoring mode if the sliding risk value is greater than or equal to a preset first risk threshold value;

[0186] a threshold value attenuation module, configured to set a monitoring time range in the risk monitoring mode, and attenuate the first risk threshold value according to a length of the monitoring time range to obtain a second risk threshold value; the length of the monitoring time range is negatively correlated with the sliding risk value;

[0187] a sliding risk pre-warning module, configured to perform a pre-warning operation on the live working of the user if any of the sliding risk values is greater than or equal to the second risk threshold value within the monitoring time range;

[0188] a risk monitoring mode canceling module, configured to cancel the risk monitoring mode if all the sliding risk values are less than the second risk threshold value within the monitoring time range.

[0189] In an embodiment of the present application, the threshold value attenuation module is further configured to:

[0190] substitute the length of the monitoring time range and the first risk threshold value into the following formula to obtain the second risk threshold value:

[0191]

[0192] wherein H1 is the first risk threshold value, H2 is the second risk threshold value, t1 is a timestamp for setting the monitoring time range, t2 is a current timestamp, w is the length of the monitoring time range, and a is an adjustment coefficient. 11

[0193] The application embodiment provides a shielding suit heatstroke risk pre-warning device based on a posture image, and application of the device can realize each step in each method embodiment.

[0194] For the device embodiment, it is basically similar to the method embodiment, so it is described more simply, and the related parts refer to the description in the method embodiment part.

[0195] Referring to Figure 4 , a schematic diagram of a terminal device provided by an embodiment of the present application is shown. As shown in Figure 4 ​As shown, the terminal device 400 in the embodiments of the present application includes a processor 410, a memory 420, and a computer program 421 stored in the memory 420 and executable on the processor 410. The processor 410 implements the steps in the above-mentioned posture image-based shielding suit heat radiation risk early warning method embodiments when executing the computer program 421. Alternatively, the processor 410 implements the functions of each module / unit in the above-mentioned device embodiments when executing the computer program 421.

[0196] For example, the computer program 421 can be divided into one or more modules / units, which are stored in the memory 420 and executed by the processor 410 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which can be used to describe the execution process of the computer program 421 in the terminal device 400.

[0197] The terminal device 400 can include, but is not limited to, the processor 410 and the memory 420. Those skilled in the art can understand that the terminal device 400 can include more or fewer components, or combine certain components, or different components, for example, the terminal device 400 can also include an input / output device, a network access device, a bus, etc. Figure 4 The terminal device 400 is only an example and does not constitute a limitation on the terminal device 400, and can include more or fewer components than the illustration, or combine certain components, or different components, for example, the terminal device 400 can also include an input / output device, a network access device, a bus, etc.

[0198] The processor 410 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0199] The memory 420 can be an internal storage unit of the terminal device 400, for example, a hard disk or a memory of the terminal device 400. The memory 420 can also be an external storage device of the terminal device 400, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like equipped on the terminal device 400. Further, the memory 420 can also include both the internal storage unit and the external storage device of the terminal device 400. The memory 420 is used to store the computer program 421 and other programs and data required by the terminal device 400. The memory 420 can also be used to temporarily store data that has been output or is to be output.

[0200] The embodiment of the present application further discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the heatstroke risk early warning method based on posture images of a shielding suit when executing the computer program.

[0201] The embodiment of the present application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the heatstroke risk early warning method based on posture images of a shielding suit.

[0202] The embodiment of the present application further discloses a computer program product, which, when running on a computer, enables the computer to execute the heatstroke risk early warning method based on posture images of a shielding suit.

[0203] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A heat radiation risk early warning method for a shielding suit based on a posture image, characterized by, The method comprises the following steps: synchronously collecting image data of a user, first temperature data inside a shielding suit of the user, and second temperature data of an electric power environment at each moment when the user performs live-line work in the electric power environment while wearing the shielding suit; identifying posture data of the user in each frame of the image data; determining a risk detection network; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module, and a classifier; inputting the posture data into the first branch structure to extract target action time sequence features; inputting the first temperature data into the second branch structure to extract first target temperature time sequence features; inputting the second temperature data into the third branch structure to extract second target temperature time sequence features; inputting the target action time sequence features, the first target temperature time sequence features, and the second target temperature time sequence features into the feature fusion module to fuse them into original multi-modal time sequence features; inputting the original multi-modal time sequence features into the classifier to detect a heat stroke risk value of the user; performing a warning operation on live-line work of the user according to the heat stroke risk value; the performing of the warning operation on the live-line work of the user according to the heat stroke risk value comprises: calculating an average value of a plurality of the heat stroke risk values as a sliding risk value; if the sliding risk value is greater than or equal to a preset first risk threshold, a risk monitoring mode is started; in the risk monitoring mode, a monitoring time range is set, and the first risk threshold is attenuated according to a length of the monitoring time range to obtain a second risk threshold; the length of the monitoring time range is negatively correlated with the sliding risk value; if any of the sliding risk values is greater than or equal to the second risk threshold within the monitoring time range, a warning operation is performed on the live-line work of the user; if all of the sliding risk values are less than the second risk threshold within the monitoring time range, the risk monitoring mode is cancelled.

2. The method of claim 1, wherein, the first branch structure has a first long short-term memory network, a first self-attention layer, and a first full connection layer; the inputting of the posture data into the first branch structure to extract target action time sequence features comprises: adding a first window to the posture data to obtain a posture sequence; the posture sequence is inputted into the first long short-term memory network to extract first candidate action time sequence features; the first candidate action time sequence features are inputted into the first self-attention layer to be converted into second candidate action time sequence features; the second candidate action time sequence features are inputted into the first full connection layer to be mapped into target action time sequence features.

3. The method of claim 2, wherein, the second branch structure has a first bidirectional long short-term memory network, a second long short-term memory network, and a second full connection layer; the inputting of the first temperature data into the second branch structure to extract first target temperature time sequence features comprises: subtracting the first temperature data at a previous moment from the first temperature data at a current moment to obtain first differential data; a second window is added to the first temperature data to obtain a first internal temperature sequence; adding a second window to the first differential data to obtain a second internal temperature sequence; inputting the first internal temperature sequence into the first bidirectional long short-term memory network to extract first internal candidate temperature features; inputting the second internal temperature sequence into the second long short-term memory network to extract second internal candidate temperature features; concatenating the first internal candidate temperature features and the second internal candidate temperature features into third internal candidate temperature features; inputting the third internal candidate temperature features into the second fully connected layer to map into first target temperature time sequence features.

4. The method of claim 3, wherein, the third branch structure has a second bidirectional long short-term memory network, a third long short-term memory network and a third fully connected layer; the second temperature data is inputted into the third branch structure to extract second target temperature time sequence features, which includes subtracting the second temperature data at the last time from the second temperature data at the current time to obtain second differential data; adding a third window to the second temperature data to obtain a first environmental temperature sequence; adding a third window to the second differential data to obtain a second environmental temperature sequence; inputting the first environmental temperature sequence into the second bidirectional long short-term memory network to extract first environmental candidate temperature features; inputting the second environmental temperature sequence into the third long short-term memory network to extract second environmental candidate temperature features; concatenating the first environmental candidate temperature features and the second environmental candidate temperature features into third environmental candidate temperature features; inputting the third environmental candidate temperature features into the third fully connected layer to map into second target temperature time sequence features; wherein the width of the third window is greater than the width of the first window, and the width of the first window is greater than the width of the second window.

5. The method of claim 1, wherein, the feature fusion module has a second self-attention layer, a third self-attention layer, a first attention layer, a second attention layer, a third attention layer and a fourth attention layer; the target action time sequence features, the first target temperature time sequence features and the second target temperature time sequence features are inputted into the feature fusion module to be fused into multi-modal time sequence features, which includes initializing first and second double modal time sequence features; the first double modal time sequence features are inputted into the second self-attention layer to be converted into first candidate modal time sequence features; the first target temperature time sequence features and the first candidate modal time sequence features are inputted into the first attention layer to be fused into second candidate modal time sequence features; the target action time sequence features and the second candidate modal time sequence features are inputted into the second attention layer to be fused into third candidate modal time sequence features; the second double modal time sequence features are inputted into the third self-attention layer to be converted into fourth candidate modal time sequence features; the second target temperature time sequence features and the fourth candidate modal time sequence features are inputted into the third attention layer to be fused into fifth candidate modal time sequence features; the target action time sequence features and the fifth candidate modal time sequence features are inputted into the fourth attention layer to be fused into sixth candidate modal time sequence features; The third candidate modality time sequence feature and the sixth candidate modality time sequence feature are spliced as original multi-modality time sequence features.

6. The method according to any one of claims 1-5, characterized in that, The classifier comprises an encoder, a decoder and a head structure; The inputting of the original multi-modality time sequence features into the classifier to detect the heat radiation risk value of the user comprises: inputting the original multi-modality time sequence features into the encoder to encode the original multi-modality time sequence features into candidate multi-modality time sequence features; The candidate multi-modality time sequence features are inputted into the decoder to decode the candidate multi-modality time sequence features into target multi-modality time sequence features; The target multi-modality time sequence features are inputted into the head structure to generate the heat radiation risk value of the user.

7. The method of claim 1, wherein, The attenuation of the first risk threshold according to the length of the monitoring time range comprises substituting the length of the monitoring time range and the first risk threshold into the following formula to obtain a second risk threshold: ; Wherein, H1 is the first risk threshold, H2 is the second risk threshold, t1 is a time stamp for setting the monitoring time range, t2 is a current time stamp, w is the length of the monitoring time range, and a is an adjustment coefficient.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the heat radiation risk early warning method based on posture images of a shielding suit according to any one of claims 1-7.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to realize the heat radiation risk early warning method based on posture images of a shielding suit according to any one of claims 1-8.

Images