Shielding clothes thermal radiation risk early warning method and device based on attitude image and medium

Abstract

The invention provides a shielding clothes thermal shooting risk early warning method and device based on a posture image and a medium. The method comprises the steps that image data are collected for a user, first temperature data in shielding clothes are collected for the shielding clothes, and second temperature data are collected for an electric power environment; identifying posture data of a user in each frame of image data; original multi-modal time sequence features are fused in the feature fusion module; inputting the original multi-modal time sequence characteristics into a classifier to detect a heat radiation risk value for a user; and performing early warning operation on the hot-line work of the user according to the heat radiation risk value. The shielding clothes are prevented from being thick and heavy in fabric, poor in air permeability and heavy in burden on operators, when the operators wear the shielding clothes to carry out hot-line work, the operators take off the shielding clothes to have a rest at certain intervals, and the operation is repeated until the hot-line work is completed, the mode mainly depends on subjective feelings of the operators, and the working efficiency is greatly improved. And the subjective feeling of operators on themselves is slow, and heat radiation is easy to occur.

Description

Technical Field

[0001] The embodiments of the present application belong to the technical field of deep learning, and in particular, relate to a method, device and medium for warning the risk of heat radiation of shielding clothing based on posture images. Background Art

[0002] When workers carry out live-line work in seasons with higher temperatures, such as the operation and maintenance of ultra-high voltage transmission lines, they often face a work environment with "three highs" - high voltage, high altitude and high temperature. In order to ensure personal safety during live-line work, workers wear shielding suits made of special shielding materials to complete their work. The shielding suits are thick and have poor air permeability, which puts a heavy burden on the workers. When workers wear shielding suits to carry out live-line work, they take off the shielding suits to rest at regular intervals, and repeat this cycle until the live-line work is completed. There are some related shielding technologies on the market:

[0003] For example, Chinese invention CN201911057582.6 discloses a perception system and method for intelligent shielding clothing for live working on transmission lines. The system performs a pre-work shielding clothing inspection process through a perception module, a self-inspection and a status assessment module, etc.; the system can provide real-time warnings and prompts for data such as safety distance, electric field strength, temperature and humidity, and environmental data, but the technology cannot judge the shielding effect based on the differences of the operator himself.

[0004] For another example, patent CN202311083642.8 discloses a test method and system for heat-resistant shielding clothing test electrodes, including obtaining an electrode placement conduction signal generated after placing a test electrode to be tested on a preset test electrode contact sheet, placing the test electrode to be tested at the test electrode test point, obtaining the current positioning electrode image, generating the actual test pull times, generating the initial insulation test result, and obtaining a stable image of the sensing area of ​​the test electrode to be tested according to the electrode sensing test indication, so as to accurately test the heat-resistant shielding clothing test electrode. However, the existing shielding clothing sensing system can provide real-time warnings and prompts for data such as electric field strength, temperature and humidity, and environmental data, and cannot effectively respond to the subjective feelings of operators. Operators are relatively slow in their subjective feelings and are prone to heat radiation problems. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a method, device and medium for warning the risk of thermal radiation of shielding clothing based on posture images, which are used to warn workers of the risk of thermal radiation when performing live work while wearing shielding clothing.

[0006] A first aspect of an embodiment of the present application provides a method for warning of heat radiation risk of shielding clothing based on posture images, comprising:

[0007] When a user wears a shielding suit and performs live work in an electric power environment, image data of the user, first temperature data inside the shielding suit, and second temperature data of the electric power environment are synchronously collected at each time;

[0008] Identifying the user's posture data in each frame of the image data;

[0009] Determine a risk detection network; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module and a classifier;

[0010] Inputting the posture data into the first branch structure to extract the target action timing features;

[0011] Inputting the first temperature data into the second branch structure to extract the first target temperature time series characteristics;

[0012] Inputting the second temperature data into the third branch structure to extract the second target temperature time series characteristics;

[0013] Inputting the target action time series feature, the first target temperature time series feature and the second target temperature target time series feature into the feature fusion module to fuse them into original multimodal time series features;

[0014] Inputting the original multimodal time series features into the classifier to detect the heat emission risk value of the user;

[0015] Perform early warning operations on the user's live work based on the heat emission risk value.

[0016] A second aspect of an embodiment of the present application provides a heat radiation risk warning device for shielding clothing based on a posture image, comprising:

[0017] A data acquisition module, used for synchronously acquiring image data of the user, first temperature data inside the shielding suit, and second temperature data of the power environment at each time when the user is wearing the shielding suit and performing live work in the power environment;

[0018] A posture data recognition module, used to recognize the posture data of the user in each frame of the image data;

[0019] A risk detection network determination module, used to determine a risk detection network; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module and a classifier;

[0020] An action timing feature extraction module, used for inputting the posture data into the first branch structure to extract the target action timing feature;

[0021] A first temperature time series feature extraction module, used for inputting the first temperature data into the second branch structure to extract a first target temperature time series feature;

[0022] A second temperature time series feature extraction module, used for inputting the second temperature data into the third branch structure to extract a second target temperature time series feature;

[0023] A multimodal time series feature fusion module, used for inputting the target action time series feature, the first target temperature time series feature and the second target temperature target time series feature into the feature fusion module to fuse them into an original multimodal time series feature;

[0024] A thermal injection risk value detection module, configured to input the original multimodal time series features into the classifier to detect a thermal injection risk value for the user;

[0025] The early warning operation execution module is used to execute early warning operations on the live working of the user according to the heat emission risk value.

[0026] A third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for warning the risk of heat radiation of shielding clothing based on posture images as described in the first aspect above is implemented.

[0027] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the heat radiation risk warning method for shielding clothing based on posture images as described in the first aspect above.

[0028] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method for warning the risk of heat radiation of shielding clothing based on posture images as described in the first aspect.

[0029] In this embodiment, when a user wears a shielding suit and performs live work in an electric power environment, image data of the user, first temperature data inside the shielding suit, and second temperature data of the electric power environment are synchronously collected at each moment; the user's posture data is identified in each frame of image data; a risk detection network is determined; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module and a classifier; the posture data is input into the first branch structure to extract the target action timing features; the first temperature data is input into the second branch structure to extract the first target temperature timing features; the second temperature data is input into the third branch structure to extract the second target temperature timing features; the target action timing features, the first target temperature timing features and the second target temperature timing features are input into the feature fusion module to be fused into the original multimodal timing features; the original multimodal timing features are input into the classifier to detect the thermal radiation risk value of the user; and an early warning operation is performed on the user's live work based on the thermal radiation risk value. This embodiment selects multimodal features related to the user's body temperature, such as the internal temperature of the shielding suit, the ambient temperature, and the user's movements, to detect the user's heat radiation risk, and promptly warns the user of live-line work while wearing the shielding suit, thereby effectively reducing the probability of the user's heat radiation problem and improving the safety of the user's live-line work while wearing the shielding suit. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 It is a schematic diagram of a method for early warning of heat radiation risk of shielding clothing based on posture images provided in an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of a risk detection network provided in an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of a heat radiation risk warning device for shielding clothing based on posture images provided in an embodiment of the present application;

[0034] Figure 4 It is a schematic diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0036] The technical solution of the present application is described below through specific embodiments.

[0037] Reference Figure 1 , shows a schematic diagram of a method for early warning of heat radiation risk of shielding clothing based on posture images provided in an embodiment of the present application, which may specifically include the following steps:

[0038] Step 101 : When a user wears a shielding suit and performs live work in an electric power environment, image data of the user, first temperature data inside the shielding suit, and second temperature data of the electric power environment are synchronously collected at each time.

[0039] A first temperature sensor is set inside the shielding suit, and a camera and a second temperature sensor are deployed on-site in an electric power environment (especially an ultra-high voltage environment). The first temperature sensor and the second temperature sensor are connected to the edge computing node wirelessly (such as Bluetooth, WiFi (Wireless Fidelity), etc.), and the camera is connected to the edge computing node by wire.

[0040] Generally speaking, edge computing nodes are terminal devices with strong computing capabilities, such as computers, servers or embedded devices, which are equipped with graphics processing units (GPUs) or embedded neural network processors (NPUs).

[0041] In the scenario of detecting the risk of thermal radiation for users, edge computing nodes refer to a new business platform built near the site of live operations (i.e., the edge of the network), providing storage, computing, network and other resources, and sinking some key business applications (i.e., detecting the risk of thermal radiation for users) to the edge of the access network to reduce the width and delay losses caused by network transmission and multi-level forwarding. The edge computing node is located between the user and the cloud (server), and is closer to the user (data source) than the traditional cloud. It is small, distributed and closer to the user. Massive data (such as audio data) no longer needs to be uploaded to the cloud for processing, and data processing is realized at the edge of the network, reducing request response time, reducing network bandwidth, and ensuring data security and privacy.

[0042] When a user is wearing a shielding suit and performing live work in an electrical environment, the camera is synchronously called at each moment to track the user and collect multiple frames of image data, the first temperature sensor is called to collect multiple first temperature data inside the shielding suit, and the second temperature sensor is called to collect multiple second temperature data of the electrical environment.

[0043] The multiple frames of image data are sorted according to the time of acquisition and stored in the first cache queue.

[0044] The plurality of first temperature data are sorted according to the time of collection and stored in the second cache queue.

[0045] The plurality of second temperature data are sorted according to the time of collection and stored in the third cache queue.

[0046] In actual applications, the first temperature data inside the shielding suit directly affects the user's body temperature, the second temperature data of the environment indirectly affects the user's body temperature, and the user's posture data reflects the user's body temperature to a certain extent, that is, the actions taken by the user on the environment inside the shielding suit and the environment outside the shielding suit under the user's given physical state (including body temperature). These actions also affect the user's body temperature in reverse and may cause the user's body temperature to rise. The data of these three modes are highly correlated with the user's body temperature.

[0047] Step 102: Identify the user's posture data in each frame of image data.

[0048] A general posture detection network is built into the edge computing node, such as CPM (Convolutional Pose Machine) and HRNet (High-Resolution Network). The image data of each frame in the image sequence is input into the posture detection network for human posture estimation (Human Pose Estimation) to obtain the posture data of the user at each moment. At this time, the posture data at each moment are all multi-dimensional vectors.

[0049] Taking into account the high time requirement for user detection of heat stroke risk and the low accuracy requirement for posture data in subsequent calculations, 15 joint points can be selected to represent the posture data of the human body. While meeting the accuracy requirement, the amount of calculation can be reduced, thereby improving the real-time performance of user detection of heat stroke risk. Then, the posture data at each moment are all 15-dimensional vectors.

[0050] The multi-frame gesture data are sorted according to the time of collection and stored in the fourth cache queue.

[0051] Step 103: Determine the risk detection network.

[0052] In this embodiment, the risk detection network may be constructed and trained in advance in an offline environment.

[0053] Among them, Figure 2 As shown, the risk detection network has a first branch structure Backbone_1, a second branch structure Backbone_2, a third branch structure Backbone_3, a feature fusion module (Feature Fusion Module, FFM) and a classifier Classification.

[0054] When training the risk detection network, the first temperature data, second temperature data and posture data of the user's history of wearing shielding clothing and performing live work in an electrical environment can be collected as samples, the user's confirmed own state (such as comfortable, uncomfortable) can be used as the label Label, and the cross entropy can be used as the loss function to perform supervised training and verification on the risk detection network. When the risk detection network completes training and verifies that the performance meets the requirements, the risk detection network can be deployed to the edge computing node.

[0055] Among them, the user's physiological information (such as body temperature, consciousness level, etc.) can be further detected, and the user's confirmed status can be corrected to improve the accuracy of the label.

[0056] When a user is wearing shielding clothing and performing live work in an electrical environment, the risk detection network can be activated and operated.

[0057] Step 104: Input the posture data into the first branch structure to extract the target action timing characteristics.

[0058] In this embodiment, if Figure 2 As shown, partial posture data Pose can be sequentially input into the first branch structure Backbone_1 to extract features, which are recorded as target action timing features.

[0059] In one design, Figure 2 As shown, the first branch structure Backbone_1 has the first long short-term memory network LSTM_1, the first self-attention layer Self-Attention and the first fully connected layer FC_1.

[0060] In this design, a sliding first window is added to the posture data Pose to obtain a posture sequence.

[0061] The posture sequence is input into the first long short-term memory network LSTM_1 to extract the first candidate action timing features.

[0062] The first candidate action timing feature is input into the first self-attention layer Self-Attention_1 to be converted into the second candidate action timing feature.

[0063] Among them, the first self-attention layer Self-Attention_1 provides a self-attention mechanism to capture important features in the posture sequence, and at the same time helps to alleviate the forgetting problem of the first long short-term memory network LSTM_1 in long-term prediction. Since the first long short-term memory network LSTM_1 processes the posture sequence according to time steps, its position information is retained. Therefore, after being processed by the first long short-term memory network LSTM_1, no additional position information is added, and it can be directly input into the first self-attention layer Self-Attention_1. This combined structure integrates the complementarity of the long short-term memory network and the self-attention mechanism, and effectively processes long-period time posture sequences.

[0064] Moreover, adding the self-attention mechanism facilitates the subsequent use of the attention mechanism for feature fusion.

[0065] The second candidate action timing features are input into the first fully connected layer FC_1 and mapped into the target action timing features to achieve dimensional alignment.

[0066] Step 105: Input the first temperature data into the second branch structure to extract the first target temperature time series characteristics.

[0067] In this embodiment, if Figure 2 As shown, part of the first temperature data Temperature_1 can be sequentially input into the second branch structure Backbone_2 to extract features, which are recorded as first target temperature time series features.

[0068] In one design, Figure 2 As shown, the second branch structure includes a first bidirectional long short-term memory network Bi-LSTM_1, a second long short-term memory network LSTM_2 and a second fully connected layer FC_2.

[0069] In this design, the first temperature data Temperature_1 at each moment is traversed, and the first temperature data Temperature_1 at the previous moment is subtracted from the first temperature data Temperature_1 at the current moment to obtain the first differential data Diff_T1. This process can be expressed as ΔT1=ΔT 1,t -ΔT 1,t-1 , where ΔT1 is the first differential data, ΔT 1,t is the first temperature data Temperature_1 at time t, ΔT 1,t-1 It is the first temperature data Temperature_1 at time t-1, where t is a positive integer, and t≥2.

[0070] The plurality of first differential data Diff_T1 are sorted according to the time of collection and stored in the fifth cache queue.

[0071] A second slidable window is added to the first temperature data Temperature_1 to obtain a first internal temperature sequence.

[0072] A second sliding window is added to the first differential data Diff_T1 to obtain a second internal temperature sequence.

[0073] The first internal temperature sequence is input into the first bidirectional long short-term memory network Bi-LSTM_1 to extract the first internal candidate temperature feature.

[0074] The second internal temperature sequence is input into the second long short-term memory network LSTM_2 to extract the second internal candidate temperature feature.

[0075] Among them, the first bidirectional long short-term memory network Bi-LSTM_1 can consider past and future contextual information, and mine important trend features in heat radiation from the original first temperature data Temperature_1, while the first differential data Diff_T1 itself is one of the trends of the first temperature data Temperature_1, and the single-layer second long short-term memory network LSTM_2 can be used to mine features.

[0076] The first internal candidate temperature feature and the second internal candidate temperature feature are concatenated to form a third internal candidate temperature feature.

[0077] The third internal candidate temperature feature is input into the second fully connected layer FC_2 and mapped to the first target temperature timing feature to achieve dimensional alignment.

[0078] Step 106: Input the second temperature data into the third branch structure to extract the second target temperature time series characteristics.

[0079] In this embodiment, if Figure 2 As shown, part of the second temperature data Temperature_2 can be sequentially input into the third branch structure Backbone_3 to extract features, which are recorded as the second target temperature time series features.

[0080] In one design, Figure 2 As shown, the third branch structure Backbone_3 includes a second bidirectional long short-term memory network Bi-LSTM_2, a third long short-term memory network LSTM_3 and a third fully connected layer FC_3.

[0081] In this design, the second temperature data Temperature_2 at each moment is traversed, and the second temperature data Temperature_2 at the previous moment is subtracted from the second temperature data Temperature_2 at the current moment to obtain the second differential data Diff_T2. This process can be expressed as ΔT2=ΔT2,t -ΔΔT 2,t-1 , where ΔT2 is the second differential data, Δ Δ T 2,t is the second temperature data Temperature_2 at time t, Δ Δ T 2,t-1 It is the second temperature data Temperature_2 at time t-1, where t is a positive integer, and t≥2.

[0082] The plurality of second differential data Diff_T2 are sorted according to the time of collection and stored in the sixth cache queue.

[0083] A third window is added to the second temperature data Temperature_2 to obtain a first ambient temperature sequence.

[0084] A third window is added to the second differential data Diff_T2 to obtain a second ambient temperature sequence.

[0085] The first ambient temperature sequence is input into the second bidirectional long short-term memory network Bi-LSTM_2 to extract the first ambient candidate temperature features.

[0086] The second environment temperature sequence is input into the third long short-term memory network LSTM_3 to extract the second environment candidate temperature features.

[0087] Among them, the second bidirectional long short-term memory network Bi-LSTM_2 can consider past and future contextual information, and mine important trend features in thermal radiation from the original second temperature data Temperature_2, while the second differential data Diff_T2 itself is one of the trends of the second temperature data Temperature_2, and a single-layer third long short-term memory network LSTM_3 can be used to mine features.

[0088] The first environment candidate temperature feature and the second environment candidate temperature feature are concatenated to form a third environment candidate temperature feature.

[0089] The third environment candidate temperature feature is input into the third fully connected layer FC_3 and mapped into the second target temperature time series feature.

[0090] Furthermore, considering that the temperature change rate inside the shielding suit is large, the user's movement change rate is medium, and the ambient temperature change rate is small, the width of the third window is greater than that of the first window, and the width of the first window is greater than that of the second window.

[0091] Step 107: Input the target action time series feature, the first target temperature time series feature and the second target temperature time series feature into a feature fusion module to be fused into an original multimodal time series feature.

[0092] In this embodiment, if Figure 2 As shown, the target action timing features, the first target temperature timing features and the second target temperature target timing features can be input into the feature fusion module FFM, and the target action timing features, the first target temperature timing features and the second target temperature target timing features can be interacted and fused into the original multimodal timing features.

[0093] In one design, the feature fusion module FFM has a second self-attention layer Self-Attention_2, a third self-attention layer Self-Attention_3, a first attention layer Attention_1, a second attention layer Attention_2, a third attention layer Attention_3 and a fourth attention layer Attention_4.

[0094] Generally speaking, the user's actions are obviously sudden, making the posture data sudden data, while the temperature inside the shielding suit and the temperature of the environment both have obvious trends, making the first temperature data and the second temperature data both trend data, which can be learned and predicted using deep learning.

[0095] In addition, the user's action is a reaction made by the user to the temperature inside the comprehensive shielding suit and the temperature of the environment, so that the posture data has the effect of enhancing the first temperature data and the second temperature data.

[0096] In this design, the first bimodal timing feature F1 and the second bimodal timing feature F2 may be initialized by random method or the like.

[0097] The first bimodal time series feature is input into the second self-attention layer Self-Attention_2 and converted into the first candidate modal time series feature under the self-attention mechanism.

[0098] The first target temperature timing feature and the first candidate modal timing feature F1 are input into the first attention layer Attention_1, and according to the attention weight of the first target temperature timing feature for the first candidate modal timing feature, the first target temperature timing feature and the first candidate modal timing feature are fused into the second candidate modal timing feature.

[0099] The target action timing features and the second candidate modality timing features are input into the second attention layer Attention_2, and according to the attention weight of the target action timing features for the second candidate modality timing features, the target action timing features and the second candidate modality timing features are fused into the third candidate modality timing features.

[0100] The second bimodal temporal feature F2 is input into the third self-attention layer Self-Attention_3 and converted into the fourth candidate modal temporal feature under the self-attention mechanism.

[0101] The second target temperature time series feature and the fourth candidate modal time series feature are input into the third attention layer Attention_3, and according to the attention weight of the second target temperature time series feature relative to the fourth candidate modal time series feature, the second target temperature time series feature and the fourth candidate modal time series feature are fused into the fifth candidate modal time series feature;

[0102] The target action timing feature and the fifth candidate modal timing feature are input into the fourth attention layer Attention_4, and according to the attention weight of the target action timing feature relative to the fifth candidate modal timing feature, the target action timing feature and the fifth candidate modal timing feature are fused into the sixth candidate modal timing feature.

[0103] The third candidate modal time series feature and the sixth candidate modal time series feature are concatenated into the original multimodal time series feature.

[0104] Step 108: Input the original multimodal time series features into the classifier to detect the heat emission risk value of the user.

[0105] In this embodiment, if Figure 2 As shown, the original multimodal time series features can be input into the classifier Classification to perform the classification task, and the heat stroke risk value of the user is detected. The heat stroke risk value is the probability that the user has heat stroke.

[0106] In one design, Figure 2 As shown, the classifier Classification is a Seq2Seq (sequence to sequence) architecture, which includes an encoder Encoder, a decoder Decoder and a head structure Head.

[0107] Among them, both the encoder Encoder and the decoder Decoder can use structures such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Units), and the head structure Head can use multi-layer FC (Fully Connected Layer), Sigmoid (S-type function) and other structures.

[0108] In this design, the original multimodal time series features are input into the encoder and encoded into candidate multimodal time series features.

[0109] The candidate multimodal time series features are input into the decoder and decoded into target multimodal time series features.

[0110] The target multimodal time series features are input into the head structure Head to generate a heat emission risk value for the user.

[0111] Step 109: Perform a warning operation on the user's live work according to the thermal radiation risk value.

[0112] In practical applications, the threshold method, trend method and other methods can be used to analyze the thermal radiation risk value, so as to perform early warning operations on the user's live work. When the user is at potential risk of thermal radiation, the user's live work can be stopped in advance and the shielding suit can be taken off for a rest.

[0113] In one embodiment of the present application, step 109 may include the following steps:

[0114] Step 1091: Calculate an average value of multiple thermal emission risk values ​​as a sliding risk value.

[0115] In this embodiment, a sliding fourth window may be added to the thermal emission risk value to obtain a risk sequence, and an average value of multiple thermal emission risk values ​​in the risk sequence is calculated and recorded as a sliding risk value.

[0116] Step 1092: If the sliding risk value is greater than or equal to the preset first risk threshold, the risk monitoring mode is started.

[0117] In this embodiment, the sliding risk value is compared with a preset first risk threshold. If the sliding risk value is greater than or equal to the first risk threshold, it means that the sliding risk value is high and the user is at potential risk of thermal emission. However, considering that there is a certain error in the risk detection network, the risk monitoring mode can be started to further monitor the user and reduce interference with the user's live operations.

[0118] Step 1093: In the risk monitoring mode, a monitoring time range is set, and the first risk threshold is attenuated according to the length of the monitoring time range to obtain a second risk threshold.

[0119] In the risk monitoring mode, a dynamic monitoring time range can be set starting from the moment when the sliding risk value is greater than or equal to the first risk threshold. The length of the monitoring time range is negatively correlated with the sliding risk value, that is, the lower the sliding risk value, the longer the length of the monitoring time range, and conversely, the higher the sliding risk value, the shorter the length of the monitoring time range.

[0120] In addition, the first risk threshold is attenuated according to the length of the monitoring time range to obtain a second risk threshold, that is, the second risk threshold is lower than the first risk threshold.

[0121] The combination of a dynamic monitoring time range and a dynamic second risk threshold can improve the sensitivity of monitoring and ensure the safety of users.

[0122] Exemplarily, the length of the monitoring time range and the first risk threshold are substituted into the following formula to obtain the second risk threshold:

[0123]

[0124] Among them, H1 is the first risk threshold, H2 is the second risk threshold, t1 is the timestamp for setting the monitoring time range, t2 is the current timestamp, w is the length of the monitoring time range, and α is the adjustment coefficient, which is used to adjust the attenuation degree of the first risk threshold.

[0125] Step 1094: within the monitoring time range, if any sliding risk value is greater than or equal to the second risk threshold, a warning operation is performed on the user's live work.

[0126] Step 1095: within the monitoring time range, if all sliding risk values ​​are less than the second risk threshold, the risk monitoring mode is canceled.

[0127] In this embodiment, the risk detection network is continuously used to output the heat-shoot risk value, and a plurality of heat-shoot risk values ​​are smoothed to obtain a sliding risk value.

[0128] During the monitoring time range, the sliding risk value is not compared with the original first risk threshold, but with the second risk threshold after attenuation.

[0129] If any sliding risk value is greater than or equal to the second risk threshold, a warning operation is performed on the user's live work, prompting the user to stop the live work and take off the shielding suit to rest.

[0130] If all sliding risk values ​​are less than the second risk threshold, indicating that the previous high sliding risk value is a false alarm, the risk monitoring mode is canceled and the normal monitoring mode is restored. At this time, the sliding risk value is restored and compared with the original first risk threshold.

[0131] In this embodiment, when a user wears a shielding suit and performs live work in an electric power environment, image data of the user, first temperature data inside the shielding suit, and second temperature data of the electric power environment are synchronously collected at each moment; the user's posture data is identified in each frame of image data; a risk detection network is determined; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module and a classifier; the posture data is input into the first branch structure to extract the target action timing features; the first temperature data is input into the second branch structure to extract the first target temperature timing features; the second temperature data is input into the third branch structure to extract the second target temperature timing features; the target action timing features, the first target temperature timing features and the second target temperature timing features are input into the feature fusion module to be fused into the original multimodal timing features; the original multimodal timing features are input into the classifier to detect the thermal radiation risk value of the user; and an early warning operation is performed on the user's live work based on the thermal radiation risk value. This embodiment selects multimodal features related to the user's body temperature, such as the internal temperature of the shielding suit, the ambient temperature, and the user's movements, to detect the user's heat radiation risk, and promptly warns the user of live-line work while wearing the shielding suit, thereby effectively reducing the probability of the user's heat radiation problem and improving the safety of the user's live-line work while wearing the shielding suit.

[0132] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0133] Reference Figure 3 , shows a schematic diagram of a shielding clothing heat radiation risk warning device based on posture images provided in an embodiment of the present application, which may specifically include the following modules:

[0134] The data acquisition module 301 is used to synchronously acquire image data of the user, first temperature data inside the shielding suit, and second temperature data of the power environment at each time when the user wears the shielding suit to perform live work in the power environment;

[0135] A gesture data recognition module 302, used to recognize the gesture data of the user in each frame of the image data;

[0136] The risk detection network determination module 303 is used to determine the risk detection network; the risk detection network has a first branch structure, a second branch structure, a third branch structure, a feature fusion module and a classifier;

[0137] An action timing feature extraction module 304 is used to input the posture data into the first branch structure to extract the target action timing feature;

[0138] A first temperature time series feature extraction module 305, configured to input the first temperature data into the second branch structure to extract a first target temperature time series feature;

[0139] A second temperature time series feature extraction module 306, configured to input the second temperature data into the third branch structure to extract a second target temperature time series feature;

[0140] A multimodal time series feature fusion module 307, configured to input the target action time series feature, the first target temperature time series feature and the second target temperature target time series feature into the feature fusion module to fuse them into an original multimodal time series feature;

[0141] A thermal injection risk value detection module 308 is used to input the original multimodal time series features into the classifier to detect a thermal injection risk value for the user;

[0142] The warning operation execution module 309 is used to execute a warning operation on the live working of the user according to the heat emission risk value.

[0143] In one embodiment of the present application, the first branch structure comprises a first long short-term memory network, a first self-attention layer and a first fully connected layer;

[0144] The action timing feature extraction module 304 is also used for:

[0145] Adding a first window to the posture data to obtain a posture sequence;

[0146] Inputting the posture sequence into the first long short-term memory network to extract the first candidate action time sequence feature;

[0147] Inputting the first candidate action time series feature into the first self-attention layer to convert it into a second candidate action time series feature;

[0148] The second candidate action timing features are input into the first fully connected layer and mapped into target action timing features.

[0149] In one embodiment of the present application, the second branch structure comprises a first bidirectional long short-term memory network, a second long short-term memory network and a second fully connected layer;

[0150] The first temperature time series feature extraction module 305 is also used for:

[0151] Subtracting the first temperature data at a previous moment from the first temperature data at a current moment to obtain first differential data;

[0152] adding a second window to the first temperature data to obtain a first internal temperature sequence;

[0153] adding a second window to the first differential data to obtain a second internal temperature sequence;

[0154] Inputting the first internal temperature sequence into the first bidirectional long short-term memory network to extract a first internal candidate temperature feature;

[0155] Inputting the second internal temperature sequence into the second long short-term memory network to extract a second internal candidate temperature feature;

[0156] concatenating the first internal candidate temperature feature and the second internal candidate temperature feature into a third internal candidate temperature feature;

[0157] The third internal candidate temperature feature is input into the second fully connected layer and mapped into the first target temperature time series feature.

[0158] In one embodiment of the present application, the third branch structure comprises a second bidirectional long short-term memory network, a third long short-term memory network and a third fully connected layer;

[0159] The second temperature time series feature extraction module 306 is further used for:

[0160] subtracting the second temperature data at the previous moment from the second temperature data at the current moment to obtain second differential data;

[0161] Adding a third window to the second temperature data to obtain a first ambient temperature sequence;

[0162] adding a third window to the second differential data to obtain a second ambient temperature sequence;

[0163] Inputting the first environment temperature sequence into the second bidirectional long short-term memory network to extract the first environment candidate temperature feature;

[0164] Inputting the second environment temperature sequence into the third long short-term memory network to extract the second environment candidate temperature feature;

[0165] splicing the first environment candidate temperature feature and the second environment candidate temperature feature into a third environment candidate temperature feature;

[0166] Inputting the third environment candidate temperature feature into the third fully connected layer and mapping it into the second target temperature time series feature;

[0167] The width of the third window is greater than the width of the first window, and the width of the first window is greater than the width of the second window.

[0168] In one embodiment of the present application, the feature fusion module comprises a second self-attention layer, a third self-attention layer, a first attention layer, a second attention layer, a third attention layer and a fourth attention layer;

[0169] The multimodal temporal feature fusion module 307 is also used for:

[0170] Initializing the first bimodal timing feature and the second bimodal timing feature;

[0171] Inputting the first bimodal time series feature into the second self-attention layer to convert it into a first candidate modality time series feature;

[0172] Inputting the first target temperature time series feature and the first candidate modality time series feature into the first attention layer to fuse them into a second candidate modality time series feature;

[0173] Inputting the target action temporal feature and the second candidate modality temporal feature into the second attention layer to fuse them into a third candidate modality temporal feature;

[0174] Inputting the second bimodal time series feature into the third self-attention layer to convert it into a fourth candidate modality time series feature;

[0175] Inputting the second target temperature time series feature and the fourth candidate modal time series feature into the third attention layer to fuse them into a fifth candidate modal time series feature;

[0176] Inputting the target action temporal feature and the fifth candidate modality temporal feature into the fourth attention layer to fuse them into a sixth candidate modality temporal feature;

[0177] The third candidate modal time series feature and the sixth candidate modal time series feature are concatenated into an original multi-modal time series feature.

[0178] In one embodiment of the present application, the classifier includes an encoder, a decoder and a header structure;

[0179] The thermal emission risk value detection module 308 includes:

[0180] An encoding module, used for inputting the original multimodal time series features into the encoder to encode them into candidate multimodal time series features;

[0181] A decoding module, used for inputting the candidate multimodal time series features into the decoder to decode them into target multimodal time series features;

[0182] A classification module is used to input the target multimodal time series features into the head structure to generate a thermal emission risk value for the user.

[0183] In one embodiment of the present application, the early warning operation execution module 309 includes:

[0184] A sliding risk value calculation module, used for calculating an average value of the plurality of thermal emission risk values ​​as a sliding risk value;

[0185] A risk monitoring mode activation module, configured to activate the risk monitoring mode if the sliding risk value is greater than or equal to a preset first risk threshold;

[0186] a threshold attenuation module, configured to set a monitoring time range in the risk monitoring mode, and attenuate the first risk threshold according to the length of the monitoring time range to obtain a second risk threshold; the length of the monitoring time range is negatively correlated with the sliding risk value;

[0187] A sliding risk warning module, configured to perform a warning operation on the live working of the user if any of the sliding risk values ​​is greater than or equal to the second risk threshold within the monitoring time range;

[0188] The risk monitoring mode canceling module is used to cancel the risk monitoring mode if all the sliding risk values ​​are less than the second risk threshold within the monitoring time range.

[0189] In one embodiment of the present application, the threshold attenuation module is further used for:

[0190] Substitute the length of the monitoring time range and the first risk threshold into the following formula to obtain the second risk threshold:

[0191]

[0192] Wherein, H1 is the first risk threshold, H2 is the second risk threshold, t 11 is the timestamp for setting the monitoring time range, t2 is the current timestamp, w is the length of the monitoring time range, and α is the adjustment coefficient.

[0193] An embodiment of the present application provides a posture image-based shielding clothing heat radiation risk warning device, and the device can be used to implement each step in the aforementioned method embodiments.

[0194] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0195] Reference Figure 4 , shows a schematic diagram of a terminal device provided in an embodiment of the present application. Figure 4As shown, the terminal device 400 in the embodiment of the present application includes: a processor 410, a memory 420, and a computer program 421 stored in the memory 420 and executable on the processor 410. When the processor 410 executes the computer program 421, the steps in each embodiment of the above-mentioned shielding clothing heat radiation risk warning method based on posture image are implemented. Alternatively, when the processor 410 executes the computer program 421, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0196] Exemplarily, the computer program 421 may be divided into one or more modules / units, which are stored in the memory 420 and executed by the processor 410 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which may be used to describe the execution process of the computer program 421 in the terminal device 400.

[0197] The terminal device 400 may include, but is not limited to, a processor 410 and a memory 420. Those skilled in the art will appreciate that Figure 4 It is only an example of the terminal device 400 and does not constitute a limitation on the terminal device 400. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 400 may also include input and output devices, network access devices, buses, etc.

[0198] The processor 410 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0199] The memory 420 may be an internal storage unit of the terminal device 400, such as a hard disk or memory of the terminal device 400. The memory 420 may also be an external storage device of the terminal device 400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 400. Further, the memory 420 may also include both an internal storage unit of the terminal device 400 and an external storage device. The memory 420 is used to store the computer program 421 and other programs and data required by the terminal device 400. The memory 420 may also be used to temporarily store data that has been output or is to be output.

[0200] An embodiment of the present application also discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for warning the heat radiation risk of shielding clothing based on posture images as described in the aforementioned embodiments is implemented.

[0201] An embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the posture image-based shielding clothing heat radiation risk warning method is implemented as described in the above-mentioned embodiments.

[0202] An embodiment of the present application further discloses a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the posture image-based heat radiation risk warning method for shielding clothing as described in the aforementioned embodiments.

[0203] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for early warning of heat radiation risk of shielding clothing based on posture images, characterized in that: include: When a user wears a shielding suit and performs live work in an electric power environment, image data of the user, first temperature data inside the shielding suit, and second temperature data of the electric power environment are synchronously collected at each time; Identifying the user's posture data in each frame of the image data; Identify risk detection networks; The risk detection network comprises a first branch structure, a second branch structure, a third branch structure, a feature fusion module and a classifier; Inputting the posture data into the first branch structure to extract the target action timing features; Inputting the first temperature data into the second branch structure to extract the first target temperature time series characteristics; Inputting the second temperature data into the third branch structure to extract the second target temperature time series characteristics; Inputting the target action time series feature, the first target temperature time series feature and the second target temperature time series feature into the feature fusion module to fuse them into an original multimodal time series feature; Inputting the original multimodal time series features into the classifier to detect the heat emission risk value of the user; Perform an early warning operation on the user's live work according to the heat emission risk value.

2. The method according to claim 1, characterized in that The first branch structure comprises a first long short-term memory network, a first self-attention layer and a first fully connected layer; The step of inputting the posture data into the first branch structure to extract the target action timing features comprises: Adding a first window to the posture data to obtain a posture sequence; Inputting the posture sequence into the first long short-term memory network to extract the first candidate action time sequence feature; Inputting the first candidate action time series feature into the first self-attention layer to convert it into a second candidate action time series feature; The second candidate action timing features are input into the first fully connected layer and mapped into target action timing features.

3. The method according to claim 2, characterized in that The second branch structure comprises a first bidirectional long short-term memory network, a second long short-term memory network and a second fully connected layer; The step of inputting the first temperature data into the second branch structure to extract the first target temperature time series feature comprises: Subtracting the first temperature data at a previous moment from the first temperature data at a current moment to obtain first differential data; adding a second window to the first temperature data to obtain a first internal temperature sequence; adding a second window to the first differential data to obtain a second internal temperature sequence; Inputting the first internal temperature sequence into the first bidirectional long short-term memory network to extract a first internal candidate temperature feature; Inputting the second internal temperature sequence into the second long short-term memory network to extract second internal candidate temperature features; splicing the first internal candidate temperature feature and the second internal candidate temperature feature into a third internal candidate temperature feature; The third internal candidate temperature feature is input into the second fully connected layer and mapped into the first target temperature time series feature.

4. The method according to claim 3, characterized in that The third branch structure comprises a second bidirectional long short-term memory network, a third long short-term memory network and a third fully connected layer; The step of inputting the second temperature data into the third branch structure to extract the second target temperature timing characteristics comprises: subtracting the second temperature data at the previous moment from the second temperature data at the current moment to obtain second differential data; Adding a third window to the second temperature data to obtain a first ambient temperature sequence; adding a third window to the second differential data to obtain a second ambient temperature sequence; Inputting the first environment temperature sequence into the second bidirectional long short-term memory network to extract the first environment candidate temperature feature; Inputting the second environment temperature sequence into the third long short-term memory network to extract the second environment candidate temperature feature; splicing the first environment candidate temperature feature and the second environment candidate temperature feature into a third environment candidate temperature feature; Inputting the third environment candidate temperature feature into the third fully connected layer and mapping it into the second target temperature time series feature; The width of the third window is greater than the width of the first window, and the width of the first window is greater than the width of the second window.

5. The method according to claim 1, characterized in that The feature fusion module comprises a second self-attention layer, a third self-attention layer, a first attention layer, a second attention layer, a third attention layer and a fourth attention layer; The step of inputting the target action time series feature, the first target temperature time series feature, and the second target temperature time series feature into the feature fusion module to fuse them into a multi-modal time series feature includes: Initializing the first bimodal timing feature and the second bimodal timing feature; Inputting the first bimodal time series feature into the second self-attention layer to convert it into a first candidate modality time series feature; Inputting the first target temperature time series feature and the first candidate modality time series feature into the first attention layer to fuse them into a second candidate modality time series feature; Inputting the target action temporal feature and the second candidate modality temporal feature into the second attention layer to fuse them into a third candidate modality temporal feature; Inputting the second bimodal time series feature into the third self-attention layer to convert it into a fourth candidate modality time series feature; Inputting the second target temperature time series feature and the fourth candidate modal time series feature into the third attention layer to fuse them into a fifth candidate modal time series feature; Inputting the target action temporal feature and the fifth candidate modality temporal feature into the fourth attention layer to fuse them into a sixth candidate modality temporal feature; The third candidate modal time series feature and the sixth candidate modal time series feature are concatenated into an original multi-modal time series feature.

6. The method according to any one of claims 1 to 5, characterized in that The classifier includes an encoder, a decoder and a head structure; The step of inputting the original multimodal time series features into the classifier to detect the heat emission risk value of the user includes: Inputting the original multimodal time series features into the encoder to be encoded into candidate multimodal time series features; Inputting the candidate multimodal time series features into the decoder to decode them into target multimodal time series features; The target multimodal time series features are input into the head structure to generate a heat emission risk value for the user.

7. The method according to any one of claims 1 to 5, characterized in that The performing of an early warning operation on the user's live work according to the thermal radiation risk value includes: Calculate an average value of the plurality of thermal emission risk values ​​as a sliding risk value; If the sliding risk value is greater than or equal to a preset first risk threshold, the risk monitoring mode is activated; In the risk monitoring mode, a monitoring time range is set, and the first risk threshold is attenuated according to the length of the monitoring time range to obtain a second risk threshold; the length of the monitoring time range is negatively correlated with the sliding risk value; Within the monitoring time range, if any of the sliding risk values ​​is greater than or equal to the second risk threshold, performing a warning operation on the live working of the user; Within the monitoring time range, if all the sliding risk values ​​are less than the second risk threshold, the risk monitoring mode is canceled.

8. The method according to claim 7, characterized in that The attenuating the first risk threshold according to the length of the monitoring time range includes: Substitute the length of the monitoring time range and the first risk threshold into the following formula to obtain the second risk threshold: Among them, H1 is the first risk threshold, H2 is the second risk threshold, t1 is the timestamp for setting the monitoring time range, t2 is the current timestamp, w is the length of the monitoring time range, and α is the adjustment coefficient.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for warning the risk of heat radiation of shielding clothing based on posture images as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for warning the risk of heat radiation of shielding clothing based on posture images as described in any one of claims 1 to 8 is implemented.

Images