Methods, equipment and media for monitoring personnel in hazardous areas

Abstract

The embodiments of the present application provide a method, device, and medium for monitoring personnel in hazardous areas. Low-level features and high-level features are obtained based on training sets of RGB images and thermal imaging images. An operation of decoupling multi-scale feature interactions into intra-scale feature interactions is performed on the high-level features to obtain decoupled features. A cross-scale fusion operation is performed on the low-level features and the decoupled features to obtain fused features. Fusion features whose classification scores and intersection-over-union scores exceed a preset threshold are selected as target features. The target features are weighted and summed according to attention weights, and the weighted sum values are nonlinearly transformed to obtain prediction results. The parameters of the detection model are adjusted according to the prediction results to obtain a detection model, and personnel are monitored on the saliency maps of the RGB images and thermal imaging images through the detection model. The amount of computation can be reduced and detection efficiency can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to methods, devices, and media for monitoring personnel in hazardous areas. Background Art

[0002] Factories or construction sites contain hazardous areas, such as the polishing workshop where containers are polished. Relying solely on human resources to monitor production workers is inefficient and wasteful. Currently, monitoring methods combine real-time camera video with deep learning models. Object detection methods, which detect intrusion zones using the intersection-of-union ratio (IoU) between the detection frame and the calibration area, are suitable for camera-based overhead viewing. However, if the camera is mounted horizontally, conventional object detection methods struggle to determine whether a worker is within the detection zone. This can lead to misidentification, high computational complexity, and low detection efficiency. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent. The embodiments of this application provide a method, equipment and medium for monitoring personnel in dangerous areas, which can reduce the amount of calculation and improve detection efficiency.

[0005] An embodiment of the first aspect of the present application is a method for monitoring personnel in a hazardous area, comprising:

[0006] Acquire RGB and thermal images of hazardous areas;

[0007] generating a saliency map of the RGB image and the thermal imaging image;

[0008] Inputting the saliency maps of the RGB image and the thermal image into a detection model to perform personnel monitoring and obtain a target detection result;

[0009] The detection model is trained according to the following method:

[0010] Performing feature extraction on a training set formed by RGB images and thermal imaging images through a convolutional layer to obtain low-level features and high-level features, wherein the high-level features are extracted from the low-level features;

[0011] Performing an operation of decoupling multi-scale feature interactions into intra-scale feature interactions on the high-level features through an encoder to obtain decoupled features, and performing a cross-scale fusion operation on the low-level features and the decoupled features to obtain fused features;

[0012] Select the fusion feature whose classification score and intersection-over-union score exceed the preset threshold as the target feature;

[0013] Calculating the attention weight of the target feature through a decoder, performing weighted summation on the target feature according to the attention weight to obtain a weighted sum value, and performing a nonlinear transformation on the weighted sum value to obtain a prediction result;

[0014] The parameters of the detection model are adjusted according to the prediction results.

[0015] According to certain embodiments of the first aspect of the present application, the method for monitoring personnel in a hazardous area includes:

[0016] When the target detection result indicates that there are people in the dangerous area, an alarm operation is performed.

[0017] According to certain embodiments of the first aspect of the present application, the method for monitoring personnel in a hazardous area includes:

[0018] Obtaining frequency information, area information, and time information of the alarm operation;

[0019] determining a first dangerous area and a second dangerous area from the dangerous areas according to the frequency information, the area information, and the time information, wherein the alarm frequency of the first dangerous area is higher than the alarm frequency of the second dangerous area, and the alarm time of the first dangerous area is longer than the alarm time of the second dangerous area;

[0020] The first dangerous area is monitored by a first detection model, and the second dangerous area is monitored by a second detection model. The detection accuracy of the first detection model is higher than the detection accuracy of the second detection model.

[0021] According to certain embodiments of the first aspect of the present application, determining a first danger zone and a second danger zone from the danger zone according to the frequency information, the area information, and the time information includes:

[0022] The frequency information, the area information and the time information are used as features to convert and obtain a feature vector corresponding to the dangerous area;

[0023] Obtaining the distance between the feature of the feature vector corresponding to the dangerous area and the feature of the known feature vector;

[0024] Determine the label of the feature vector corresponding to the dangerous area according to the label with the largest number among the K known feature vectors closest to the distance;

[0025] The feature vectors corresponding to the plurality of dangerous areas are clustered according to the labels of the feature vectors corresponding to the dangerous areas to determine a first dangerous area and a second dangerous area.

[0026] According to certain embodiments of the first aspect of the present application, the method for monitoring personnel in a hazardous area includes:

[0027] Constructing a teacher model having the same network framework as the detection model, wherein the teacher model includes a first teacher model and a second teacher model;

[0028] Training the first teacher model using a first training image and training the second teacher model using a second training image, wherein pixels of the first training image are higher than pixels of the second training image;

[0029] fusing the first teacher model and the second teacher model to obtain a fused teacher model;

[0030] The parameters of the fusion teacher model are distilled to the detection model.

[0031] According to certain embodiments of the first aspect of the present application, the first teacher model and the second teacher model are each provided with multiple feature extraction layers, and the output features of a later feature extraction layer of the first teacher model and the second teacher model are smaller in size than the output features of a previous feature extraction layer; and fusing the first teacher model and the second teacher model to obtain a fused teacher model includes:

[0032] Fusing the output features of the kth feature extraction layer of the first teacher model with the output features of the k+1th feature extraction layer of the second teacher model to obtain fused features, where the output features of the kth feature extraction layer of the first teacher model and the output features of the k+1th feature extraction layer of the second teacher model have the same size;

[0033] A prediction result is obtained by detecting according to the fusion feature, a loss function is obtained according to the prediction result and a preset loss weight, and the parameters of the teacher model are adjusted by the loss function to obtain a fusion teacher model.

[0034] According to certain embodiments of the first aspect of the present application, fusing the output features of the kth feature extraction layer of the first teacher model with the output features of the k+1th feature extraction layer of the second teacher model to obtain the fused features includes:

[0035] Splicing the output features of the kth feature extraction layer of the first teacher model and the output features of the k+1th feature extraction layer of the second teacher model to obtain a splicing result;

[0036] Performing global pooling on the splicing result to obtain a global pooling result;

[0037] The global pooling result is sequentially passed through a fully connected layer for dimension transformation and an activation function layer for nonlinear transformation to obtain an integer result;

[0038] Normalize the integer results by passing them through normalization layers in sequence, and map them to obtain probability results;

[0039] The probability results are weighted and summed to obtain the fusion feature.

[0040] According to certain embodiments of the first aspect of the present application, generating a saliency map of the RGB image and the thermal imaging image includes:

[0041] performing an attention operation on local semantic information of local adjacent regions of the RGB image and the thermal image through a convolutional layer to obtain a preliminary saliency map;

[0042] The residual between the true value and the preliminary saliency map is recursively learned by a residual refinement block, and the preliminary saliency map is refined according to the residual to obtain a saliency map.

[0043] An embodiment of the second aspect of the present application is an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for monitoring personnel in a hazardous area as described above is implemented.

[0044] An embodiment of the third aspect of the present application is a computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method for monitoring personnel in a dangerous area as described above.

[0045] The above scheme has at least the following beneficial effects: low-level features and high-level features are obtained based on the training sets of RGB images and thermal imaging images, the operation of decoupling multi-scale feature interactions into intra-scale feature interactions is performed on the high-level features to obtain decoupled features, and a cross-scale fusion operation is performed on the low-level features and the decoupled features to obtain fused features; the fused features whose classification scores and intersection-over-union scores exceed a preset threshold are selected as target features; the target features are weighted and summed according to the attention weights, and the weighted sum values are nonlinearly transformed to obtain prediction results; the parameters of the detection model are adjusted according to the prediction results to obtain a detection model, and the detection model is used to monitor personnel on the saliency maps of the RGB images and thermal imaging images; it can reduce the amount of calculation and improve detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0047] Figure 1is a step diagram of a method for monitoring personnel in a dangerous area provided by an embodiment of the present application;

[0048] Figure 2 It is a step diagram of the training method of the detection model;

[0049] Figure 3 This is a step diagram of the knowledge distillation method for the detection model;

[0050] Figure 4 It is a sub-step diagram of step S731;

[0051] Figure 5 It is a step diagram of alarm operation;

[0052] Figure 6 is a sub-step diagram of step S820;

[0053] Figure 7 is a schematic diagram of the detection model;

[0054] Figure 8 It is a schematic diagram of the first teacher model and the second teacher model. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0056] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and the like in the specification, claims, or accompanying drawings are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0057] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0058] An embodiment of the present application provides a method for monitoring personnel in a dangerous area.

[0059] Reference Figure 1 , a method for monitoring personnel in hazardous areas, including but not limited to the following steps:

[0060] Step S110, acquiring an RGB image and a thermal imaging image of the dangerous area;

[0061] Step S120, generating a saliency map of the RGB image and the thermal imaging image;

[0062] Step S130: Input the saliency maps of the RGB image and the thermal image into a detection model to perform personnel monitoring and obtain target detection results.

[0063] Reference Figure 2 , where the detection model is trained according to the following method:

[0064] Step S200 , performing feature extraction on a training set formed by the RGB image and the thermal imaging image through a convolutional layer to obtain low-level features and high-level features;

[0065] Step S300: Decoupling multi-scale feature interactions into intra-scale feature interactions on high-level features through an encoder to obtain decoupled features, and performing a cross-scale fusion operation on low-level features and decoupled features to obtain fused features.

[0066] Step S400, selecting a fusion feature whose classification score and intersection-over-union score exceed a preset threshold as a target feature;

[0067] Step S500: Calculate the attention weight of the target feature through the decoder, perform weighted summation on the target feature according to the attention weight to obtain a weighted sum value, and perform a nonlinear transformation on the weighted sum value to obtain a prediction result;

[0068] Step S600: Adjust the parameters of the detection model according to the prediction results.

[0069] The RGB image of the video stream is obtained by an optical camera installed in the hazardous area, and the thermal imaging image of the video stream is obtained by a thermal imaging camera installed in the hazardous area.

[0070] Specifically, the camera image resolution is 3820*2180 and the frame rate is 25FPS.

[0071] The RGB image of the video stream is extracted to obtain a single-frame RGB image, and the thermal imaging image of the video stream is extracted to form a single-frame thermal imaging image.

[0072] The two modal data sets are mixed, and then the training data set formed by RGB images and thermal imaging images is cleaned to construct the Per-det data set. Then, LABELME is used to perform a small amount of manual annotation on the Per-det data set to obtain a Json format annotation file.

[0073] The training set consisting of RGB images and thermal imaging images containing Json format annotation files is used to train the detection model.

[0074] The backbone network of the detection model uses convolutional neural networks to process the images of the training set of B×3×H0×W0 dimensions through CNN, and then converts the images into feature maps f∈R B×C×H×W, where H, W, and C are the height, width, and number of channels of the feature map, respectively. C is set to 2048 or 256, the height is set to 1 / 32, and the width is set to 1 / 32. B is the number of predicted boxes, usually set to 100, and R is the Euclidean space.

[0075] From the feature map f∈R of the backbone network B×C×H×W Images at three scales are extracted, with output strides of 8, 16, and 32. Feature maps with strides of 8, 16, and 32 are input to the encoder.

[0076] Among them, the feature map with a step size of 32 is a high-level feature, and the feature maps with step sizes of 8 and 16 are low-level features. High-level features are extracted from low-level features.

[0077] For step S300, the encoder consists of two modules, including an attention-based intra-scale feature interaction module and a CNN-based cross-scale feature fusion module.

[0078] The attention-based intra-scale feature interaction module performs intra-scale interactions on feature maps with a stride of 32. An operation that decouples multi-scale feature interactions into intra-scale feature interactions is performed on high-level features to obtain decoupled features.

[0079] Specifically, the attention-based intra-scale feature interaction module adopts a Transformer Encoder layer, which includes standard MSAH (or Deformable Attention) and FFN.

[0080] The attention-based intra-scale feature interaction module pulls the two-dimensional high-level features into vectors, passes through the process of multi-head attention and FFN, and then resizes the output back to two dimensions.

[0081] It can be expressed by the formula: Q=K=V=Flatten(S5),F5=Reshape(Attn(Q,K,V)).

[0082] Among them, S5 is the feature map with a step size of 32, Q, K, V are the query, key and value of the attention mechanism, and F5 is the output of the attention-based intra-scale feature interaction module.

[0083] This is different from the previous method of flattening and splicing multi-scale features to form a very long sequence vector. Although this can enable full interaction between multi-scale features, it also causes a huge amount of computation and time consumption, as well as the use of high-level features and low-level features to interact at the same time.

[0084] Because high-level features possess deeper, more advanced, and richer semantic characteristics than low-level features, these semantic features are of greater interest and need to the Transformer, making them more useful for distinguishing features between different objects. Shallow features, however, lack strong semantic features and are less effective. Due to the lack of semantic concepts, there is a risk of duplication and confusion in interactions with high-level features, making intra-scale interactions with low-level features unnecessary. The attention-based intra-scale feature interaction module uses only high-level features for interaction, reducing computational effort, saving time, and improving speed. Applying self-attention to high-level features with richer semantic concepts can capture the connections between conceptual entities in the image, facilitating subsequent modules' detection and recognition of objects in the image.

[0085] The CNN-based cross-scale feature fusion module inserts several fusion blocks consisting of convolutional layers into the fusion path. The fusion block fuses adjacent features into a new feature. The fusion block contains N convolutional blocks. The outputs of the two paths are fused by element-wise addition. A cross-scale fusion operation is performed on low-level features and decoupled features to produce a fused feature.

[0086] For step S400 , a fusion feature whose classification score and intersection-over-union score exceed a preset threshold is selected as a target feature.

[0087] Specifically, the object query of the detection model is a set of learnable embeddings, which are optimized by the decoder and the classification scores and bounding boxes obtained by the prediction head mapping. However, object queries are difficult to interpret and optimize. If the classification score is used to select the top K features from the encoder to initialize the object query, due to the inconsistency of the classification score and the position confidence, some predicted boxes will have high classification scores but are not close to the true box. This results in some predicted boxes with high classification scores and low intersection over union (IoU) scores being selected, while predicted boxes with low classification scores and high intersection over union scores are discarded, which will damage the performance of the detector. To solve this problem, IoU-aware query selection is used. By constraining the model, during training, the model is allowed to produce high classification scores for features with high IoU and low classification scores for features with low IoU during training.

[0088] Therefore, the prediction boxes corresponding to the top K encoder features selected by the model based on the classification score have higher classification scores and higher intersection-over-union scores. The optimization objective of the detector is reformulated as follows:

[0089]

[0090] For step S500, the attention weight of the target feature is calculated by the decoder, the target feature is weighted and summed according to the attention weight to obtain a weighted sum value, and the weighted sum value is nonlinearly transformed to obtain a prediction result.

[0091] Specifically, the high classification scores and high-confidence positions obtained by the Iou perception query selection mechanism are fed into the decoder for decoding. The encoder enables the feature map to capture the global information of the input image and, after position encoding, is input into the decoder. The decoder processes the input feature map through a self-attention mechanism. The self-attention layer calculates the attention weights for each position in the feature map and then performs a weighted summation of the feature map based on these weights to capture the correlation between features. The output of each self-attention layer passes through a feedforward network, which uses multi-layer perceptrons to perform nonlinear transformations on the features, further enhancing the representation capabilities of the features and finally outputting the prediction results of the detection model. The prediction results include the category probability, bounding box position, and background / no-target probability of each detected target.

[0092] In step S600, the parameters of the detection model are adjusted according to the prediction results until the number of training times reaches a preset training time threshold or the model converges, thereby obtaining the desired detection model.

[0093] Considering camera hardware resources and factory environment constraints, the image resolution captured by optical cameras is not conducive to the AI model's ability to detect and recognize small objects. Furthermore, downsampling the camera's 3820*2180 resolution image severely degrades the features predicted by the high-resolution teacher model, rendering it ineffective in reflecting its learning. By training a multi-scale teacher network and transferring knowledge to the student detection model through knowledge distillation, we can improve detection performance.

[0094] Reference Figure 3 ,The knowledge distillation method for the detection model includes but is not limited to the following steps:

[0095] Step S710, constructing a first teacher model and a second teacher model;

[0096] Step S720, training the first teacher model using the first training image, and training the second teacher model using the second training image;

[0097] Step S730, fusing the first teacher model and the second teacher model to obtain a fused teacher model;

[0098] Step S740: distill the parameters of the fusion teacher model to the detection model.

[0099] For step S710, a teacher model with the same network framework as the detection model is constructed, and the teacher model includes a first teacher model and a second teacher model. The detection model is a student model. The first teacher model and the second teacher model are both provided with multiple feature extraction layers, and the output features of the latter feature extraction layers of the first teacher model and the second teacher model are smaller than the output features of the former feature extraction layer. Specifically, the output features of the latter feature extraction layers of the first teacher model and the second teacher model are 0.5 times the size of the output features of the former feature extraction layer. Then the input image resolution of the student model is also 0.5 times that of the teacher model, which is conducive to the alignment of feature maps at different stages. The feature pyramid can align the feature maps of the teacher model after the network with the feature maps of the student model after the network.

[0100] In step S720, the first teacher model is trained using the first training image, and the second teacher model is trained using the second training image. The resolution of the first training image is higher than that of the second training image. Therefore, the first teacher model is a high-resolution model and performs better in small object detection; the second teacher model is a low-resolution model and performs better in large object detection.

[0101] Reference Figure 7 , for step S730, the output features of the first teacher model and the output features of the second teacher model are fused through the cross-modal fusion module.

[0102] Fusing the first teacher model and the second teacher model to obtain a fused teacher model includes but is not limited to the following steps:

[0103] The output features of the kth feature extraction layer of the first teacher model are fused with the output features of the k+1th feature extraction layer of the second teacher model to obtain fused features;

[0104] The prediction results are obtained by detection based on the fusion features, the loss function is obtained based on the prediction results and the preset loss weights, and the parameters of the teacher model are adjusted through the loss function to obtain the fusion teacher model.

[0105] Reference Figure 8 , where the output features of the k-th feature extraction layer of the first teacher model have the same size as the output features of the k+1-th feature extraction layer of the second teacher model.

[0106] Reference Figure 4 , for step S731, fusing the output features of the kth feature extraction layer of the first teacher model with the output features of the k+1th feature extraction layer of the second teacher model to obtain a fused feature, including but not limited to the following steps:

[0107] Step S7311, concatenating the output features of the kth feature extraction layer of the first teacher model and the output features of the k+1th feature extraction layer of the second teacher model to obtain a concatenation result;

[0108] Step S7312, performing global pooling on the splicing result to obtain a global pooling result;

[0109] Step S7313: The global pooling result is sequentially passed through a fully connected layer for dimensional transformation and an activation function layer for nonlinear transformation to obtain an integer result;

[0110] Step S7314: normalize the integer results through the normalization layer in sequence and map them to obtain probability results;

[0111] Step S7315: Perform weighted summation on the probability results to obtain fusion features.

[0112] The teacher model's feature pyramid structure extracts feature maps with a height of Hps and a width of Wps*256. The feature maps of the high-resolution image input model differ from those of the low-resolution image input model by only one layer. Therefore, the feature maps of the first and second teacher models have the same size at that layer. For example, the third-layer feature map of the first teacher model is the same size as the second-layer feature map of the second teacher model. Similarly, after the feature pyramid extracts features from both resolution models, it is passed to the detection head module for processing.

[0113] During the training process of the model, each iteration uses a random scaling factor q to change the size of the original input image, and the range of q is [0.8, 1].

[0114] The output features of the feature extraction layer of the first teacher model and the output features of the feature extraction layer of the second teacher model are fused through the cross-feature-level fusion module. Specifically, two feature maps Ps and P'S-1 of the same spatial size, with the height, width and number of channels of the feature maps being HPS*WPS*C, are then concatenated. While the height and width remain unchanged, the number of channels is changed from the original C to 2C. Global pooling is then performed, which sums the H*W element values of the feature map of a channel and divides them by H*W to obtain an integer. Since there are 2C channels, there are 2C integers, so the output of global pooling is 1*1*2C. After dimensionality transformation through the fully connected layer and the ReLU activation function layer, a 1*1*2-dimensional integer is generated. The integer is then mapped to a probability through softmax. Finally, PS and P'S-1 are multiplied by their corresponding weights and added together to obtain the final fused feature.

[0115] The loss function for the teacher model is calculated as λ*L(H&H')*(W&W'), where λ is the loss weight and L(H&H')*(W&W') represents the fused features. The loss function is actually the regression loss plus the classification loss and the center loss. The parameters of the teacher model are adjusted using the loss function to obtain the fused teacher model.

[0116] The parameters of the fusion teacher model are distilled to the detection model, allowing the detection model as a student model to learn high-resolution and low-resolution multi-scale features.

[0117] The trained detection model is applied to personnel monitoring in dangerous areas.

[0118] In step S100 , a real-time RGB image and thermal imaging image of the dangerous area are acquired through a camera installed near the dangerous area.

[0119] In step S200, a convolutional layer performs an attention operation on the local semantic information of local neighboring regions of the RGB image and the thermal image to obtain a preliminary saliency map. This is then used to integrate the hierarchical salient objects of PiCA-Nets through a U-Net architecture. The residual refinement block recursively learns the residual between the ground truth and the preliminary saliency map, and refines the preliminary saliency map based on the residual to obtain a saliency map. The residual refinement block alternately utilizes low-level and high-level features, refining the saliency map at each recursive step by adding the previous saliency map to the learned residual.

[0120] In step S300, the saliency maps of the RGB image and the thermal image are input into the detection model to obtain the target detection results. It is understood that the detection model can perform bimodal reasoning, inputting a single RGB or thermal image to obtain the detected target in the RGB image or thermal image, or inputting both RGB and thermal images simultaneously to similarly detect and identify targets in the image.

[0121] The method for monitoring personnel in a dangerous area includes an alarm operation; specifically, when a target detection result indicates that there are personnel in the dangerous area, an alarm operation is performed.

[0122] Reference Figure 5 , the alarm operations specifically include:

[0123] Step S810, obtaining frequency information, area information and time information of the alarm operation;

[0124] Step S820, determining a first dangerous area and a second dangerous area from the dangerous areas according to the frequency information, the area information and the time information;

[0125] Step S830: monitor the first dangerous area using the first detection model, and monitor the second dangerous area using the second detection model.

[0126] Among them, the alarm frequency of the first dangerous area is higher than that of the second dangerous area, and the alarm time of the first dangerous area is longer than that of the second dangerous area; the detection accuracy of the first detection model is higher than that of the second detection model.

[0127] Reference Figure 6 For step S820, determining the first dangerous area and the second dangerous area from the dangerous areas according to the frequency information, the area information and the time information includes but is not limited to the following steps:

[0128] Step S821: frequency information, area information, and time information are used as features to convert and obtain a feature vector corresponding to the dangerous area;

[0129] Step S822, obtaining the distance between the feature of the feature vector corresponding to the dangerous area and the feature of the known feature vector;

[0130] Step S823, determining the label of the feature vector corresponding to the dangerous area according to the label with the largest number among the K known feature vectors closest to the dangerous area;

[0131] Step S824 : clustering the feature vectors corresponding to the plurality of dangerous areas according to the labels of the feature vectors corresponding to the dangerous areas to determine a first dangerous area and a second dangerous area.

[0132] The first danger zone is where dangers frequently occur. Increase attention to the first danger zone and use a high-precision, high-FPS model to detect it. Use a less accurate, lightweight model to detect the second danger zone, thereby improving hardware utilization.

[0133] An embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for monitoring personnel in a hazardous area when executing the computer program.

[0134] The electronic device may be any intelligent terminal including a computer.

[0135] In general, for the hardware structure of the electronic device, the processor can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0136] The memory can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and is called by the processor to execute the methods of the embodiments of this application.

[0137] The input / output interface is used to realize information input and output.

[0138] The communication interface is used to realize the communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0139] The bus transmits information between the various components of the device (such as the processor, memory, input / output interface, and communication interface). The processor, memory, input / output interface, and communication interface communicate with each other within the device through the bus.

[0140] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for executing the above-mentioned method for monitoring personnel in a dangerous area.

[0141] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium. In the above description of this specification, the reference terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" and the like are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0142] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0143] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purpose of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0147] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for monitoring personnel in a dangerous area, characterized in that: include: Acquire RGB and thermal images of hazardous areas; generating a saliency map of the RGB image and the thermal imaging image; Inputting the saliency maps of the RGB image and the thermal image into a detection model to perform personnel monitoring and obtain a target detection result; The generating of the saliency map of the RGB image and the thermal imaging image includes: Performing an attention operation on local semantic information of local adjacent regions of the RGB image and the thermal image through a convolutional layer to obtain a preliminary saliency map; Recursively learning the residual between the true value and the preliminary saliency map through a residual refinement block, and refining the preliminary saliency map according to the residual to obtain a saliency map; The detection model is trained according to the following method: Performing feature extraction on a training set formed by RGB images and thermal imaging images through a convolutional layer to obtain low-level features and high-level features, wherein the high-level features are extracted from the low-level features; Performing an operation of decoupling multi-scale feature interactions into intra-scale feature interactions on the high-level features through an encoder to obtain decoupled features, and performing a cross-scale fusion operation on the low-level features and the decoupled features to obtain fused features; Select the fusion feature whose classification score and intersection-over-union score exceed the preset threshold as the target feature; Calculating the attention weight of the target feature through a decoder, performing weighted summation on the target feature according to the attention weight to obtain a weighted sum value, and performing a nonlinear transformation on the weighted sum value to obtain a prediction result; Adjusting parameters of the detection model according to the prediction result; The method for monitoring personnel in dangerous areas further includes: Constructing a teacher model having the same network framework as the detection model, wherein the teacher model includes a first teacher model and a second teacher model; Training the first teacher model using a first training image and training the second teacher model using a second training image, wherein pixels of the first training image are higher than pixels of the second training image; fusing the first teacher model and the second teacher model to obtain a fused teacher model; Distilling the parameters of the fusion teacher model to the detection model; The first teacher model and the second teacher model are each provided with a plurality of feature extraction layers, and the output feature of a later feature extraction layer of the first teacher model and the second teacher model is smaller in size than the output feature of a previous feature extraction layer; and the fusing of the first teacher model and the second teacher model to obtain a fused teacher model includes: Fusing the output features of the kth feature extraction layer of the first teacher model with the output features of the k+1th feature extraction layer of the second teacher model to obtain fused features, where the output features of the kth feature extraction layer of the first teacher model and the output features of the k+1th feature extraction layer of the second teacher model have the same size; A prediction result is obtained by detecting according to the fusion feature, a loss function is obtained according to the prediction result and a preset loss weight, and the parameters of the teacher model are adjusted by the loss function to obtain a fusion teacher model.

2. A method for monitoring personnel in a dangerous area according to claim 1, characterized in that: The method for monitoring personnel in dangerous areas includes: When the target detection result indicates that there are people in the dangerous area, an alarm operation is performed.

3. A method for monitoring personnel in a dangerous area according to claim 2, characterized in that: The method for monitoring personnel in dangerous areas includes: Obtaining frequency information, area information, and time information of the alarm operation; determining a first dangerous area and a second dangerous area from the dangerous areas according to the frequency information, the area information, and the time information, wherein the alarm frequency of the first dangerous area is higher than the alarm frequency of the second dangerous area, and the alarm time of the first dangerous area is longer than the alarm time of the second dangerous area; The first dangerous area is monitored by a first detection model, and the second dangerous area is monitored by a second detection model. The detection accuracy of the first detection model is higher than the detection accuracy of the second detection model.

4. A method for monitoring personnel in a dangerous area according to claim 3, characterized in that: The determining a first dangerous area and a second dangerous area from the dangerous areas according to the frequency information, the area information and the time information includes: The frequency information, the area information and the time information are used as features to convert and obtain a feature vector corresponding to the dangerous area; Obtaining the distance between the feature of the feature vector corresponding to the dangerous area and the feature of the known feature vector; Determine the label of the feature vector corresponding to the dangerous area according to the label with the largest number among the K known feature vectors closest to the distance; The feature vectors corresponding to the plurality of dangerous areas are clustered according to the labels of the feature vectors corresponding to the dangerous areas to determine a first dangerous area and a second dangerous area.

5. The method for monitoring personnel in a dangerous area according to claim 1, characterized in that: The fusing the output features of the kth feature extraction layer of the first teacher model with the output features of the k+1th feature extraction layer of the second teacher model to obtain fused features, comprising: Splicing the output features of the kth feature extraction layer of the first teacher model and the output features of the k+1th feature extraction layer of the second teacher model to obtain a splicing result; Performing global pooling on the splicing result to obtain a global pooling result; The global pooling result is sequentially passed through a fully connected layer for dimension transformation and an activation function layer for nonlinear transformation to obtain an integer result; Normalize the integer results by passing them through normalization layers in sequence, and map them to obtain probability results; The probability results are weighted and summed to obtain the fusion feature.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for monitoring personnel in a hazardous area according to any one of claims 1 to 5 when executing the computer program.

7. A computer storage medium, characterized in that Computer-executable instructions are stored, and the computer-executable instructions are used to execute the method for monitoring personnel in a hazardous area according to any one of claims 1 to 5.

Images