A method, device and terminal for identifying safety protection compliance of operating personnel

Abstract

The present invention is applicable to the technical field of safety protection compliance identification, and provides a method, device, and terminal for identifying safety protection compliance of operators. The method includes: acquiring an operation image of a target operation area; detecting operators and safety protection equipment worn by operators in the operation image based on a preset target detection model, obtaining a first framed image that selects the operators, and a second framed image that selects the safety protection equipment worn by the operators; finally, inputting the first framed image and the second framed image into a preset multi-task recognition model to determine whether the safety protection equipment worn by each operator in the target operation area is in compliance. The present invention can improve the efficiency of safety protection compliance identification of operators.

Description

Technical Field

[0001] The present invention relates to the field of safety protection compliance identification technology, and in particular to a method, device and terminal for safety protection compliance identification of operating personnel. Background Art

[0002] With the development of economy and the improvement of people's living standards, users have higher and higher requirements for power supply reliability, and non-stop operation is the most direct and effective way to improve power supply reliability. However, non-stop workers need to work in a working environment with a voltage of up to 10kV, so they need to wear various safety protection equipment. At present, the issue of identifying whether the safety protection of non-stop workers is compliant has attracted more and more attention.

[0003] Due to the large number and variety of safety protection equipment, there are also many visual inspection tasks for safety protection compliance. The existing technology usually sets a separate algorithm for each visual inspection task. This design method leads to complicated identification steps for safety protection, which reduces the efficiency of safety protection identification for operators. Summary of the invention

[0004] The present invention provides a method, device and terminal for identifying compliance of safety protection for operators, so as to solve the problems of complicated steps and low identification efficiency in the prior art when identifying whether safety protection is compliant.

[0005] In a first aspect, an embodiment of the present invention provides a method for identifying safety protection compliance of workers, comprising:

[0006] Acquire a working image of a target working area;

[0007] Based on a preset target detection model, detect the operator and the safety protection equipment worn by the operator in the operation image, and obtain a first framed image in which the operator is framed, and a second framed image in which the safety protection equipment worn by the operator is framed;

[0008] The first framed image and the second framed image are input into a preset multi-task recognition model to determine whether the safety protection equipment worn by each operator in the target working area is in compliance with regulations.

[0009] In a possible implementation, the security protection compliance identification method further includes a method for training a target detection model;

[0010] The training method of the target detection model includes:

[0011] Acquire a historical operation image of a target operation area and first annotation information corresponding to the historical operation image; wherein the first annotation information is used to frame an operator in the historical operation image and a safety protection device worn by the operator in the historical operation image;

[0012] The historical operation image and the first annotation information are input into a preset CenterNet network structure to train and obtain a target detection model.

[0013] In a possible implementation, the security protection compliance identification method further includes a training method for a target detection model and a training method for a multi-task recognition model;

[0014] The training method of the multi-task recognition model includes:

[0015] Obtain a first historical framed image in which a worker is framed in a target operation area, a second historical framed image in which a safety protection device worn by the worker is framed, and second annotation information corresponding to the worker; wherein the second annotation information is used to identify whether the safety protection device worn by the worker is compliant;

[0016] The first historical framed image, the second historical framed image and the second annotation information are input into a preset neural network structure to train a multi-task recognition model.

[0017] In a possible implementation, the security protection compliance identification method further includes a training method for a target detection model and a training method for a multi-task recognition model;

[0018] The training method of the target detection model and the training method of the multi-task recognition model include:

[0019] Acquire historical operation images of the target operation area and third annotation information corresponding to the historical operation images; wherein the third annotation information is used to select the operator in the historical operation image, select the safety protection equipment worn by the operator in the historical operation image, and mark whether the safety protection equipment worn by the operator is compliant;

[0020] Inputting the historical operation image into a preset CenterNet network structure, obtaining a first historical framed image in which an operator is framed, and a second historical framed image in which a safety protection device worn by the operator is framed;

[0021] Inputting the first historical framed image and the second historical framed image into a preset neural network structure to obtain compliance recognition results of the safety protection equipment worn by the operator;

[0022] Determine a joint error of a target detection model and a multi-task recognition model based on the first historical framed image, the second historical framed image, the compliance recognition result, and the third annotation information;

[0023] If the joint error is greater than a preset error value, the weight coefficient of the CenterNet network structure and the weight coefficient of the neural network structure are optimized according to a preset rule, and the step of inputting the historical job image into the preset CenterNet network structure is returned to be executed;

[0024] If the joint error is not greater than the preset error value, the current CenterNet network is used as the target detection model, and the current neural network is used as the multi-task recognition model;

[0025] Among them, the current CenterNet network includes the CenterNet network structure and the weight coefficient currently corresponding to the CenterNet network structure, and the current neural network includes the neural network structure and the weight coefficient currently corresponding to the neural network structure.

[0026] In a possible implementation manner, determining a joint error of a target detection model and a multi-task recognition model based on the first historical framed image, the second historical framed image, the compliance recognition result, and the third annotation information includes:

[0027] The first historical framed image, the second historical framed image, the compliance recognition result and the third annotation information are compared to determine the joint error of the target detection model and the multi-task recognition model:

[0028] L = λ1l1+λ2l2+λ3l3;

[0029] Among them, l1 is the error obtained by comparing the first historical framed image with the third annotation information, l2 is the error obtained by comparing the second historical framed image with the third annotation information, l3 is the error obtained by comparing the compliant recognition result with the third annotation information, and λ1, λ2, and λ3 are the error factors of l1, l2, and l3 respectively.

[0030] In a possible implementation, inputting the first framed image and the second framed image into a preset multi-task recognition model to determine whether the safety protection equipment worn by each operator in the target working area is compliant includes:

[0031] Inputting the first framed image and the second framed image into a preset multi-task recognition model to obtain a multi-task recognition result;

[0032] Accumulate the multi-task recognition results of the preset time;

[0033] If the multi-task recognition result within the preset time period shows that the time period during which a worker does not wear safety protection equipment in compliance with regulations exceeds a preset threshold, it is determined that the worker does not wear safety protection equipment in compliance with regulations;

[0034] If the multi-task recognition result within the preset time period shows that the time period during which a worker does not wear safety protection equipment in compliance with regulations within the preset time period does not exceed a preset threshold, it is determined that the worker is wearing safety protection equipment in compliance with regulations.

[0035] In a possible implementation, the operator safety protection compliance identification method further includes:

[0036] If it is determined that a worker's safety protection equipment is not worn in compliance with regulations, a control is performed to send a prompt signal; the prompt signal is used to remind the worker that he or she has worn the equipment in violation of regulations.

[0037] A second aspect of the present invention provides a worker safety protection compliance identification device, comprising:

[0038] An image acquisition unit, used for acquiring an operation image of a target operation area;

[0039] An image detection unit, configured to detect the operator and the safety protection equipment worn by the operator in the operation image based on a preset target detection model to obtain a first framed image in which the operator is framed, and a second framed image in which the safety protection equipment worn by the operator is framed;

[0040] A judgment unit is used to input the first framed image and the second framed image into a preset multi-task recognition model to determine whether the safety protection equipment worn by each operator in the target working area is in compliance with regulations.

[0041] A third aspect of an embodiment of the present invention provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for identifying safety protection compliance of operating personnel as described in any one of the above items are implemented.

[0042] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method for identifying safety protection compliance of operators as described in any of the above items are implemented.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention obtains the working image of the target working area, and obtains the first framed image of the working personnel and the second framed image of the safety protection equipment worn by the working personnel through the preset target detection model, and then inputs the first framed image and the second framed image into the preset multi-task recognition model for compliance recognition. In other words, the present invention first selects and classifies the working personnel and the safety protection equipment worn by the working personnel in the working image, and then performs multi-task recognition on the selected working personnel. Only two models are trained to complete the compliance detection of all the safety protection equipment of the working personnel, thereby improving the efficiency of the safety protection compliance recognition of the working personnel. On this basis, different from the solution of the prior art in which a deep learning model is used to comprehensively complete the image positioning and recognition in the recognition process, the present invention divides the image selection and the recognition of the safety protection equipment into two steps, which can not only reduce the spatial complexity of the network model, further improve the safety compliance recognition efficiency, but also improve the accuracy of safety compliance detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 It is a flowchart of the implementation of the method for identifying the safety protection compliance of workers provided by an embodiment of the present invention;

[0047] Figure 2 It is a schematic diagram of the structure of a worker safety protection compliance identification device provided by an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0051] When non-stop workers are working in the work area, the working environment is a 10KV high voltage environment, so they need to wear various types of safety protection equipment to protect themselves. After obtaining the working images of non-stop workers in the target work area, it is found that in many cases there are multiple workers in one working image, and each worker wears many types of safety protection equipment, such as insulating clothes or insulating capes, insulating helmets, insulating gloves, and insulating safety belts. If you want to identify these safety protection equipment worn by each worker one by one and judge whether they are compliant, it is a huge task, and due to the large amount of tasks, the accuracy of identifying each safety protection equipment and judging compliance will also decrease.

[0052] See also Figure 1 , which shows a flowchart of the implementation of the worker safety protection compliance identification method provided by an embodiment of the present invention, and is described in detail as follows:

[0053] In step 101, a work image of a target work area is acquired.

[0054] In the embodiment of the present invention, the target operation area refers to the working scene where the non-stop operation personnel are working, and the operation image of the target operation area must include the non-stop operation personnel. There are various ways to obtain the operation image of the target operation area, such as directly capturing the operation image of the target operation area through a device equipped with a camera module, or directly obtaining it from a storage device containing the operation image, etc. This embodiment does not limit the method for obtaining the operation image of the target operation area.

[0055] In the embodiment of the present invention, it can be understood that after the work image of the target work area is acquired, the acquired work image can be preprocessed, wherein the preprocessing of the work image includes but is not limited to image enhancement, image restoration, and cropping all the work images to the same size.

[0056] In step 102, based on a preset target detection model, the workers and the safety protection equipment worn by the workers in the work image are detected to obtain a first framed image in which the workers are framed, and a second framed image in which the safety protection equipment worn by the workers is framed.

[0057] In an embodiment of the present invention, the preset target detection model is a preset CenterNet network structure, which detects each target in the work image as a point, that is, a target is represented by the center point of the target box. The preset CenterNet network structure includes a main network that can extract features from the input work image and three branch prediction networks, wherein the three branch prediction networks are a heat map branch prediction network, a width and height branch prediction network, and a target center point offset branch prediction network.

[0058] In an embodiment of the present invention, the main network used for feature extraction in the preset CenterNet network structure is a ResNet50 neural network, which includes 1 convolution pooling layer, 12 Identity Block modules, 4 Conv Block modules, and 1 fully connected layer, wherein the Conv Block module is a module comprising 3 convolution layers for changing the input and output dimensions, wherein the first part thereof includes a 1×1 convolution layer for changing the input and output dimensions, the second part includes a 3×3 convolution layer for feature extraction, the third part includes a 1×1 convolution layer for changing the input and output dimensions, and the fourth part includes a 1×1 convolution layer for changing the input and output dimensions and connected to the output; the Identity Block module is a module for a continuous series network comprising 3 convolution layers, wherein the first part thereof includes a 1×1 convolution layer for changing the input and output dimensions, the second part includes a 3×3 convolution layer for feature extraction, the third part includes a 1×1 convolution layer for changing the output dimension, and the fourth part is the input image jump connected to the output part. The ResNet50 neural network can downsample the input job image and obtain a feature map that is downsampled 32 times. Compared with the general ResNet18 neural network, the network's feature extraction capability and detection accuracy are improved. In addition, since the Conv Block module and the Identity Block module first reduce the dimension of the input image and then use the convolution layer to extract features, compared with the ResNet18 operation that directly extracts features without reducing the dimension, the network parameters are reduced and the network training speed is improved.

[0059] In an embodiment of the present invention, the heat map branch prediction network can output a heat map of the job image. The heat map represents category information. Each category in the image will have a heat map. The position of the target center point can be predicted by performing peak detection on the heat map. On each heat map, if there is a target center point at a certain coordinate, a Gaussian circle is generated at the coordinate; the width and height branch prediction network can be used to predict the width and height information of the target and generate a predicted target frame; the center point offset branch prediction network is used to predict the offset of the target center point.

[0060] In the embodiment of the present invention, the targets included in the operation image include but are not limited to the operators and the safety protection equipment worn by the operators, but in the embodiment of the present invention, only the operators and the safety protection equipment worn by the operators are considered, wherein the safety protection equipment worn by the operators includes insulating clothing or insulating capes, insulating helmets, insulating gloves, and insulating safety belts. Accordingly, the first framed image is to input the operation image into the preset CenterNet network structure, and output an image in which the operators in the operation image are framed; the second framed image is to input the operation image into the preset CenterNet network structure, and output an image in which the safety protection equipment worn by the operators in the operation image are framed.

[0061] In step 103, the first framed image and the second framed image are input into a preset multi-task recognition model to determine whether the safety protection equipment worn by each operator in the target working area is in compliance with regulations.

[0062] In an embodiment of the present invention, a preset multi-task recognition model is used to determine whether the wearing of safety protection equipment is in compliance. The recognition modules of the preset multi-task recognition model include: an insulating clothing or insulating shawl recognition module, an insulating helmet recognition module, an insulating glove recognition module, and an insulating safety belt recognition module. Different recognition modules correspond to different recognition tasks, wherein different recognition tasks can be performed simultaneously. For example, after determining the features of the input framed image, multiple recognition tasks can be completed simultaneously. If the corresponding information is recognized, such as recognizing that an insulating helmet is worn, the output of the insulating helmet recognition module displays the information that the operator is wearing an insulating helmet.

[0063] In an embodiment of the present invention, the first framed image may be input into a preset multi-task recognition model for feature extraction. This step is used to identify whether the operator in the first framed image can be identified. The following situations may exist in the acquired work image: there is no operator in the image, the part of the operator to be identified is incomplete, and these work images cannot be used for multi-task recognition. If the first framed image is input into the preset multi-task network model and it is identified that the first framed image cannot be multi-task recognized, all task recognition modules output the same information, and then the first framed image and the second framed image corresponding to the work image are filtered out. The remaining second framed image after screening is cropped. The cropping step is to crop the operator in the corresponding second framed image according to the target frame of the operator in the remaining first framed image according to the target frame in the first framed image. The purpose of cropping is to enable the multi-task network model to identify the safety protection equipment worn by one operator when performing task recognition, so as to prevent interference from safety protection equipment worn by other operators in the image. Finally, the cropped second framed image is normalized and unified in size.

[0064] Optionally, in one embodiment, the above step 103 may include:

[0065] The first framed image and the second framed image are input into a preset multi-task recognition model to obtain a multi-task recognition result.

[0066] Accumulate the multi-task recognition results for the preset time.

[0067] If the multi-task recognition result within the preset time period shows that the time period during which a certain operator does not wear safety protection equipment in compliance with regulations exceeds a preset threshold, it is determined that the operator does not wear safety protection equipment in compliance with regulations.

[0068] If the multi-task recognition result within the preset time period shows that the time period during which a worker does not wear safety protection equipment in compliance with regulations within the preset time period does not exceed a preset threshold, it is determined that the worker is wearing safety protection equipment in compliance with regulations.

[0069] As mentioned above, the first framed image and the second framed image are input into the preset multi-task recognition model for recognition. Since non-power outage workers need to stay in the working environment for a period of time when working, if you want to identify whether the safety protection equipment worn by a certain worker is compliant, you need to identify whether the safety protection equipment worn by the worker within a certain period of time is compliant. If you only identify whether the safety protection equipment worn by a certain worker in a work image is compliant, it may cause the safety protection equipment worn by the worker at that moment to be compliant, and the safety protection equipment worn at the next moment to be non-compliant. However, the multi-task recognition model believes that the safety protection tools worn by the worker are compliant based on the judgment result at that moment, resulting in a safety accident.

[0070] In this embodiment, by accumulating the multi-task recognition results for a certain period of time, and extracting the non-compliant time of the safety protection equipment worn by a certain operator from the period of time, the non-compliant time is compared with a specified threshold. If it is higher than the specified threshold, the safety protection equipment worn by the certain operator is considered to be non-compliant. If it is lower than the specified threshold, the safety protection equipment worn by the certain operator is considered to be compliant. By accumulating this method, the recognition accuracy of the multi-task recognition model can also be improved.

[0071] Optionally, in one embodiment, the above step 103 may further include:

[0072] If it is determined that a worker's safety protection equipment is not worn in compliance with regulations, the control will send a prompt signal; the prompt signal is used to remind the worker that he or she has not worn the equipment in compliance with regulations.

[0073] In this embodiment, if it is found through identification and judgment that the safety protection equipment worn by the workers is non-compliant, a prompt signal can be sent to the workers in the target work area so that the workers can adjust the non-compliant safety protection equipment they are wearing. The prompt signal sent to the workers in the target work area can have a variety of forms, which are not limited here.

[0074] In some embodiments, the method further includes a method for training a target detection model;

[0075] The training methods for the target detection model include:

[0076] Obtain historical operation images of the target operation area and first annotation information corresponding to the historical operation images; wherein the first annotation information is used to frame the operators in the historical operation images and the safety protection equipment worn by the operators in the historical operation images.

[0077] The historical operation images and the first annotation information are input into the preset CenterNet network structure to train the target detection model.

[0078] In an embodiment of the present invention, before training the CenterNet network structure, it is necessary to first obtain a large number of historical work images of the target work area, and select the workers and the safety protection equipment worn by the workers in the historical work images to obtain the first annotation information. In this way, the historical work images and annotation information will be used as input to train the target detection model. In the process of applying the target detection model, it is not necessary to select the work images. It is only necessary to input the work images into the target detection model to output the first framed image of the workers and the second framed image of the safety protection equipment worn by the workers.

[0079] In an embodiment of the present invention, before training the CenterNet network structure, the loss function of the CenterNet network structure is first determined, and the loss function includes heat map loss, width and height prediction loss, and center point offset loss. Various losses include their loss values ​​and their corresponding weight coefficients. The historical operation image is input into the preset CenterNet network structure to obtain the first historical framed image and the second historical framed image. The specific acquisition method is the same as the record of step 102 in the embodiment, and will not be repeated here. Then the first historical framed image, the second historical framed image and the first annotation information are compared one by one according to the loss function, and the difference between them is reduced by repeated forward and reverse propagation. Finally, when the difference between the output result of the preset CenterNet network structure and the first annotation information is small enough to be ignored, the weight coefficient of the preset CenterNet network structure at this time is the final weight coefficient, and the preset CenterNet network structure at this time is the target detection model obtained by training.

[0080] In some embodiments, the method further includes a method for training a multi-task recognition model;

[0081] The training methods of the multi-task recognition model include:

[0082] Obtain a first historical framed image that frames a worker in a target work area, a second historical framed image that frames the safety protection gear worn by the worker, and second annotation information corresponding to the worker; wherein the second annotation information is used to identify whether the safety protection gear worn by the worker is compliant.

[0083] The first historical framed image, the second historical framed image and the second annotation information are input into a preset neural network structure and trained to obtain a multi-task recognition model.

[0084] In an embodiment of the present invention, before training the multi-task recognition model, the historical work images of the target work area are first input into the trained target detection model to obtain a first historical framed image and a second historical framed image, and then the first historical framed image and the second historical framed image are annotated to obtain the second annotation information according to whether the insulating clothing or insulating shawl, insulating helmet, insulating gloves, and insulating safety belt actually worn by the operator are compliant. In this way, the first historical framed image, the second historical framed image, and the second annotation information will be used as input to train the model. In the process of applying the trained multi-task recognition model, no annotation information is required. Only the first framed image and the second framed image are needed to output the judgment result of whether the insulating clothing or insulating shawl, insulating helmet, insulating gloves, and insulating safety belt actually worn by the operator are compliant.

[0085] In an embodiment of the present invention, before training the preset neural network structure, the loss function of the preset neural network structure is first determined, and the historical operation image is input into the preset neural network structure to obtain a judgment result on whether the safety protection equipment worn by the operator is compliant. The specific judgment result is the same as the record of step 103 embodiment, and will not be repeated here. Then the judgment result is compared one by one with the second annotation information according to the loss function, and the difference between them is reduced by repeated forward and backward transmission. Finally, when the difference between the output result of the preset neural network structure and the second annotation information is small enough to be ignored, the weight coefficient of the preset neural network structure at this time is the final weight coefficient, and the preset neural network structure at this time is the multi-task recognition model obtained by training.

[0086] In some embodiments, the method further includes a training method for a target detection model and a training method for a multi-task recognition model;

[0087] The training method of the target detection model and the training method of the multi-task recognition model include:

[0088] Obtain historical operation images of the target operation area and third annotation information corresponding to the historical operation images; wherein the third annotation information is used to select the operators in the historical operation images, select the safety protection equipment worn by the operators in the historical operation images, and mark whether the safety protection equipment worn by the operators is compliant.

[0089] The historical operation images are input into the preset CenterNet network structure to obtain a first historical framed image in which the operator is framed, and a second historical framed image in which the safety protection equipment worn by the operator is framed.

[0090] The first historical framed image and the second historical framed image are input into a preset neural network structure to obtain compliance recognition results of the safety protection equipment worn by the operators.

[0091] The joint error of the target detection model and the multi-task recognition model is determined based on the first historical framed image, the second historical framed image, the compliance recognition result and the third annotation information.

[0092] If the joint error is greater than the preset error value, the weight coefficient of the CenterNet network structure and the weight coefficient of the neural network structure are optimized according to the preset rules, and the step of inputting the historical job image into the preset CenterNet network structure is returned.

[0093] If the joint error is not greater than the preset error value, the current CenterNet network is used as the target detection model and the current neural network is used as the multi-task recognition model.

[0094] Among them, the current CenterNet network includes the CenterNet network structure and the weight coefficient currently corresponding to the CenterNet network structure, and the current neural network includes the neural network structure and the weight coefficient currently corresponding to the neural network structure.

[0095] In an embodiment of the present invention, before training the target detection model and the multi-task network model, a large number of historical operation images of the target operation area are first obtained, and the operators and the safety protection equipment worn by the operators are framed in the historical operation images, and whether the safety protection equipment worn by the operators is compliant is marked, thereby obtaining the third annotation information; the historical operation images are input into the preset CenterNet network structure to obtain the first historical framed image and the second historical framed image. The specific acquisition method is the same as that recorded in the step 102 embodiment, which is not repeated here; the first historical framed image and the second historical framed image are input into the preset neural network structure to determine whether the safety protection equipment worn by the operators is compliant. The specific judgment method is the same as that recorded in the step 103 embodiment, which is not repeated here; the differences between the first historical framed image, the second historical framed image, the result of the compliance recognition and the third annotation information are compared to determine a joint error, which is the weighted sum of the error of the first historical framed image, the error of the second historical framed image, and the error of the compliance recognition result.

[0096] In an embodiment of the present invention, if the joint error is greater than the preset error value, it means that the detection and recognition process of the preset CenterNet network structure and the preset neural network structure in the present invention is not accurate enough, then the first historical framed image, the second historical framed image, the compliant recognition result and the third annotation information are compared one by one according to the loss function, and the difference between them is reduced by repeated forward and backward propagation. Finally, when the difference between the output results of the preset CenterNet network structure and the preset neural network structure and the third annotation information is small enough to be ignored, the weight coefficient of the preset CenterNet network structure and the weight coefficient of the preset neural network structure are the final weight coefficients.

[0097] In an embodiment of the present invention, the CenterNet network structure and the neural network structure are trained together. Compared with the conventional method of training two models separately, the adaptability of the two models can be improved, and the accuracy of safety protection compliance identification of operators can be improved. For example, in the process of training the two models separately, after the preset CenterNet network structure inputs the historical operation image, the difference between the first historical framed image, the second historical framed image and the first annotation information obtained is small enough to be ignored, indicating that the CenterNet network structure has been trained and can be used for target detection. However, at this time, the difference between the compliance identification result obtained by inputting the first historical framed image and the second historical framed image into the preset neural network structure and the second annotation information is very large, indicating that the preset neural network structure cannot be used to identify whether the safety protection equipment of operators is compliant. In general, in the method of separate training, the accuracy of the final recognition result may be reduced due to the low accuracy of one of the models after training. However, if the two models are trained jointly and only one loss function is set during the training process, the two models can be matched with each other. The accuracy of the overall recognition result will not be affected by the training degree of one of the models, thereby improving the accuracy of the final recognition result.

[0098] Optionally, determining the joint error of the target detection model and the multi-task recognition model based on the first historical framed image, the second historical framed image, the compliance recognition result, and the third annotation information may include:

[0099] The first historical framed image, the second historical framed image, the compliance recognition result and the third annotation information are compared to determine the joint error of the target detection model and the multi-task recognition model:

[0100] L = λ1l1+λ2l2+λ3l3;

[0101] Among them, l1 is the error obtained by comparing the first historical framed image with the third annotation information, l2 is the error obtained by comparing the second historical framed image with the third annotation information, l3 is the error obtained by comparing the compliant recognition result with the third annotation information, and λ1, λ2, and λ3 are the error factors of l1, l2, and l3 respectively.

[0102] As can be seen from the above, the present invention obtains the working image of the target working area, and obtains the first framed image of the working personnel and the second framed image of the safety protection equipment worn by the working personnel through the preset target detection model, and then inputs the first framed image and the second framed image into the preset multi-task recognition model for compliance recognition. Since the working personnel and the safety protection equipment worn by the working personnel in the working image are first framed and classified, and then the multi-task recognition is performed on the framed working personnel, the original recognition method is divided into two more accurate steps, and the multi-task network model is used in the multi-task recognition process, which improves the accuracy of the compliance recognition of the working personnel's safety protection.

[0103] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0104] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0105] Figure 2 The following is a schematic diagram showing the structure of a device for identifying compliance with worker safety protection provided by an embodiment of the present invention, which is described in detail as follows:

[0106] like Figure 2 As shown, the operator safety protection compliance identification device 2 may include: an image acquisition unit 21 , an image detection unit 22 , and a judgment unit 23 .

[0107] An image acquisition unit 21 is used to acquire an operation image of a target operation area;

[0108] The image detection unit 22 is used to detect the operator and the safety protection equipment worn by the operator in the operation image based on a preset target detection model to obtain a first framed image in which the operator is framed, and a second framed image in which the safety protection equipment worn by the operator is framed;

[0109] The judgment unit 23 is used to input the first framed image and the second framed image into a preset multi-task recognition model to judge whether the safety protection equipment worn by each operator in the target working area is in compliance with regulations.

[0110] In a possible implementation, the operator safety protection compliance identification device 2 may also include:

[0111] The model training unit 24 is used to perform the following steps:

[0112] Obtain historical operation images of the target operation area and first annotation information corresponding to the historical operation images; wherein the first annotation information is used to frame the operators in the historical operation images and the safety protection equipment worn by the operators in the historical operation images.

[0113] The historical operation images and the first annotation information are input into the preset CenterNet network structure to train the target detection model.

[0114] The model training unit 24 is further configured to perform the following steps:

[0115] Obtain a first historical framed image that frames a worker in a target work area, a second historical framed image that frames the safety protection gear worn by the worker, and second annotation information corresponding to the worker; wherein the second annotation information is used to identify whether the safety protection gear worn by the worker is compliant.

[0116] The first historical framed image, the second historical framed image and the second annotation information are input into a preset neural network structure and trained to obtain a multi-task recognition model.

[0117] The model training unit 24 is further configured to perform the following steps:

[0118] Obtain historical operation images of the target operation area and third annotation information corresponding to the historical operation images; wherein the third annotation information is used to select the operators in the historical operation images, select the safety protection equipment worn by the operators in the historical operation images, and mark whether the safety protection equipment worn by the operators is compliant.

[0119] The historical operation images are input into the preset CenterNet network structure to obtain a first historical framed image in which the operator is framed, and a second historical framed image in which the safety protection equipment worn by the operator is framed.

[0120] The first historical framed image and the second historical framed image are input into a preset neural network structure to obtain compliance recognition results of the safety protection equipment worn by the operators.

[0121] The joint error of the target detection model and the multi-task recognition model is determined based on the first historical framed image, the second historical framed image, the compliance recognition result and the third annotation information.

[0122] If the joint error is greater than the preset error value, the weight coefficient of the CenterNet network structure and the weight coefficient of the neural network structure are optimized according to the preset rules, and the step of inputting the historical job image into the preset CenterNet network structure is returned to be executed.

[0123] If the joint error is not greater than the preset error value, the current CenterNet network is used as the target detection model, and the current neural network is used as the multi-task recognition model; wherein, the current CenterNet network includes the CenterNet network structure and the weight coefficient currently corresponding to the CenterNet network structure, and the current neural network includes the neural network structure and the weight coefficient currently corresponding to the neural network structure.

[0124] Correspondingly, the model training unit 24 is specifically used to compare the first historical framed image, the second historical framed image, the compliant recognition result and the third annotation information to determine that the joint error of the target detection model and the multi-task recognition model is: L=λ1l1+λ2l2+λ3l3; wherein l1 is the error obtained by comparing the first historical framed image with the third annotation information, l2 is the error obtained by comparing the second historical framed image with the third annotation information, and l3 is the error obtained by comparing the compliant recognition result with the third annotation information, and λ1, λ2, and λ3 are the error factors of l1, l2, and l3 respectively.

[0125] Correspondingly, the judgment unit 23 is specifically configured to input the first framed image and the second framed image into a preset multi-task recognition model to obtain a multi-task recognition result.

[0126] Accumulate the multi-task recognition results for the preset time.

[0127] If the multi-task recognition result within the preset time length shows that the time period during which a worker does not wear safety protection equipment in compliance with regulations exceeds a preset threshold, then it is determined that the worker does not wear safety protection equipment in compliance with regulations; if the multi-task recognition result within the preset time length shows that the time period during which a worker does not wear safety protection equipment in compliance with regulations does not exceed a preset threshold, then it is determined that the worker does wear safety protection equipment in compliance with regulations.

[0128] Correspondingly, the judgment unit 23 can also be used to control the issuance of a prompt signal when it is determined that a worker's safety protection equipment is not worn in compliance with regulations; the prompt signal is used to prompt the worker that he or she has engaged in non-compliant wearing behavior.

[0129] The above-mentioned device can execute the method provided in the embodiment of the present application, and has the corresponding functional units and beneficial effects of the execution method.

[0130] Figure 3FIG. 1 is a schematic diagram of a terminal provided by an embodiment of the present invention. Figure 3 As shown, the vehicle terminal 3 of this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, the steps in the above-mentioned embodiments of the safety protection compliance identification method for operators are implemented, such as Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of each unit in the above-mentioned device embodiments are realized, for example, Figure 2 The functions of units 21 to 24 are shown.

[0131] Exemplarily, the computer program 32 may be divided into one or more units, which are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 32 in the terminal 3. For example, the computer program 32 may be divided into an image acquisition unit, an image detection unit, a judgment unit, and a model training unit.

[0132] The terminal may be a drone, which may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of terminal 3 and does not constitute a limitation on terminal 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal may also include input and output devices, network access devices, buses, etc.

[0133] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0134] The memory 31 may be an internal storage unit of the terminal 3, such as a hard disk or memory of the terminal 3. The memory 31 may also be an external storage device of the terminal 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal 3. Further, the memory 31 may also include both an internal storage unit of the terminal 3 and an external storage device. The memory 31 is used to store the computer program and other programs and data required by the terminal. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0135] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0136] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0137] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0138] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0142] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for identifying safety protection compliance of operators, characterized in that: include: Acquire a working image of a target working area; Based on a preset target detection model, detect the operator and the safety protection equipment worn by the operator in the operation image, and obtain a first framed image in which the operator is framed, and a second framed image in which the safety protection equipment worn by the operator is framed; Inputting the first framed image and the second framed image into a preset multi-task recognition model to determine whether the safety protection equipment worn by each operator in the target working area complies with regulations; The security protection compliance identification method also includes a training method for a target detection model and a training method for a multi-task recognition model; The training method of the target detection model and the training method of the multi-task recognition model include: Acquire historical operation images of the target operation area and third annotation information corresponding to the historical operation images; wherein the third annotation information is used to select the operator in the historical operation image, select the safety protection equipment worn by the operator in the historical operation image, and mark whether the safety protection equipment worn by the operator is compliant; Inputting the historical operation image into a preset CenterNet network structure, obtaining a first historical framed image in which an operator is framed, and a second historical framed image in which a safety protection device worn by the operator is framed; Inputting the first historical framed image and the second historical framed image into a preset neural network structure to obtain compliance recognition results of the safety protection equipment worn by the operator; Determine a joint error of a target detection model and a multi-task recognition model based on the first historical framed image, the second historical framed image, the compliance recognition result, and the third annotation information; If the joint error is greater than a preset error value, the weight coefficient of the CenterNet network structure and the weight coefficient of the neural network structure are optimized according to a preset rule, and the step of inputting the historical job image into the preset CenterNet network structure is returned to be executed; If the joint error is not greater than the preset error value, the current CenterNet network is used as the target detection model, and the current neural network is used as the multi-task recognition model; Among them, the current CenterNet network includes the CenterNet network structure and the weight coefficient currently corresponding to the CenterNet network structure, and the current neural network includes the neural network structure and the weight coefficient currently corresponding to the neural network structure.

2. The method for identifying worker safety protection compliance according to claim 1, characterized in that: The determining a joint error of a target detection model and a multi-task recognition model based on the first historical framed image, the second historical framed image, the compliance recognition result, and the third annotation information includes: The first historical framed image, the second historical framed image, the compliance recognition result and the third annotation information are compared to determine the joint error of the target detection model and the multi-task recognition model: L = λ1l1+λ2l2+λ3l3; Among them, l1 is the error obtained by comparing the first historical framed image with the third annotation information, l2 is the error obtained by comparing the second historical framed image with the third annotation information, l3 is the error obtained by comparing the compliant recognition result with the third annotation information, and λ1, λ2, and λ3 are the error factors of l1, l2, and l3 respectively.

3. The method for identifying worker safety protection compliance according to claim 1 or 2, characterized in that: The step of inputting the first framed image and the second framed image into a preset multi-task recognition model to determine whether the safety protection equipment worn by each operator in the target operation area complies with regulations includes: Inputting the first framed image and the second framed image into a preset multi-task recognition model to obtain a multi-task recognition result; Accumulate the multi-task recognition results of the preset time; If the multi-task recognition result within the preset time period shows that the time period during which a worker does not wear safety protection equipment in compliance with regulations exceeds a preset threshold, it is determined that the worker does not wear safety protection equipment in compliance with regulations; If the multi-task recognition result within the preset time period shows that the time period during which a worker does not wear safety protection equipment in compliance with regulations within the preset time period does not exceed a preset threshold, it is determined that the worker is wearing safety protection equipment in compliance with regulations.

4. The method for identifying worker safety protection compliance according to claim 1 or 2, characterized in that: The operator safety protection compliance identification method also includes: If it is determined that a worker's safety protection equipment is not worn in compliance with regulations, a control is performed to send a prompt signal; the prompt signal is used to remind the worker that he or she has worn the equipment in violation of regulations.

5. A worker safety protection compliance identification device, characterized in that: include: An image acquisition unit, used for acquiring an operation image of a target operation area; An image detection unit, configured to detect the operator and the safety protection equipment worn by the operator in the operation image based on a preset target detection model to obtain a first framed image in which the operator is framed, and a second framed image in which the safety protection equipment worn by the operator is framed; A judgment unit, used for inputting the first framed image and the second framed image into a preset multi-task recognition model to judge whether the safety protection equipment worn by each operator in the target operation area is in compliance with regulations; The security protection compliance identification method also includes a training method for a target detection model and a training method for a multi-task recognition model; The training method of the target detection model and the training method of the multi-task recognition model include: Acquire historical operation images of the target operation area and third annotation information corresponding to the historical operation images; wherein the third annotation information is used to select the operator in the historical operation image, select the safety protection equipment worn by the operator in the historical operation image, and mark whether the safety protection equipment worn by the operator is compliant; Inputting the historical operation image into a preset CenterNet network structure, obtaining a first historical framed image in which an operator is framed, and a second historical framed image in which a safety protection device worn by the operator is framed; Inputting the first historical framed image and the second historical framed image into a preset neural network structure to obtain compliance recognition results of the safety protection equipment worn by the operator; Determine a joint error of a target detection model and a multi-task recognition model based on the first historical framed image, the second historical framed image, the compliance recognition result, and the third annotation information; If the joint error is greater than a preset error value, the weight coefficient of the CenterNet network structure and the weight coefficient of the neural network structure are optimized according to a preset rule, and the step of inputting the historical job image into the preset CenterNet network structure is returned to be executed; If the joint error is not greater than the preset error value, the current CenterNet network is used as the target detection model, and the current neural network is used as the multi-task recognition model; Among them, the current CenterNet network includes the CenterNet network structure and the weight coefficient currently corresponding to the CenterNet network structure, and the current neural network includes the neural network structure and the weight coefficient currently corresponding to the neural network structure.

6. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images