Unsafe behavior identification method and device, equipment, storage medium and product

Abstract

The invention discloses an unsafe behavior recognition method, device and equipment, a storage medium and a product, and relates to the field of intelligent monitoring and diagnosis technology and computer vision cross technology, and the unsafe behavior recognition method comprises the steps: obtaining a working image of a user collected by at least one shooting device; behavior recognition is carried out on the working image through a preset behavior analysis model, a behavior recognition image is obtained, and the preset behavior analysis model comprises an article recognition sub-model and a joint point prediction sub-model; and performing unsafe behavior judgment on the behavior recognition image to obtain an unsafe behavior recognition result. According to the application, the problem that whether the monitored object is using the mobile phone or not cannot be accurately detected by independently using the OPENPOSE model and the YOLO model can be avoided.

Description

Technical Field

[0001] The present application relates to the cross-technical field of intelligent monitoring and diagnosis technology and computer vision, and in particular to an unsafe behavior identification method, device, equipment, storage medium and product. Background Art

[0002] The safe operation of nuclear power plants is of vital importance. The status and behavior of operators directly affect the safety of nuclear power plants. When operators are not focused on their work, serious safety hazards may occur. Therefore, whether the operators of nuclear power plants are focused has a direct impact on whether the nuclear power plants can operate safely. In the current working environment, improper use of smart devices (such as mobile phones) by operators will greatly reduce their concentration on the current work, thus leading to serious safety hazards.

[0003] In the related technology, the OPENPOSE model is used to detect unsafe behaviors, and the other is to use the YOLO model to detect cigarette butts. For the method that only uses the OPENPOSE model, since the human posture can be very random and the human skeleton state is similar to grasping an object, in some cases, even a mobile phone can be used with only one finger, the human skeleton network extracted by the OPENPOSE model cannot reflect whether the object is using a mobile phone. The method that only uses the YOLO model can only detect the object of the mobile phone, but cannot determine whether the monitored object is using the mobile phone. Considering that mobile phones are very popular, it is unreasonable to judge that this is an unsafe behavior based solely on the appearance of the mobile phone object. Therefore, these two methods cannot accurately detect whether the monitored object is using a mobile phone when used alone. Summary of the invention

[0004] The main purpose of this application is to provide an unsafe behavior identification method, device, equipment, storage medium and product, aiming to solve the technical problem that neither the OPENPOSE model nor the YOLO model alone can accurately detect whether the monitored object is using a mobile phone.

[0005] To achieve the above objectives, the present application proposes an unsafe behavior identification method, which includes:

[0006] Acquire a working image of a user captured by at least one photographing device;

[0007] Performing behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image, wherein the preset behavior analysis model includes an object recognition sub-model and a joint point prediction sub-model;

[0008] An unsafe behavior judgment is performed on the behavior recognition image to obtain an unsafe behavior recognition result.

[0009] In one embodiment, the step of performing behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image includes:

[0010] The object to be identified in the working image is identified by the object identification sub-model to obtain an object frame image, wherein the object frame image is marked with a target category corresponding to the object to be identified;

[0011] Predicting the joint points in the working image and the vectors of the joint points through a joint point prediction sub-model to obtain a joint point confidence map and a vector field map of the joint points, and determining a connection vector map of the joint points based on the joint point confidence map and the vector field map, wherein the joint point prediction sub-model is based on a convolutional layer and a prediction layer.

[0012] The object frame image and the connection vector image are superimposed to obtain a behavior recognition image.

[0013] In one embodiment, the step of performing unsafe behavior judgment on the behavior analysis image to obtain an unsafe behavior recognition result includes:

[0014] Based on the connection vector graph, generating a joint point connection graph;

[0015] Calculating the node degree of each node in the joint point connection graph to obtain the node degree corresponding to each node;

[0016] Based on the node degree, determining the finger node of the joint point connection graph;

[0017] The edge connecting any two finger nodes in the joint point connection diagram is judged to intersect with the border of the object border diagram. If there is an intersection, the judgment result is that an unsafe behavior exists, and the judgment result is used as the unsafe behavior identification result.

[0018] In one embodiment, before the step of identifying the target object in the working image by the object recognition sub-model to obtain the object frame map, the step includes:

[0019] Obtain sample working images with sample labels and obtain an initial object recognition model;

[0020] Inputting the sample working image into the initial object recognition model, dividing the input sample working image into a preset number of grids, predicting a bounding box in each grid and a class probability corresponding to the bounding box, and obtaining a predicted initial bounding box of the sample working image;

[0021] Comparing the initial bounding box with the bounding box in the sample working image to obtain a target loss;

[0022] Based on the target loss, the initial object recognition model is iteratively trained to obtain an object recognition sub-model that meets the accuracy requirements.

[0023] In one embodiment, before the step of acquiring a sample working image with a sample label, the following steps are included:

[0024] Acquire a sample working image, and perform image adjustment on the sample working image to obtain the sample working image of a preset size after adjustment;

[0025] Annotating the adjusted sample working image with objects to obtain the annotated sample working image with object boundary boxes, and adding category labels to the annotated objects in the sample working image to obtain the sample working image with sample labels;

[0026] Perform image adjustment on the sample working image with the sample label to obtain the sample working image after data expansion.

[0027] In one embodiment, the step of predicting the human joints in the working image and the connection relationship between the human joints by the joint point prediction submodel to obtain a joint prediction graph includes:

[0028] Acquire a sample image, wherein the sample image is an image of a human body within a preset range;

[0029] Marking the joint points and vector fields of the sample image, and using the marked sample image as a sample label;

[0030] Obtain the initial joint point prediction model;

[0031] Inputting the sample image into the initial joint point prediction model, performing joint point prediction on the sample image, and obtaining a target joint point and an initial vector field map corresponding to the target joint point;

[0032] Based on the initial connection vector graph and the sample labels, the initial joint prediction model is iteratively trained to obtain a joint point prediction sub-model that meets the accuracy requirements.

[0033] In addition, to achieve the above purpose, the present application also proposes an unsafe behavior identification device, the unsafe behavior identification device comprising:

[0034] An acquisition module, used to acquire a working image of a user captured by at least one photographing device;

[0035] A recognition module, used to perform behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image;

[0036] The judgment module is used to judge the unsafe behavior of the behavior recognition image to obtain an unsafe behavior recognition result.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes an unsafe behavior identification device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the unsafe behavior identification method described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the unsafe behavior identification method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the unsafe behavior identification method described above are implemented.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] Compared with the related art, in which the OPENPOSE model or the YOLO model alone cannot accurately detect whether the monitored object is using a mobile phone, the present application obtains a working image of the user captured by at least one shooting device; performs behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image, wherein the preset behavior analysis model includes an object recognition sub-model and a joint point prediction sub-model; performs unsafe behavior judgment on the behavior recognition image to obtain an unsafe behavior recognition result. It can be understood that after obtaining the working image, the present application performs behavior recognition on the working image through the object recognition sub-model and the joint point prediction sub-model in the preset behavior analysis model, and performs segmentation on the recognized image to obtain an unsafe behavior recognition result, thereby avoiding the problem that the OPENPOSE model and the YOLO model alone cannot accurately detect whether the monitored object is using a mobile phone. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart of the first embodiment of the unsafe behavior identification method of the present application is provided;

[0045] Figure 2 This is an unsafe behavior identification flow chart of the unsafe behavior identification method of this application;

[0046] Figure 3 A flowchart of the second embodiment of the unsafe behavior identification method of the present application is provided;

[0047] Figure 4 This is a schematic diagram of the module structure of the unsafe behavior identification device according to an embodiment of the present application;

[0048] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the unsafe behavior identification method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: obtaining a working image of a user captured by at least one shooting device; performing behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image, wherein the preset behavior analysis model includes an object recognition sub-model and a joint point prediction sub-model; performing unsafe behavior judgment on the behavior recognition image to obtain an unsafe behavior recognition result.

[0053] In the related technology, the OPENPOSE model is used to detect unsafe behaviors, and the other is to use the YOLO model to detect cigarette butts. For the method that only uses the OPENPOSE model, since the human posture can be very random and the human skeleton state is similar to grasping an object, in some cases, even a mobile phone can be used with only one finger, the human skeleton network extracted by the OPENPOSE model cannot reflect whether the object is using a mobile phone. The method that only uses the YOLO model can only detect the object of the mobile phone, but cannot determine whether the monitored object is using the mobile phone. Considering that mobile phones are very popular, it is unreasonable to judge that this is an unsafe behavior based solely on the appearance of the mobile phone object. Therefore, these two methods cannot accurately detect whether the monitored object is using a mobile phone when used alone.

[0054] After acquiring the working image, the present application performs behavior recognition on the working image through the object recognition sub-model and the joint point prediction sub-model in the preset behavior analysis model, and segments the recognized image to obtain the unsafe behavior recognition result, thereby avoiding the problem that neither the OPENPOSE model nor the YOLO model can accurately detect whether the monitored object is using a mobile phone when used alone.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes the unsafe behavior recognition device as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a method for identifying unsafe behavior, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the unsafe behavior identification method of the present application.

[0057] In this embodiment, the unsafe behavior identification method includes steps S100 to S300:

[0058] Step S100, obtaining a working image of a user captured by at least one photographing device;

[0059] It should be noted that the executor of this embodiment is an unsafe behavior recognition device. The unsafe behavior recognition device is provided with an overhead camera, which monitors unsafe behaviors in real time. It is used to collect the user's working images. The collection device is an overhead camera. The unsafe behavior recognition device obtains the user's working images collected by at least one overhead camera. The multi-camera system can be used to capture the user's gestures, facial expressions and other body language to help perform more accurate behavior recognition, wherein the overhead camera is for a preset range of working images, wherein the preset range is the working image of the user's upper body. By limiting the working image to the preset range, more nodes can be allocated to the user's hands, and the relationship between the hands and the object to be identified can be accurately judged.

[0060] In addition, the unsafe behavior recognition device is also equipped with a front camera to detect the user's facial features, and then monitor the user's fatigue status in real time.

[0061] Step S200, performing behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image, wherein the preset behavior analysis model includes an object recognition sub-model and a joint point prediction sub-model;

[0062] It is understandable that the preset behavior analysis model is used to generate the user's behavior recognition image. The unsafe behavior recognition device performs behavior recognition on the working image through the preset behavior analysis model to obtain a behavior recognition image that annotates the connection vectors between key points in the working image and the boundary box of the object to be recognized.

[0063] In a feasible implementation manner, step S200 includes the following steps:

[0064] The object to be identified in the working image is identified by the object identification sub-model to obtain an object frame image, wherein the object frame image is marked with a target category corresponding to the object to be identified;

[0065] It should be noted that the object recognition sub-model is used to detect and classify objects in an image. The object to be recognized is a mobile phone. The object frame map is used to frame the mobile phone. The target category is a mobile phone. The unsafe behavior recognition device recognizes the mobile phone in the working image of the object recognition sub-model and obtains a working image for framing the mobile phone, wherein the object frame map marks the framed object as a mobile phone.

[0066] Specifically, before the preset behavior analysis model processes the input working image, it is necessary to preprocess the working image and adjust the image size. The input image is adjusted to the input size set during model training (such as 368x368), and the adjusted image is normalized to normalize the image pixel value to the range of [0, 1] to meet the input requirements of the neural network. The unsafe behavior recognition device inputs the preprocessed working image into the object recognition sub-model in the preset behavior analysis model. The model outputs multiple bounding boxes, each of which includes the probability of belonging to the mobile phone category and the confidence of the bounding box. When a model makes a prediction, it not only gives the predicted category, but also provides a confidence score, indicating the probability that the model believes the prediction is correct, and performs non-maximum suppression on the bounding box, which filters out overlapping boxes and retains only the boxes with the highest confidence.

[0067] Specifically, all detected bounding boxes are sorted from high to low according to confidence. The higher the confidence score, the greater the possibility that the detection box contains the correct target. Then, the first (i.e., the highest confidence) bounding box is selected from the sorted list and added to the final retained result list. For each remaining detection box, the intersection over union (IoU) between it and the selected bounding box is calculated. IoU is the ratio of the area of ​​the overlapping part of two rectangles to the area of ​​their union. The formula is as follows:

[0068]

[0069] If the IoU exceeds a preset threshold (e.g., 0.5 or higher), then the two bounding boxes are considered to overlap too much and can be considered to be repeated output bounding boxes for the same target. At this time, the bounding box with lower confidence should be excluded, so that the bounding box of the object to be identified (such as a mobile phone) can be obtained.

[0070] In addition, the object recognition sub-model can also be used to identify the helmet on the user's head to ensure the user's safety while working.

[0071] Predicting the joint points in the working image and the vectors of the joint points through a joint point prediction sub-model to obtain a joint point confidence map and a vector field map of the joint points, and determining a connection vector map of the joint points based on the joint point confidence map and the vector field map, wherein the joint point prediction sub-model is based on a convolutional layer and a prediction layer.

[0072] It is understandable that the joint prediction submodel is used to generate connection vectors between key points based on the output limb key points and PAFs (Part Affinity Fields). The connection vector graph includes connection vectors constructed from one key point to another key point. The unsafe behavior recognition device inputs the preprocessed working image into the joint prediction submodel in the preset behavior analysis model. The model is pre-set with a fixed number of node recognitions, and sets the user's hand to have more nodes to accurately judge the relationship between the hand and the object to be identified. The model outputs two main results: Limb key points: the position coordinates of each detected human key point (for example, head, shoulder, elbow, wrist, hip, knee, ankle, etc.), and the key point is used as the joint point of the human body and PAFs (Part Affinity Fields): a vector field describing the relationship between adjacent key points, indicating the direction and strength of the limb part. For each detected key point, the maximum response path in the PAF field is searched to determine the connection between the key point and other key points. According to the direction information of the PAF field, a connection vector is constructed from one key point to another key point, and then a connection vector graph of the node is generated.

[0073] The object frame image and the connection vector image are superimposed to obtain a behavior recognition image.

[0074] It should be noted that the behavior recognition image is an image that marks the bounding box, joint points, and connection vectors between the joint points. The unsafe behavior recognition device superimposes the connection vector graph output by the item recognition sub-model and the joint point prediction sub-model to obtain a behavior recognition image that marks the bounding box, joint points, and connection vectors between the joint points after superposition.

[0075] Step S300, performing unsafe behavior judgment on the behavior recognition image to obtain an unsafe behavior recognition result.

[0076] It is understandable that the unsafe behavior recognition result is the recognition result obtained by judging the user's current working status. The unsafe behavior recognition device judges the behavior recognition image output by the preset behavior analysis model to obtain the unsafe behavior recognition result obtained by judging the user's current working status. Please refer to Figure 2 , Figure 2 A flowchart for identifying unsafe behaviors is provided.

[0077] In a feasible implementation manner, step S300 includes the following steps:

[0078] Based on the connection vector graph, generating a joint point connection graph;

[0079] It should be noted that the joint point connection graph is a connection graph based on the connection vectors between joint points in the connection vector graph. The unsafe behavior recognition device traverses each joint point, determines the connection between joint points based on the connection vectors in the connection vector graph, the direction and strength of the PAF, regards each joint point as a node in the graph, and regards the connection between joint points as edges in the graph, and constructs a joint point connection graph. in, represents a node set, ε represents an edge set, where the joint point connection graph includes the connection edges from the head node to the finger node.

[0080] Calculating the node degree of each node in the joint point connection graph to obtain the node degree corresponding to each node;

[0081] It is understandable that the node degree refers to the number of edges connected to the node, and the in-degree (the number of edges entering the node) and the out-degree (the number of edges leaving the node) need to be calculated separately. The unsafe behavior identification device counts the degree (v) of each node, where:

[0082] Based on the node degree, determining the finger node of the joint point connection graph;

[0083] It should be noted that the finger node is a node with a node degree of 1. The unsafe behavior identification device can determine the node with a node degree of 1 as the finger node according to the node degree.

[0084] In particular, the node degree of the head node is also 1. By determining the head node and the finger node with the node degree of 1, and then determining the connection between the head node and the finger node, the user's hand movements can be identified more accurately.

[0085] The edge connecting any two finger nodes in the joint point connection diagram is judged to intersect with the border of the object border diagram. If there is an intersection, the judgment result is that an unsafe behavior exists, and the judgment result is used as the unsafe behavior identification result.

[0086] It is understandable that the unsafe behavior recognition result is that the user is using a mobile phone. For the bounding box of the category mobile phone, the unsafe behavior recognition device checks the edges connecting any two finger nodes in turn to see if they are connected to the edge ε f If they intersect, it can be determined that the target is using a mobile phone, and the above conclusion is used as the unsafe behavior recognition result.

[0087] In this embodiment, the safe behavior recognition device performs behavior recognition on the working image through a preset behavior analysis model, and obtains the object border map based on the object recognition sub-model and the joint point prediction sub-model to obtain the connection vector map of the joint points, judges the unsafe behavior of the user, and obtains the unsafe behavior recognition result. Through the joint action of the object recognition sub-model and the joint point prediction sub-model, the accuracy of detecting whether the monitored object is using a mobile phone is improved.

[0088] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 Before the object to be identified in the working image is identified by the object identification sub-model and the object frame image is obtained, the unsafe behavior identification method further includes steps A10 to A40:

[0089] Step A10, obtaining a sample working image with a sample label, and obtaining an initial object recognition model;

[0090] It should be noted that the sample working images with sample labels are images for training the initial object recognition model. The initial object recognition model is obtained based on the YOLOv8 model, where the training configuration of the model includes: learning rate, batch size, loss function, etc., as follows:

[0091] The learning rate is gradually reduced with the training time, which can be expressed as:

[0092] η t =η0·e -λt

[0093] Where: η t is the learning rate of the tth training, η0 is the initial learning rate, λ is a constant that controls the decay rate of the learning rate, and t is the current number of training times.

[0094] The batch size is determined by the computer configuration used for training and the actual training effect.

[0095] The loss function includes positioning loss, confidence loss and classification loss, and its mathematical expression is:

[0096]

[0097] Among them, λ coord is the weight of the localization loss. noobj is the weight of the undetected object part in the confidence loss, S is the size of the grid (i.e., the grid is divided into S×S cells), B is the number of bounding boxes predicted in each grid cell, Indicates whether grid cell i and bounding box j contain an object, Indicates whether the grid cell i and bounding box j do not contain the object, x i ,y i , w i ,h i are the center coordinates and width and height of the predicted bounding box, is the center coordinates and width and height of the actual bounding box, C i is the predicted bounding box confidence, is the true bounding box confidence, p i (c) is the predicted class probability, is the true class probability.

[0098] It is understandable that the unsafe behavior recognition device obtains images for training the initial object recognition model and obtains the initial object recognition model based on the YOLOv8 model.

[0099] Step A20, inputting the sample working image into the initial object recognition model, dividing the input sample working image into a preset number of grids, predicting a bounding box in each grid and a class probability corresponding to the bounding box, and obtaining a predicted initial bounding box of the sample working image;

[0100] It is understandable that the initial bounding box is the bounding box of the object to be identified that is predicted by the initial object recognition model. The unsafe behavior recognition device divides the sample working image into a training set and a validation set, inputs the sample working image into the initial object recognition model, and performs forward propagation through the model. The model will divide the training set image into an S×S grid, predicting that there are B bounding boxes and C categories in each grid. Each bounding box contains 5 parameters: center coordinates (x, y), width and height (w, h) and confidence, and these parameters are normalized. The calculation formula of confidence can be expressed as:

[0101]

[0102] Among them, P(Object) is the probability that an object exists in the grid cell, is the intersection-over-union ratio between the predicted bounding box and the true bounding box;

[0103] The probability that the object in each bounding box belongs to the i-th category is:

[0104] p(class i )=p(class i |Object)×Confidence

[0105] Among them, p(class i|Object) represents the probability that the object belongs to category i if there is an object in the grid cell;

[0106] After the calculation is completed, the model will output multiple bounding boxes, each of which contains the center coordinates (x, y), width and height (w, h), confidence, and probability distribution of the mobile phone category.

[0107] Step A30, comparing the initial bounding box with the bounding box in the sample working image to obtain a target loss;

[0108] It should be noted that the target loss is the difference between the initial bounding box predicted by the model and the bounding box in the sample working image. The unsafe behavior recognition device compares the bounding box output by the model with the true bounding box to obtain the difference between the initial bounding box predicted by the model and the bounding box in the sample working image.

[0109] Step A40: iteratively train the initial object recognition model based on the target loss to obtain an object recognition sub-model that meets the accuracy requirements.

[0110] It is understandable that the unsafe behavior identification device calculates the target loss based on the loss function, where the first two items are positioning losses, which calculate the mean square error of the bounding box coordinates, the second and third items are confidence losses, which calculate the mean square error of the confidence of the bounding box, and the last item is classification loss, which is the cross entropy loss function. Back propagation calculates the gradient of the model parameters based on the loss function, and then updates the model parameters based on the gradient. The gradient represents the rate of change of the loss function relative to the model parameters, reflecting the contribution of each parameter to the loss function. For each parameter, it is necessary to calculate the partial derivative of the loss function with respect to the parameter to determine how to adjust the parameter to minimize the loss function. The calculation formula for the gradient of a parameter θ of the model is:

[0111]

[0112] in, is the loss function;

[0113] After completing the gradient calculation, update the model parameters according to the calculated step size to reduce the loss function. If you want to update a parameter θ at time t, use the following formula:

[0114]

[0115] Among them, θ t represents the parameter at the tth iteration, η is the learning rate, is the parameter θ at the tth iteration t The gradient of

[0116] The unsafe behavior recognition device repeats the above process of forward propagation, loss calculation and back propagation in each training iteration. Through multiple iterations, the loss function value of the model is gradually reduced, thereby optimizing the model parameters.

[0117] In particular, after each training cycle (epoch), the unsafe behavior recognition device uses the validation set to evaluate the performance of the model. There are two major indicators to evaluate the performance, namely precision P and recall rate R. The specific calculation formula is:

[0118]

[0119] Among them, TP represents true positive examples, FP represents false positive examples, and FN represents false negative examples;

[0120] After the training is completed, the unsafe behavior recognition device saves the final model weights and hyperparameter configurations so that the trained model that meets the accuracy requirements can be used for prediction in actual applications.

[0121] In a feasible implementation manner, the step of obtaining a sample working image with a sample label includes the following steps:

[0122] Acquire a sample working image, and perform image adjustment on the sample working image to obtain the sample working image of a preset size after adjustment;

[0123] It should be noted that the sample work images contain image data of nuclear power operators, ensuring that the images contain the operator's facial information and the mobile phone that may be used, as well as other objects that may affect normal work. The preset size is 368×368. The unsafe behavior recognition device collects sample work images containing image data of nuclear power operators, ensuring that the images contain the operator's facial information and the mobile phone that may be used, as well as other objects that may affect normal work, and annotates the collected images and adjusts the images to a fixed size, the common size is 368×368.

[0124] Annotating the adjusted sample working image with objects to obtain the annotated sample working image with object boundary boxes, and adding category labels to the annotated objects in the sample working image to obtain the sample working image with sample labels;

[0125] It is understandable that the unsafe behavior identification device uses a labeling tool to mark the adjusted push, marks the boundary box of the mobile phone, and adds a corresponding category label to each mark, thereby obtaining an unsafe behavior identification device.

[0126] In particular, the unsafe behavior recognition device can mark the bounding boxes of the helmet and the mobile phone at the same time to achieve more specific unsafe behavior recognition.

[0127] Perform image adjustment on the sample working image with the sample label to obtain the sample working image after data expansion.

[0128] It should be noted that image adjustment includes random cropping, flipping, rotation and color adjustment, etc., to increase data diversity and improve the robustness of the model. The unsafe behavior recognition device performs data enhancement operations on the labeled images, such as random cropping, flipping, rotation and color adjustment, etc., and then obtains the sample working image after data enhancement.

[0129] In a feasible implementation manner, the human body joints in the working image and the connection relationship between the human body joints are predicted by the joint point prediction submodel, and the step of obtaining the joint prediction graph includes the following steps:

[0130] Acquire a sample image, wherein the sample image is an image of a human body within a preset range;

[0131] It should be noted that the sample image is an image of the upper body of the user. The unsafe behavior recognition device obtains the upper body image of the user, wherein the sample image is used to train the joint point prediction sub-model.

[0132] Marking the joint points and vector fields of the sample image, and using the marked sample image as a sample label;

[0133] It is understandable that the unsafe behavior recognition device labels the sample image and generates a connection vector graph as a sample label.

[0134] Obtain the initial joint point prediction model;

[0135] It should be noted that the initial joint prediction model includes a basic convolution layer and a PAFs (Part Affinity Fields) prediction layer. The unsafe behavior recognition device obtains an initial joint prediction model including a basic convolution layer and PAFs. The design of the initial joint prediction model includes two sub-steps: the design of the basic convolution layer and the design of the PAFs (Part Affinity Fields) prediction layer. The specific steps are as follows:

[0136] Basic convolutional layer design:

[0137] The basic convolution layer contains multiple convolution layers and pooling layers. The main function of the convolution layer is to extract the features of the input image and output the feature map of the image. Through the convolution operation, local spatial features such as edges and corners can be extracted. Each convolution layer scans the input image through several convolution kernels (also called filters) to generate a feature map. The convolution formula is:

[0138]

[0139] Among them, Y(i, j, k) is the value of the output feature map at position (i, j) and channel k. X(i, j, c) is the value of the input image at position (i, j) and channel c; W(m, n, c, k) is the weight of the convolution kernel at position (m, n) and channel c to output channel k; b(k) is the bias of output channel k; K is the size of the convolution kernel.

[0140] The activation function uses the ReLU function, which is mathematically expressed as follows:

[0141] ReLU(x)=max(0,x)

[0142] The pooling layer is usually used for downsampling to reduce the spatial size of the feature map, thereby reducing the amount of calculation and the number of parameters, while enhancing the translation invariance of the model. Commonly used pooling operations include Max Pooling and Average Pooling. The formula for the pooling operation is:

[0143]

[0144] Among them, Y(i, j, k) is the value of the output feature map at position (i, j) and channel k, X(i, j, k) is the value of the input feature map at position (i, j) and channel k, P is the size of the pooling window, and S is the pooling stride.

[0145] Reasonable selection of the number and structure of convolutional layers and pooling layers can effectively extract the underlying features of the input image. To facilitate subsequent processing, the following is an example of the convolutional layer architecture:

[0146] The first layer uses a 3×3 convolution kernel, 64 channels, and a 2×2 maximum pooling;

[0147] The second layer uses a 3×3 convolution kernel, 128 channels, and a 2×2 maximum pooling.

[0148] The third layer uses a 3×3 convolution kernel, 256 channels, and a 2×2 maximum pooling.

[0149] After the unsafe behavior recognition device builds the basic convolution layer, it needs to design the PAF prediction layer to predict the human joints and the connection relationship between these joints. The PAF prediction layer will gradually generate joint confidence maps and PAFs in multiple stages. The following operations will be performed in each stage:

[0150] Operation 1: Get the feature map output by the base convolution layer or the output layer of the previous stage;

[0151] Operation 2: Perform convolution operation on the feature map;

[0152] Operation 3: The output layer consists of two parts, which are used to generate joint point confidence maps and PAFs respectively;

[0153] Among them, each pixel value of the joint point confidence map represents the probability that the position is a certain joint point, which is generated by a convolutional layer. The convolutional layer can be expressed as:

[0154]

[0155] Among them, assuming that the input feature map is X features , the output key point confidence map is

[0156] PAFs is a two-dimensional vector field. Each vector represents the connection relationship between a pair of human joints in the image. It can capture the spatial and topological relationship between human body parts, so that the model can perform posture estimation more accurately. For each pair of joints, PAFs generates a vector field. The direction of the vector points from one joint point to another. The magnitude of the vector represents the connectivity between the two joints. The larger the magnitude, the greater the possibility of connectivity. Its mathematical expression is:

[0157]

[0158] Among them, v ij is the unit vector pointing from key point i to key point j;

[0159] The vector field will be generated by a convolutional layer, which can be expressed as:

[0160]

[0161] The number of convolution layers, convolution kernels, and channels in each stage of the PAF prediction layer needs to be determined by the actual training situation. Here is an example architecture:

[0162] Stage 1: This is where the output of the base convolutional layer (256 channels) is accepted.

[0163] Convolutional layer: 4 layers of 3x3 convolution kernels, 128 channels, ReLU activation;

[0164] Output layer: 1 layer of 1x1 convolution kernel, 38 channels (output PAFs);

[0165] Then the PAFs are merged with the output feature map (256 channels) of the base convolutional layer as the input of stage 2, which is mathematically expressed as:

[0166]

[0167] in, is the input of the t+1th stage, and They are the key point confidence map and PAFs prediction results of the tth stage respectively;

[0168] Phase 2 to Phase 5:

[0169] Convolutional layer: 4 layers of 3x3 convolution kernels, 128 channels, ReLU activation;

[0170] Output layer: 1 layer of 1x1 convolution kernel, 38 channels (output PAFs);

[0171] Then the output of the current stage is merged with the output feature map (256 channels) of the base convolutional layer as the input of the next stage;

[0172] Phase 6:

[0173] Convolutional layer: 3 layers of 3x3 convolution kernels, 128 channels, ReLU activation;

[0174] Output layer: 1 layer of 1x1 convolution kernel, 38 channels (output PAFs).

[0175] Inputting the sample image into the initial joint point prediction model, performing joint point prediction on the sample image, and obtaining a target joint point and an initial vector field map corresponding to the target joint point;

[0176] It can be understood that the unsafe behavior recognition device inputs the sample image into the initial joint prediction model, performs forward propagation through the basic convolutional layer, extracts the underlying features of the image, further processes the feature map at each stage, and outputs the key point confidence map and PAFs.

[0177] Based on the initial connection vector graph and the sample labels, the initial joint prediction model is iteratively trained to obtain a joint point prediction sub-model that meets the accuracy requirements.

[0178] It should be noted that the unsafe behavior recognition device calculates the difference between the model prediction result and the actual annotation by using the loss function. The loss function includes the key point confidence map loss and PAFs loss, and its mathematical expression is:

[0179]

[0180] Among them, the first term on the right side of the equation is the mean square error between the predicted joint point confidence map and the true confidence map, and the second term is the mean square error between the predicted PAFs and the true PAFs;

[0181] And calculate the gradient of the model parameters based on the calculated loss function, and use the optimizer to update the parameters, and calculate the gradient of the model parameters based on the loss function; the gradient represents the rate of change of the loss function relative to the model parameters, reflecting the contribution of each parameter to the loss function, and uses the optimization algorithm (such as stochastic gradient descent method SGD and Adam, etc.) to update the model parameters according to the gradient; in each training iteration, repeat the above forward propagation and loss calculation and back propagation process. After multiple iterations, the loss function value of the model is gradually reduced, thereby optimizing the model parameters; through the above steps, the model will be able to gradually improve the prediction accuracy of the joint confidence map and PAFs in the continuous training and optimization process, and then obtain the joint prediction sub-model that meets the accuracy requirements. Finally, efficient posture estimation is achieved.

[0182] In this embodiment, the unsafe behavior recognition device obtains the object recognition sub-model and the joint point prediction sub-model through iterative training, and finally realizes efficient estimation of unsafe behaviors.

[0183] This application also provides an unsafe behavior identification device, please refer to Figure 4 , the unsafe behavior identification device comprises:

[0184] An acquisition module 10 is used to acquire a working image of a user captured by at least one photographing device;

[0185] The recognition module 20 is used to perform behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image;

[0186] The judgment module 30 is used to judge the unsafe behavior on the behavior recognition image to obtain an unsafe behavior recognition result.

[0187] Optionally, the identification module includes:

[0188] The superposition module is used to identify the objects to be identified in the working image through the object recognition sub-model to obtain an object frame map, wherein the object frame map is annotated with the target category corresponding to the object to be identified; predict the joint points in the working image and the vectors of the joint points through the joint point prediction sub-model to obtain a joint point confidence map and a vector field map of the joint points, and determine the connection vector map of the joint points based on the joint point confidence map and the vector field map, wherein the joint point prediction sub-model is based on a convolutional layer and a prediction layer; and superimpose the object frame map and the connection vector map to obtain a behavior recognition image.

[0189] Optionally, the superposition module includes:

[0190] The recognition training module is used to obtain a sample working image with a sample label and an initial object recognition model; input the sample working image into the initial object recognition model, divide the input sample working image into a preset number of grids, predict the bounding box in each grid and the category probability corresponding to the bounding box, and obtain the predicted initial bounding box of the sample working image; compare the initial bounding box with the bounding box in the sample working image to obtain a target loss; based on the target loss, iteratively train the initial object recognition model to obtain an object recognition sub-model that meets the accuracy requirements.

[0191] A prediction training module is used to obtain a sample image, wherein the sample image is an image of a human body within a preset range; mark the joint points and vector fields of the sample image, and use the marked sample image as a sample label; obtain an initial joint point prediction model; input the sample image into the initial joint point prediction model, perform joint point prediction on the sample image, and obtain a target joint point and an initial vector field map corresponding to the target joint point; based on the initial connection vector map and the sample label, iteratively train the initial joint prediction model to obtain a joint point prediction sub-model that meets the accuracy requirements.

[0192] Optionally, the recognition training module includes:

[0193] The sample adjustment module is used to obtain a sample working image, and perform image adjustment on the sample working image to obtain the sample working image of a preset size after adjustment; perform object labeling on the adjusted sample working image to obtain the sample working image with annotated object boundary boxes, and add category labels to the annotated objects in the sample working image to obtain the sample working image with sample labels; perform image adjustment on the sample working image with sample labels to obtain the sample working image after data expansion.

[0194] Optionally, the judging module includes:

[0195] The intersection judgment module is used to generate a joint point connection diagram based on the connection vector diagram; calculate the node degree of each node in the joint point connection diagram to obtain the node degree corresponding to each node; determine the finger nodes of the joint point connection diagram based on the node degree; and perform intersection judgment on the edge connecting any two finger nodes in the joint point connection diagram and the border of the object border diagram. If there is an intersection, the judgment result is that unsafe behavior exists, and the judgment result is used as the unsafe behavior identification result.

[0196] The unsafe behavior identification device provided by the present application adopts the unsafe behavior identification method in the above embodiment, which can solve the technical problem of unsafe behavior identification. Compared with the prior art, the beneficial effects of the unsafe behavior identification device provided by the present application are the same as the beneficial effects of the unsafe behavior identification method provided by the above embodiment, and the other technical features in the unsafe behavior identification device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0197] The present application provides an unsafe behavior identification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the unsafe behavior identification method in the above-mentioned embodiment one.

[0198] Reference below Figure 5 , which shows a schematic diagram of the structure of an unsafe behavior identification device suitable for implementing the embodiment of the present application. The unsafe behavior identification device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, tablet computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The unsafe behavior identification device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0199] like Figure 5As shown, the unsafe behavior recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the unsafe behavior recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the unsafe behavior identification device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows an unsafe behavior identification device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0200] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0201] The unsafe behavior identification device provided by the present application adopts the unsafe behavior identification method in the above embodiment, which can solve the technical problem of unsafe behavior identification. Compared with the prior art, the beneficial effects of the unsafe behavior identification device provided by the present application are the same as the beneficial effects of the unsafe behavior identification method provided by the above embodiment, and the other technical features in the unsafe behavior identification device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0202] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0203] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0204] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the unsafe behavior identification method in the above-mentioned embodiment.

[0205] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0206] The computer-readable storage medium may be included in the unsafe behavior identification device; or may exist independently without being assembled into the unsafe behavior identification device.

[0207] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the unsafe behavior recognition device, the unsafe behavior recognition device: obtains a working image of the user captured by at least one shooting device; performs behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image, wherein the preset behavior analysis model includes an object recognition sub-model and a joint point prediction sub-model; performs unsafe behavior judgment on the behavior recognition image to obtain an unsafe behavior recognition result.

[0208] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0209] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0210] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0211] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned unsafe behavior identification method, and can solve the technical problem of unsafe behavior identification. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the unsafe behavior identification method provided in the above-mentioned embodiment, and will not be repeated here.

[0212] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned unsafe behavior identification method when executed by a processor.

[0213] The computer program product provided by this application can solve the technical problem of unsafe behavior identification. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the unsafe behavior identification method provided by the above embodiment, which will not be repeated here.

[0214] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for identifying unsafe behavior, characterized in that: The unsafe behavior identification method comprises: Acquire a working image of a user captured by at least one photographing device; Performing behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image, wherein the preset behavior analysis model includes an object recognition sub-model and a joint point prediction sub-model; An unsafe behavior judgment is performed on the behavior recognition image to obtain an unsafe behavior recognition result.

2. The unsafe behavior identification method according to claim 1, characterized in that: The step of performing behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image comprises: The object to be identified in the working image is identified by the object identification sub-model to obtain an object frame image, wherein the object frame image is marked with a target category corresponding to the object to be identified; Predicting the joint points in the working image and the vectors of the joint points through a joint point prediction sub-model to obtain a joint point confidence map and a vector field map of the joint points, and determining a connection vector map of the joint points based on the joint point confidence map and the vector field map, wherein the joint point prediction sub-model is based on a convolutional layer and a prediction layer. The object frame image and the connection vector image are superimposed to obtain a behavior recognition image.

3. The unsafe behavior identification method according to claim 2, characterized in that: The step of performing unsafe behavior judgment on the behavior recognition image to obtain an unsafe behavior recognition result comprises: Based on the connection vector graph, generating a joint point connection graph; Calculating the node degree of each node in the joint point connection graph to obtain the node degree corresponding to each node; Based on the node degree, determining the finger node of the joint point connection graph; The edge connecting any two finger nodes in the joint point connection diagram is judged to intersect with the border of the object border diagram. If there is an intersection, the judgment result is that an unsafe behavior exists, and the judgment result is used as the unsafe behavior identification result.

4. The unsafe behavior identification method according to claim 2, characterized in that: The step of identifying the target object in the working image by the object recognition sub-model to obtain the object frame map includes: Obtain sample working images with sample labels and obtain an initial object recognition model; Inputting the sample working image into the initial object recognition model, dividing the input sample working image into a preset number of grids, predicting a bounding box in each grid and a class probability corresponding to the bounding box, and obtaining a predicted initial bounding box of the sample working image; Comparing the initial bounding box with the bounding box in the sample working image to obtain a target loss; Based on the target loss, the initial object recognition model is iteratively trained to obtain an object recognition sub-model that meets the accuracy requirements.

5. The unsafe behavior identification method according to claim 4, characterized in that: The step of obtaining a sample working image with a sample label includes: Acquire a sample working image, and perform image adjustment on the sample working image to obtain the sample working image of a preset size after adjustment; Annotating the adjusted sample working image with objects to obtain the annotated sample working image with object boundary boxes, and adding category labels to the annotated objects in the sample working image to obtain the sample working image with sample labels; Perform image adjustment on the sample working image with the sample label to obtain the sample working image after data expansion.

6. The unsafe behavior identification method according to claim 2, characterized in that: The step of predicting the human joints in the working image and the connection relationship between the human joints by the joint point prediction submodel to obtain a joint prediction graph includes: Acquire a sample image, wherein the sample image is an image of a human body within a preset range; Marking the joint points and vector fields of the sample image, and using the marked sample image as a sample label; Obtain the initial joint point prediction model; Inputting the sample image into the initial joint point prediction model, performing joint point prediction on the sample image, and obtaining a target joint point and an initial vector field map corresponding to the target joint point; Based on the initial connection vector graph and the sample labels, the initial joint prediction model is iteratively trained to obtain a joint point prediction sub-model that meets the accuracy requirements.

7. An unsafe behavior identification device, characterized in that: The device comprises: An acquisition module, used to acquire a working image of a user captured by at least one photographing device; A recognition module, used to perform behavior recognition on the working image through a preset behavior analysis model to obtain a behavior recognition image; The judgment module is used to judge the unsafe behavior of the behavior recognition image to obtain an unsafe behavior recognition result.

8. An unsafe behavior recognition device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the unsafe behavior identification method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the unsafe behavior identification method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the unsafe behavior identification method according to any one of claims 1 to 6 are implemented.

Images