Video analysis-based illegal behavior detection method, device and equipment

Abstract

The invention discloses an illegal behavior detection method, device and equipment based on video analysis, and belongs to the technical field of video analysis. The method comprises the following steps: respectively extracting a main feature matrix corresponding to a central key frame and reference feature matrixes corresponding to front and back key frames; calculating a first attention map based on the main feature matrix and the reference feature matrix, aggregating the main feature matrix to obtain a key frame feature matrix, and extracting a personnel feature matrix and an association feature matrix from the central key frame, and calculating a second attention map based on the personnel characteristic matrix and the association characteristic matrix to obtain a scene characteristic matrix so as to perform action scene classification, thereby matching a two-tuple set of the personal protection equipment associated with each personnel with a rule set of the personal protection equipment corresponding to an action scene classification result of each personnel. And identifying whether each person has an illegal behavior or not. According to the invention, the accuracy of personnel action scene classification and illegal behavior detection can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video image analysis of artificial intelligence, in particular to a method and device for detecting illegal behavior based on video analysis. BACKGROUND

[0002] With the development of deep learning-based computer vision technology, the current industrial safety monitoring in the fields of energy, chemical industry, and power industry relies on worker illegal behavior detection based on video analysis, such as personal protective equipment (safety helmet, protective gloves, welding mask, etc.) detection based on video intelligent analysis to identify whether workers have safety illegal behavior in different work scenarios. In the prior art, a pre-trained deep learning-based target detection model (such as a YOLO model) is usually used to directly identify whether workers in the factory area wear personal protective equipment such as safety helmets, protective gloves, welding masks, and whether there are other illegal behaviors that violate safety regulations. However, this traditional detection scheme has significant limitations, as it cannot distinguish between the special wearing requirements of personal protective equipment in different work scenarios, which can lead to a large number of false positives in the system. For example, the system may incorrectly determine that a worker not wearing a welding mask or protective gloves while walking is an illegal behavior, which seriously affects the credibility of the system detection.

[0003] To reduce system false positives, existing technologies can also use a deep learning model for action behavior detection, such as the SlowFast model, to identify the action scenario of the worker's current work. For example, the existing patent document CN119091360B relates to a posture recognition management system and method for factory workers, which preprocesses video data in each key area based on the YOLOv8 algorithm, identifies the relevant information of the target workers and the trajectory information of the automated equipment in the target factory, and outputs the relevant information and trajectory information. At the same time, the SlowFast model is used to process the video data in each key area simultaneously, analyze the actions and postures of the workers, and trigger the recognition of the interactive behavior of the workers under the condition of interaction between the workers and the automated equipment, to comprehensively determine whether the workers have illegal operations.

[0004] Then, this scheme simultaneously uses the YOLOv8 model and the SlowFast model, which requires a large amount of computation and high hardware computing resources, making it difficult to ensure real-time performance and unsuitable for deployment on edge computing devices in a factory environment. Moreover, in the actual complex factory environment with multi-target visual monitoring using a wide-angle camera, due to image distortion, non-uniform resolution, and complex background of the video captured by the wide-angle camera, the existing scheme has low recognition accuracy in identifying the action scenario of the workers and detecting small-sized personal protective equipment. Therefore, there is an urgent need for an improved technical solution to solve the above technical problems. SUMMARY

[0005] Therefore, the application provides a rule violation detection method, device and equipment based on video analysis to improve the accuracy of identifying the action scene and target object detection of workers in a factory environment, and thus effectively identify the rule violation behavior of each worker in a specific action scene.

[0006] In a first aspect, the application provides a rule violation detection method based on video analysis, comprising: extracting a 3D feature map containing continuous multiple frames from a real-time collected video stream through a video backbone network, and cutting a center key frame and front and rear key frames located at a preset step before and after the center key frame from the 3D feature map; extracting a main feature matrix corresponding to the center key frame and a reference feature matrix corresponding to the front and rear key frames respectively, calculating a first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map to obtain a key frame feature matrix; identifying the detection bounding box of personnel, key equipment and personal protective equipment from the key frame feature matrix through a target detection classifier to obtain a binary tuple set of personal protective equipment associated with each personnel; extracting a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment from the center key frame according to the detection bounding box of the personnel and the key equipment, calculating a second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix; classifying the scene feature matrix through an action scene classifier to output an action scene classification result of each personnel; matching the binary tuple set of personal protective equipment associated with each personnel with a rule set of personal protective equipment corresponding to the action scene classification result of each personnel to identify whether each personnel has a rule violation behavior.

[0007] In an optional implementation, the extracting a main feature matrix corresponding to the center key frame and a reference feature matrix corresponding to the front and rear key frames respectively, calculating a first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map to obtain a key frame feature matrix comprises: splicing and fusing a front feature matrix extracted from a front key frame in the front and rear key frames and a rear feature matrix extracted from a rear key frame in the front and rear key frames to obtain the reference feature matrix; calculating a first linear transformation of the main feature matrix through a transformer model to obtain a query matrix, calculating a second linear transformation of the reference feature matrix to obtain a key matrix, and calculating the first attention map based on a point product of the query matrix and the key matrix through a softmax function operation; calculating a third linear transformation of a point product of the first attention map and the reference feature matrix, and adding the third linear transformation to the main feature matrix to obtain the key frame feature matrix.

[0008] In an optional implementation, the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment are extracted from the center key frame according to the detection bounding boxes of the personnel and the key equipment, including: determining an association bounding box between each personnel and each key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, the association bounding box containing each personnel and each key equipment at least; extracting a one-dimensional feature vector corresponding to the association bounding box between each personnel and each key equipment from the center key frame to obtain the association feature matrix between the personnel and the key equipment.

[0009] In an optional implementation, the association bounding box between each personnel and each key equipment is determined from the center key frame according to the detection bounding boxes of the personnel and the key equipment, the association bounding box containing each personnel and each key equipment at least, including: determining the association bounding box between each personnel and each key equipment according to minimum values of top-left corner coordinates and maximum values of bottom-right corner coordinates of the detection bounding boxes of each personnel and each key equipment.

[0010] In an optional implementation, the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment are extracted from the center key frame according to the detection bounding boxes of the personnel and the key equipment, including: calculating distances between centers of the detection bounding boxes of each personnel and each key equipment, and determining the association bounding box between each personnel and each key equipment when a distance between the centers of the detection bounding boxes of each personnel and each key equipment is less than or equal to a preset distance threshold from the center key frame.

[0011] In an optional implementation, the second attention map is calculated based on the personnel feature matrix and the association feature matrix, and the scene feature matrix is obtained by aggregating the personnel feature matrix based on the second attention map, including: performing a dimension alignment operation on the personnel feature matrix, and copying and filling elements in each row in a row dimension to obtain a personnel feature transformation matrix with the same dimension as the association feature matrix; performing a softmax function operation on a Hadamard product of the personnel feature transformation matrix and the association feature matrix to obtain the second attention map; performing a fourth linear transformation on a dot product between the second attention map and the association feature matrix, and adding the personnel feature matrix to obtain the scene feature matrix.

[0012] In an optional implementation, the method further includes: performing, for the front and rear key frames, the steps of extracting the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment, calculating a second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix; and performing, by an action scene classifier, action scene classification on the scene feature matrix to output an action scene classification result of each personnel. performing multiplication fusion calculation on a confidence of the action scene classification result of each personnel output by the center key frame and the front and rear key frames to obtain a final action scene classification and fusion confidence of each personnel.

[0013] In an optional implementation, the matching of the binary tuple set of the personal protective equipment associated with each personnel with the rule set of the personal protective equipment corresponding to the action scene classification result of each personnel to identify whether each personnel has a rule violation includes: According to the index order of each category of personal protective equipment, mapping the identification of whether each category of personal protective equipment contained in the rule set must be worn to a binary rule bit mask, mapping the identification of whether each category of personal protective equipment contained in the binary tuple set is detected to a binary wearing state bit mask of the same bit number, and performing an inversion operation on the wearing state bit mask to obtain an inverse bit mask corresponding to the wearing state bit mask; performing a logical AND operation on the rule bit mask and the inverse bit mask to identify the binary bits that are 1 in the operation result, thereby identifying the category of personal protective equipment involved in the rule violation of each personnel.

[0014] In a second aspect, the present application further provides a rule violation detection device based on video analysis, the device comprising: A key frame acquisition unit is configured to extract a 3D feature map containing continuous multiple frames from a video stream collected in real time by a video backbone network, and to cut a center key frame and front and rear key frames located at a preset step before and after the center key frame from the 3D feature map. The key frame feature extraction unit is configured to extract a main feature matrix corresponding to the center key frame and reference feature matrices corresponding to the front and rear key frames respectively, calculate a first attention map based on the main feature matrix and the reference feature matrices, and aggregate the main feature matrix based on the first attention map to obtain a key frame feature matrix. The target detection unit is configured to identify detection bounding boxes of personnel, key equipment and personal protective equipment from the key frame feature matrix by using a target detection classifier to obtain a set of two-tuples of personal protective equipment associated with each person; The scene feature extraction unit is configured to extract a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, calculate a second attention map based on the personnel feature matrix and the association feature matrix, and aggregate the personnel feature matrix based on the second attention map to obtain a scene feature matrix. The scene classification unit is configured to perform action scene classification on the scene feature matrix by using an action scene classifier to output an action scene classification result of each person. The violation behavior identification unit is configured to match the set of two-tuples of personal protective equipment associated with each person with a set of rules of personal protective equipment corresponding to the action scene classification result of each person to identify whether each person has a violation behavior.

[0015] In a third aspect, the present application further provides an edge computing device, comprising: one or more processors; a memory configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors are caused to perform the steps of the method according to any one of the preceding embodiments.

[0016] The present application can achieve the following beneficial effects: In one aspect of the present application, the center key frame and the front and rear key frames located a preset step before and after the center key frame are intercepted from the 3D feature map output from the video backbone network, the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames are extracted respectively, the first attention map is calculated based on the main feature matrix and the reference feature matrix, and the main feature matrix is enhanced based on the first attention map to obtain the enhanced key frame feature matrix. The detection bounding box of personnel, key equipment and personal protective equipment and the binary set of personal protective equipment associated with each personnel are identified from the enhanced key frame feature matrix through the target detection classifier. Thus, the detection error of the target object in the image frame, especially the small-size target object such as the personal protective equipment worn by the personnel, can be effectively improved, and the accuracy of target detection can be improved, in view of the image distortion, non-uniform resolution and complex background of the original video captured by the wide-angle camera, and the like, which leads to the occlusion and blur between objects in the original frame.

[0017] In another aspect, the personnel feature matrix corresponding to the personnel and the associated feature matrix between the personnel and the key equipment are extracted from the center key frame according to the detection bounding box of the personnel and the key equipment, the second attention map is calculated based on the personnel feature matrix and the associated feature matrix, and the personnel feature matrix is aggregated based on the second attention map to obtain the scene feature matrix. The action scene classification result of each personnel is output by performing action scene classification on the scene feature matrix through the action scene classifier. Thus, the personnel can be quickly and accurately identified in a specific action scene in the factory environment, and the violation behavior of each personnel in the specific action scene can be effectively identified, in view of the image distortion, non-uniform resolution and complex background of the original video captured by the wide-angle camera, and the like, which leads to the occlusion and blur between objects in the original frame.

[0018] In addition, the lightweight model used in the present application can effectively reduce the consumption of hardware computing power and computing resources, and can be deployed in the edge computing device in the factory environment to realize real-time visual safety behavior monitoring in the complex environment of the factory. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and should not be regarded as a limitation on the scope of the present application.

[0020] Figure 1 is a system architecture schematic diagram suitable for the violation behavior detection method based on video analysis of the embodiments of the present application; Figure 2 is a flow schematic diagram of the violation behavior detection method based on video analysis according to the first embodiment of the present application; Figure 3 is a part flowchart of a method for detecting a violation behavior based on video analysis according to a second embodiment of the present application; Figure 4 is a part flowchart of a method for detecting a violation behavior based on video analysis according to a third embodiment of the present application; Figure 5 is a part flowchart of a method for detecting a violation behavior based on video analysis according to a fourth embodiment of the present application; Figure 6 is a part flowchart of a method for detecting a violation behavior based on video analysis according to a fifth embodiment of the present application; Figure 7 is a part flowchart of a method for detecting a violation behavior based on video analysis according to a sixth embodiment of the present application; Figure 8 is a structure diagram of a device for detecting a violation behavior based on video analysis according to a first embodiment of the present application; Figure 9 is a part structure diagram of a device for detecting a violation behavior based on video analysis according to a second embodiment of the present application; Figure 10 is a part structure diagram of a device for detecting a violation behavior based on video analysis according to a third embodiment of the present application; Figure 11 is a part structure diagram of a device for detecting a violation behavior based on video analysis according to a fourth embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. However, it should be understood that the described embodiments are only some exemplary embodiments of the present application, but not all embodiments, and therefore the following detailed description of the embodiments of the present application is not intended to limit the scope of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0022] It should be noted that the terms “first”, “second” and the like in the specification and claims of the present application are only used to distinguish similar objects, and are not intended to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance.

[0023] As described above, the prior art solution usually has high requirements for hardware computing resources, and it is difficult to guarantee real-time performance, which is not suitable for deployment on edge computing devices in factory environments. Moreover, in the actual complex factory environment of visual monitoring of multiple targets by using a wide-angle camera, due to the image distortion, non-uniform resolution and complex background of the video collected by the wide-angle camera, the recognition accuracy of the prior art solution for recognizing the action scene of workers and detecting small-sized personal protective equipment is low. Therefore, the present application proposes a method, device and equipment for detecting illegal behavior based on video analysis to improve the accuracy of recognizing the action scene of workers and target object detection in a factory environment, and thus effectively identify the illegal behavior of each worker in a specific action scene.

[0024] Figure 1 is a system architecture diagram applicable to the method for detecting illegal behavior based on video analysis according to an embodiment of the present application. The system collects video streams in real time in a factory environment through a video stream acquisition module 10, extracts frames through a video backbone network 20, and obtains a continuous multi-frame 3D feature map. Then, on the one hand, a feature extraction module 30 extracts a group of key frames from the 3D feature map output by the video backbone network 20, the group of key frames including a center key frame and a front and rear key frame located at a preset step length, and outputs a main feature matrix and a reference feature matrix of the group of key frames. The main feature matrix and the reference feature matrix of the group of key frames are input into a multi-frame feature aggregation module 40 for time sequence feature aggregation. The multi-frame feature aggregation module 40 calculates a first attention map based on the main feature matrix and the reference feature matrix of the group of key frames, and aggregates the main feature matrix based on the first attention map to obtain an enhanced key frame feature matrix. Subsequently, a target detection classifier 50 identifies the detection bounding box and category of personnel, key equipment and personal protective equipment worn by personnel based on the key frame feature matrix, and simultaneously outputs a binary tuple set of personal protective equipment associated with each personnel.

[0025] On the other hand, the detection bounding box of personnel and key equipment output by the target detection classifier 50 is fed back to the feature extraction module 30, which extracts a personnel feature matrix corresponding to personnel and an association feature matrix between personnel and key equipment from the group of key frames according to the detection bounding box of personnel and key equipment, and inputs the association feature matrix into an association feature aggregation module 60. The association feature aggregation module 60 calculates a second attention map based on the personnel feature matrix and the association feature matrix, and aggregates the personnel feature matrix based on the second attention map to obtain an enhanced scene feature matrix. Subsequently, an action scene classifier 70 performs action scene classification prediction on the aggregated scene feature matrix, and outputs an action scene classification result of each personnel.

[0026] Finally, the violation behavior judgment module 80 matches the personal protective equipment binary tuple set associated with each person with the personal protective equipment rule set corresponding to the action scene classification result of each person, to identify whether each person has a violation behavior.

[0027] Figure 2 is a flow diagram of a video analysis-based violation behavior detection method according to the first embodiment of the present application. As shown in Figure 2 , the method comprises the following steps: Step 110, extracting a 3D feature map containing continuous multiple frames from a real-time collected video stream through a video backbone network, and cutting a center key frame and front and rear key frames located at a preset step before and after the center key frame from the 3D feature map; Step 120, respectively extracting a main feature matrix corresponding to the center key frame and a reference feature matrix corresponding to the front and rear key frames, calculating a first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map to obtain a key frame feature matrix; Step 130, identifying detection bounding boxes of personnel, key equipment, and personal protective equipment from the key frame feature matrix through a target detection classifier, to obtain a personal protective equipment binary tuple set associated with each person; Step 140, extracting a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, calculating a second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix; Step 150, performing action scene classification on the scene feature matrix through an action scene classifier, and outputting an action scene classification result of each person; Step 160, matching the personal protective equipment binary tuple set associated with each person with the personal protective equipment rule set corresponding to the action scene classification result of each person, to identify whether each person has a violation behavior.

[0028] In step 110 of the present embodiment, a 3D feature map containing continuous multiple frames is extracted from a real-time collected video stream through a video backbone network, which can be realized based on an existing lightweight video backbone network (backbone), such as MobileNet, EfficientNet, Video Swim Transformer, etc. The 3D feature map containing continuous multiple frames can be represented as . T represents the number of frames, H represents the height of the image frame, and W represents the width of the image frame.

[0029] In the embodiment, a group of key frames are first extracted from the 3D feature map containing continuous multiple frames, the group of key frames including a center key frame and front and rear key frames located at a preset step length before and after the center key frame. Specifically, the front and rear key frames located at the preset step length before and after the center key frame can include at least one front key frame and at least one rear key frame located at the preset step length before and after the center key frame.

[0030] As one of the examples, the center key frame can be extracted from the 3D feature map at a certain time interval , and according to a preset time step t, a front key frame and a rear key frame located at the preset time step t before and after the center key frame are extracted respectively; or, according to the preset time step t, multiple front key frames 、 and multiple rear key frames 、 located at the preset time step t before and after the center key frame are extracted respectively. For the convenience of description, in the description of the following embodiments, only the embodiment of the front key frame and the rear key frame located at the preset time step t before and after the center key frame is described, and it can be understood by those skilled in the art that the steps in the embodiment can be extended to the embodiment of multiple front key frames and multiple rear key frames.

[0031] In step 120 of the embodiment, after the center key frame and the front and rear key frames located at the preset step length before and after the center key frame are extracted from the 3D feature map in the foregoing steps, the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames are further extracted respectively, the first attention map is calculated based on the main feature matrix and the reference feature matrix by a transformer model (Transformer), and the main feature matrix is aggregated based on the first attention map to obtain a key frame feature matrix for feature aggregation between multiple key frames.

[0032] The main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames can be obtained through feature extraction based on a region proposal network (RPN) and an ROI pooling operation. In this step, due to the image distortion, non-uniform resolution, and complex background of the original video captured by the wide-angle camera, the occlusion and blur between objects in the original frame, and the like, the detection of target objects in the image frame, especially small-size target objects such as personal protective equipment worn by personnel, will have a large error. By extracting the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames, respectively, calculating a first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map, a key frame feature matrix that is enhanced through feature aggregation between multiple key frames is obtained, which can significantly reduce the detection error of target objects in the image frame, especially small-size target objects such as personal protective equipment, caused by the occlusion and blur between objects in the original frame.

[0033] In step 130 of this embodiment, after obtaining the key frame feature matrix in the foregoing steps, a target detection classifier (for example, a fully connected layer based on a CNN network) can be further used to identify the detection bounding boxes of personnel, key equipment, and personal protective equipment from the key frame feature matrix, to obtain a set of binary tuples of personal protective equipment associated with each personnel. The personnel are workers in the video images collected in the current factory environment, the key equipment can include electrical equipment, mechanical equipment, and work tools that interact or are associated with personnel in different work scenarios in the current factory environment, the key equipment that interacts or is associated with personnel can be used to identify the action scenario classification of the personnel, and the personal protective equipment represents personal protective equipment worn by the personnel, such as safety helmets, protective gloves, safety shoes, welding masks, reflective clothing, safety ropes, and the like. In different work scenarios, there are specific safety regulations for the personal protective equipment that personnel need to wear. One of the purposes of this embodiment is to accurately identify whether there is a violation of not wearing a specified type of personal protective equipment by personnel in different work scenarios, so as to timely perform industrial safety warning.

[0034] In this step, the set of detection bounding boxes of personnel can be represented as The set of detection bounding boxes of key equipment can be represented as The personal protective equipment associated with each personnel represents a subset of personal protective equipment currently worn by the personnel, such as the detection bounding box The personal protective equipment associated with the personnel corresponding to the detection bounding box is represented as a set of binary tuples where n and m are the number of personnel and key equipment, respectively, and s is the number of each type of personal protective equipment involved by each personnel. the jth category of personal protective equipment, for detecting the bounding box The identification of whether the corresponding personnel detects the jth category of personal protective equipment can be identified as 0 or 1, 0 for not detected, and 1 for detected. As an example, the categories of personal protective equipment can include: : safety helmet, : protective gloves, : safety shoes, : welding mask, : reflective clothing, : safety rope, and many other categories.

[0035] In step 140 of the embodiment, the personnel feature matrix corresponding to the personnel and the associated feature matrix between the personnel and the key equipment are further extracted from the center key frame according to the detection bounding box of the personnel and the key equipment, a second attention map is calculated based on the personnel feature matrix and the associated feature matrix, and the personnel feature matrix is aggregated based on the second attention map to obtain a scene feature matrix. The personnel feature matrix corresponding to the personnel is a one-dimensional feature vector set extracted from the center key frame by the ROI pooling operation of the feature extraction module 30 according to the detection bounding box of the personnel. The associated feature matrix between the personnel and the key equipment is a feature matrix representing the association relationship between each personnel and the key equipment, which can be obtained based on the associated bounding box between the personnel and the key equipment.

[0036] In this step, the second attention map can be calculated based on the personnel feature matrix and the associated feature matrix by a transformer model (Transformer), and the personnel feature matrix is aggregated based on the second attention map, so as to obtain the scene feature matrix after the features of the personnel and the associated key equipment are aggregated. The scene feature matrix enhances the features of the interaction and association between the personnel and the key equipment. Subsequently, in step 150 of the embodiment, the aggregated scene feature vector can be further predicted by a pre-trained action scene classifier, and the action scene classification result of each personnel can be obtained.

[0037] Finally, in step 160 of the embodiment, the binary tuple set of personal protective equipment associated with each personnel is matched with the rule set of personal protective equipment corresponding to the action scene classification result of each personnel, and whether each personnel has a violation behavior is identified.

[0038] In this step, each action scene classification in which the personnel is located is associated with a preset rule set of personal protective equipment, The meaning of the rule set is to set the identification of whether the personal protective equipment of each category must be worn for the gth action scene classification, the jth category of personal protective equipment set for the gth action scene classification, the jth category of personal protective equipment set for the gth action scene classification,

[0039] Therefore, the embodiment can quickly identify the category of personal protective equipment involved in the violation behavior of each personnel based on the set matching between the binary tuple set of personal protective equipment associated with each personnel and the rule set of personal protective equipment corresponding to the action scene classification result of each personnel, and determine whether each personnel has a violation behavior of not wearing a certain category of personal protective equipment that does not comply with the constraint of the rule set of personal protective equipment.

[0040] In the embodiment of the application, on the one hand, the center key frame and the front and rear key frames located a preset step before and after the center key frame are intercepted from the 3D feature map output by the video backbone network, the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames are extracted respectively, the first attention map is calculated based on the main feature matrix and the reference feature matrix, and the main feature matrix is aggregated and enhanced based on the first attention map to obtain an enhanced key frame feature matrix. The detection bounding box of personnel, key equipment and personal protective equipment and the binary tuple set of personal protective equipment associated with each personnel are identified from the enhanced key frame feature matrix by a target detection classifier, so that the detection error of the target object in the image frame, especially the small-size target object such as personal protective equipment worn by personnel, can be effectively improved, and the accuracy of target detection can be improved.

[0041] On the other hand, the personnel feature matrix corresponding to the personnel and the associated feature matrix between the personnel and the key equipment are extracted from the center key frame according to the detection bounding box of the personnel and the key equipment, the second attention map is calculated based on the personnel feature matrix and the associated feature matrix, the personnel feature matrix is aggregated based on the second attention map to obtain a scene feature matrix, and the action scene classification result of each personnel is output by performing action scene classification on the scene feature matrix by an action scene classifier, so that the occlusion and blur between objects in the original frame caused by the image distortion, non-uniform resolution and complex background of the original video collected by the wide-angle camera can be quickly and accurately identified, and the specific action scene of the personnel in the factory environment can be quickly and accurately identified, and then the violation behavior of each personnel in the specific action scene can be effectively identified.

[0042] In one embodiment, as Figure 3As shown, in the step 120, the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames are extracted respectively, the first attention map is calculated based on the main feature matrix and the reference feature matrix, and the main feature matrix is aggregated based on the first attention map to obtain the key frame feature matrix, which can include the following steps: In step 210, the front feature matrix extracted from the front key frame in the front and rear key frames and the rear feature matrix extracted from the rear key frame in the front and rear key frames are spliced and fused to obtain the reference feature matrix. In step 220, the first linear transformation of the main feature matrix is calculated by the converter model to obtain the query matrix, the second linear transformation of the reference feature matrix is calculated to obtain the key matrix, and the first attention map is calculated based on the point product of the softmax function operation of the query matrix and the key matrix. In step 230, the third linear transformation of the point product of the first attention map and the reference feature matrix is calculated, and the main feature matrix is added to obtain the key frame feature matrix.

[0043] In this embodiment, first, the center key frame , the front key frame and the rear key frame are respectively extracted by the RPN (Region Proposal Network) region candidate network and the ROI pooling operation, and the feature vectors containing N candidate bounding boxes can be identified from each key frame. The feature vector set extracted from the center key frame is the main feature matrix, denoted as ; the front feature matrix extracted from the front key frame is denoted as ; and the rear feature matrix extracted from the rear key frame is denoted as . The target objects involved in the N candidate bounding boxes can include personnel, key equipment, and personal protective equipment worn by each person detected in each frame. Personnel are workers in the video images collected in the current factory environment, key equipment are key equipment in the current factory environment, such as electrical equipment, mechanical equipment, and work tools that interact or are associated with personnel in different work scenes, and the key equipment that interacts or is associated with personnel can be used to identify the current action scene classification of personnel, and personal protective equipment represents personal protective equipment worn by each person.

[0044] Subsequently, the front feature matrix and the rear feature matrix can be spliced and fused, for example, the front feature matrix and the rear feature matrix can be added or averaged to obtain a fused feature matrix as the reference feature matrix, denoted as .

[0045] Due to the image distortion, non-uniform resolution and complex background of the original video collected by the wide-angle camera, etc., resulting in occlusion and blur between objects in the original frame, the detection of target objects in the image frame, especially small-size target objects such as personal protective equipment worn by personnel, will have a large error. In this step, the first linear transformation of the main feature matrix can be obtained by calculating the query matrix through the transformer model, and the second linear transformation of the reference feature matrix can be obtained by calculating the key matrix, so that the first attention map can be calculated based on the point product of the softmax function operation of the query matrix and the key matrix.

[0046] The calculation method of the query matrix is as follows: The calculation method of the key matrix is as follows: . and is a linear transformation operation in the transformer model.

[0047] The calculation formula of the first attention map is as follows: , wherein represents the transpose operation of the key matrix, and d is the dimension of the main feature matrix and the reference feature matrix. The function is used to convert the matrix operation result into a probability distribution with a sum of 1 as the attention weight.

[0048] Subsequently, taking the reference feature matrix as the value matrix, the third linear transformation of the point product of the first attention map and the reference feature matrix is calculated, and the main feature matrix of the center key frame is added, so that the enhanced key frame feature matrix can be obtained. This can be realized by the feedforward neural network layer of the transformer model. The enhanced key frame feature matrix can be represented as . is a linear transformation operation in the transformer model.

[0049] In one embodiment, as shown in Figure 4 , in the step 140, the personnel feature matrix corresponding to the personnel and the associated feature matrix between the personnel and the key equipment are extracted from the center key frame according to the detection bounding box of the personnel and the key equipment, which can include: Step 310, determining the associated bounding box between each personnel and each key equipment from the center key frame according to the detection bounding box of the personnel and the key equipment, wherein the associated bounding box at least contains each personnel and each key equipment; Step 320: Extract the one-dimensional feature vector corresponding to the associated bounding box between each person and each key device from the central keyframe to obtain the associated feature matrix between the person and the key device.

[0050] In this embodiment, the associated bounding box between each person and each key device can be determined from the central key frame based on the detected bounding boxes of the personnel and key devices. Thus, a one-dimensional feature vector corresponding to the associated bounding box between each person and each key device can be extracted from the central key frame based on the associated bounding box between each person and each key device, thereby obtaining the associated feature matrix between personnel and key devices.

[0051] The associated bounding box between each person and each key device represents the minimum detection bounding box containing each person and each key device.

[0052] In one implementation, the associated bounding box between each person and each key device can be determined based on the minimum value of the top-left corner coordinate and the maximum value of the bottom-right corner coordinate of the detection bounding box of each person and each key device. Assume the coordinates of the detection bounding box of the i-th person are represented as... The coordinates of the detection bounding box of the h-th critical device are represented as follows: ;in, This represents the coordinates of the top-left corner of the bounding box for the i-th person. This represents the coordinates of the bottom right corner of the detection bounding box for the i-th person. This represents the coordinates of the top-left corner of the detection bounding box of the h-th critical device. This represents the coordinates of the lower right corner of the detection bounding box of the h-th critical device. , .

[0053] Then, the coordinates of the bounding box connecting the i-th person and the h-th key device can be represented as: ,Right now , , , min represents the minimum value function, and max represents the maximum value function.

[0054] In one implementation, such as Figure 5 As shown, in step 140, calculating the second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain the scene feature matrix, may include: Step 410: Perform dimension alignment on the personnel feature matrix by copying and filling each row element according to the row dimension to obtain a personnel feature transformation matrix with the same dimension as the associated feature matrix. Step 420: Calculate the softmax function operation of the Hadamard product of the personnel feature transformation matrix and the associated feature matrix to obtain the second attention map; Step 430: Calculate the fourth linear transformation of the dot product between the second attention map and the associated feature matrix, and add it to the personnel feature matrix to obtain the scene feature matrix.

[0055] In this embodiment, based on the detected bounding boxes of the personnel, a one-dimensional feature vector set, i.e., a personnel feature matrix, can be extracted from the central keyframe using ROI pooling operations. This personnel feature matrix can be represented as... .

[0056] Subsequently, using convolutional layer operations with zero padding and aligned ROI pooling operations, one-dimensional feature vectors corresponding to the associated bounding boxes between the i-th person and the h-th key equipment are extracted from the center keyframe. Therefore, the association feature matrix corresponding to the associated bounding box between each person and each key piece of equipment can be obtained. , The associated feature matrix is ​​an n*m dimensional matrix. n and m are the number of personnel and key equipment detected from the central keyframe, respectively, which are also the number of detection bounding boxes for personnel and key equipment.

[0057] In one implementation, to focus on the relationships between key devices within a certain radius of a person's movement scene, the associated bounding boxes between the person and the key devices can be limited to a certain distance threshold range. This involves first calculating the distance between the centers of the detection bounding boxes of each person and each key device, and then extracting only the associated bounding boxes between each person and each key device where the distance between their centers is less than or equal to a preset distance threshold. In this case, the aforementioned association feature matrix... In this context, when the distance between the centers of the detection bounding boxes of each person and each key device is greater than a preset distance threshold, the one-dimensional feature vector corresponding to the associated bounding box between the target person and the key device can be set to a zero vector.

[0058] Subsequently, the Hadamard product between the association feature matrix and the personnel feature matrix can be calculated using a transformer model to obtain the second attention map. Specifically, the personnel feature matrix is ​​first dimensionally aligned by copying and padding each row according to its row dimensions to obtain the second attention map. The personnel feature transformation matrix has the same n*m ​​dimensions as the given matrix. This personnel feature transformation matrix can be represented as: .

[0059] The second attention map can be calculated as follows: .in, This represents the Hadamard product between matrices. The function is used to convert the result of matrix operations into a probability distribution that sums to 1, which is then used as attention weights.

[0060] Subsequently, the second attention map is calculated using the converter model. and correlation feature matrix The fourth linear transformation of the dot product between the elements involves summing the associated feature matrices by row-wise attention weights, and finally adding this sum to the target person's feature matrix to obtain the enhanced scene feature matrix. . This refers to the linear transformation operation in the converter model.

[0061] This scene feature matrix enhances the characteristics of the interaction and association between personnel and key equipment. Finally, a pre-trained action scene classifier is used to predict the action scene classification of the scene feature matrix, which can accurately obtain the action scene classification result for each person. As an example, action scene classification can include multiple categories such as welding operations, factory walking, equipment operation, and working at height.

[0062] In one implementation, such as Figure 6 As shown, the method further includes: Step 510 involves the following steps: for the preceding and following keyframes, respectively, extracting the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and key equipment; calculating a second attention map based on the personnel feature matrix and the association feature matrix; and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix; and classifying the scene feature matrix into action scenes using an action scene classifier and outputting the action scene classification result for each personnel. Step 520: Perform multiplication fusion calculation on the confidence scores of the action scene classification results of each person output by the central keyframe and the preceding and following keyframes to obtain the final action scene classification and fusion confidence score for each person.

[0063] In this embodiment, the action scene classification steps described in the previous embodiment can be performed on both the preceding and following keyframes. The action scene classification results of each person predicted by the center keyframe, the preceding keyframe, and the following keyframe are used to determine the final action scene classification result of each person based on the confidence level multiplicative fusion calculation. The advantage of this approach is that it can reduce the impact of misclassification in individual keyframes. If the prediction probability of a certain action scene category in a keyframe is very low, then the confidence level after multiplicative fusion will also be very low. Only when the confidence levels of the classification prediction results in each keyframe tend to be high probability values ​​will the final fusion confidence level obtained after multiplicative fusion calculation exceed the preset threshold. At this time, the final action scene classification will be more reliable, further reducing the error of action scene classification prediction.

[0064] Suppose that, for the central keyframe, the confidence (prediction probability) of classifying the scene of the g-th action of the i-th person can be expressed as: For the preceding keyframes, the confidence (prediction probability) of classifying the scene of the g-th action of the i-th person can be expressed as: For later keyframes, the confidence (prediction probability) of classifying the scene of the g-th action of the i-th person can be expressed as: Then, the fusion confidence score for the scene classification of the g-th action of the i-th person can be calculated as follows: .

[0065] In one implementation, such as Figure 7 As shown, in step 160, matching the set of binary pairs of personal protective equipment associated with each person with the set of rules for personal protective equipment corresponding to the action scenario classification result of each person to identify whether each person has violated regulations may further include: Step 610: Based on the index order of each category of personal protective equipment, map the identifier of whether each category of personal protective equipment in the rule set is required to wear to a binary rule bitmask; map the identifier of whether each category of personal protective equipment in the tuple set is detected to a binary wearing status bitmask of the same number of bits; and invert the wearing status bitmask to obtain the inverted bitmask corresponding to the wearing status bitmask. Step 620: Perform a logical AND operation on the rule bitmask and the reverse bitmask, identify the binary bits that are 1 in the operation result, and thus identify the category of personal protective equipment involved in each person's violation.

[0066] In this embodiment, to reduce the consumption of computing resources and improve the processing performance of violation identification, the identifiers of whether each category of personal protective equipment (PPE) must be worn in the rule set can be mapped to a binary rule bitmask according to the index order of each category of PPE. Similarly, the identifiers of whether each category of PPE is detected in the set of binary PPE associated with each person can also be mapped to a binary wearing status bitmask of the same number of bits. The wearing status bitmask identifier is then inverted to obtain the inverse bitmask corresponding to the wearing status bitmask. Subsequently, the rule bitmask and the inverse bitmask are logically ANDed. If the result contains a 1, it is the bit corresponding to the PPE that was not worn in violation. Thus, by converting the set into bit operations, the category of PPE involved in each person's violation can be quickly identified, and it can be determined whether each person has violated the rule set constraints by not wearing a certain type of PPE.

[0067] Figure 8 This is a schematic diagram of the structure of a violation detection device based on video analysis according to the first embodiment of this application. Figure 8 As shown, the device includes the following units: The keyframe acquisition unit 710 is used to extract a 3D feature map containing multiple consecutive frames from the real-time acquired video stream through the video backbone network, and to extract the central keyframe and the keyframes before and after the central keyframe with a preset step size from the 3D feature map. The keyframe feature extraction unit 720 is used to extract the main feature matrix corresponding to the central keyframe and the reference feature matrix corresponding to the preceding and following keyframes, respectively, calculate a first attention map based on the main feature matrix and the reference feature matrix, and aggregate the main feature matrix based on the first attention map to obtain the keyframe feature matrix. The target detection unit 730 is used to identify the detection bounding boxes of personnel, key equipment, and personal protective equipment from the key frame feature matrix through the target detection classifier, and obtain a set of binary personal protective equipment associated with each personnel. The scene feature extraction unit 740 is used to extract the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment from the central key frame based on the detection bounding boxes of the personnel and key equipment, calculate a second attention map based on the personnel feature matrix and the association feature matrix, and aggregate the personnel feature matrix based on the second attention map to obtain the scene feature matrix; Scene classification unit 750 is used to classify the scene feature matrix by an action scene classifier and output the action scene classification result for each person. The violation identification unit 760 is used to match the set of binary personal protective equipment associated with each person with the set of rules for personal protective equipment corresponding to the action scene classification result of each person, and to identify whether each person has violated regulations.

[0068] In one implementation, such as Figure 9 As shown, the keyframe feature extraction unit 720 may further include: The feature splicing subunit 721 is used to splice and fuse the earlier feature matrix extracted from the earlier keyframe and the later feature matrix extracted from the later keyframe to obtain the reference feature matrix. The first computational subunit 722 is used to calculate the first linear transformation of the main feature matrix to obtain the query matrix through the converter model, calculate the second linear transformation of the reference feature matrix to obtain the key matrix, and calculate the first attention map based on the softmax function operation of the dot product of the query matrix and the key matrix. The second computational subunit 723 is used to calculate the third linear transformation of the dot product of the first attention map and the reference feature matrix, and add it to the main feature matrix to obtain the keyframe feature matrix.

[0069] In one implementation, such as Figure 10 As shown, the scene feature extraction unit 740 may include: The associated bounding box determination subunit 741 is used to determine the associated bounding box between each person and each key device from the central key frame based on the detected bounding boxes of the personnel and key devices, wherein the associated bounding box at least includes each person and each key device. The associated feature extraction subunit 742 is used to extract the one-dimensional feature vector corresponding to the associated bounding box between each person and each key device from the central key frame, so as to obtain the associated feature matrix between the person and the key device.

[0070] In one implementation, such as Figure 11 As shown, the scene feature extraction unit 740 may include: The feature transformation subunit 743 is used to perform a dimension alignment operation on the personnel feature matrix, copying and filling each row element according to the row dimension to obtain a personnel feature transformation matrix with the same dimension as the associated feature matrix. The third calculation subunit 744 is used to calculate the softmax function operation of the Hadamard product of the personnel feature transformation matrix and the associated feature matrix to obtain the second attention map; The fourth calculation subunit 745 is used to calculate the fourth linear transformation of the dot product between the second attention map and the associated feature matrix, and add it to the personnel feature matrix to obtain the scene feature matrix.

[0071] In one embodiment, the scene feature extraction unit 740 is further configured to: extract the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment from the preceding and following key frames according to the detection bounding boxes of the personnel and key equipment, calculate a second attention map based on the personnel feature matrix and the association feature matrix, and aggregate the personnel feature matrix based on the second attention map to obtain a scene feature matrix.

[0072] The scene classification unit 750 is further configured to: classify the scene feature matrix obtained from the preceding and following keyframes into action scenes and output the action scene classification result for each person.

[0073] The scene classification unit 750 is also used to: perform multiplicative fusion calculation on the confidence of the action scene classification results of each person output by the central keyframe and the preceding and following keyframes respectively, to obtain the final action scene classification and fusion confidence of each person.

[0074] In one embodiment, the violation identification unit 760 is further configured to: Based on the index order of each category of personal protective equipment, the identifier of whether each category of personal protective equipment in the rule set must be worn is mapped to a binary rule bitmask. The identifier of whether each category of personal protective equipment in the tuple set is detected is mapped to a binary wearing status bitmask of the same number of bits. The wearing status bitmask is then inverted to obtain the inverted bitmask corresponding to the wearing status bitmask. By performing a logical AND operation on the rule bitmask and the inverse bitmask, the binary bits that are 1 in the result are identified, thereby identifying the category of personal protective equipment involved in each person's violation.

[0075] The video analysis-based violation detection method and apparatus proposed in this application, on the one hand, extracts a central keyframe and keyframes located before and after the central keyframe at preset step lengths from the 3D feature map output by the video backbone network. It then extracts the main feature matrix corresponding to the central keyframe and the reference feature matrix corresponding to the keyframes before and after it. Based on the main feature matrix and the reference feature matrix, a first attention map is calculated, and the main feature matrix is ​​aggregated and enhanced based on the first attention map to obtain an enhanced keyframe feature matrix. A target detection classifier identifies the detection bounding boxes of personnel, key equipment, and personal protective equipment, as well as the set of binary pairs of personal protective equipment associated with each personnel, from the enhanced keyframe feature matrix. This effectively addresses issues such as image distortion, uneven resolution, and complex backgrounds in the original video captured by wide-angle cameras, which cause occlusion and blurring between objects in the original frames. It also improves the detection error of target objects in image frames, especially small-sized target objects such as personal protective equipment worn by personnel, thereby increasing the accuracy of target detection.

[0076] On the other hand, based on the detection bounding boxes of personnel and key equipment, the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and key equipment are extracted from the central key frame. A second attention map is calculated based on the personnel feature matrix and the association feature matrix, and the personnel feature matrix is ​​aggregated based on the second attention map to obtain the scene feature matrix. The scene feature matrix is ​​then classified into action scenes using an action scene classifier, and the action scene classification result for each person is output. This can quickly and accurately identify specific action scenes of personnel in the factory environment, which can be addressed by addressing issues such as image distortion, uneven resolution, and complex backgrounds in the original video captured by the wide-angle camera, which can lead to occlusion and blurring between objects in the original frame. This can effectively identify the violations of each person in specific action scenes.

[0077] Furthermore, the lightweight model used in this application embodiment can effectively reduce the consumption of hardware computing power and computing resources, and can be deployed in edge computing devices in a factory environment to realize real-time visual safety behavior monitoring in complex factory environments.

[0078] It should be noted that those skilled in the art will understand that the different implementation methods, their descriptions and explanations, and the technical effects achieved as described in the method embodiments of this application are also applicable to the device embodiments of this application, and will not be repeated here.

[0079] Furthermore, this application also proposes an edge computing device, which may include a processor and a memory. The memory stores computer program instructions, and the processor can invoke these instructions to execute all or part of the steps of the method described in any embodiment of this application. The computer program instructions in the memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. This edge computing device can be deployed in complex factory environments to achieve real-time visual safety behavior monitoring in such environments.

[0080] Furthermore, embodiments of this application also propose an electronic device, which may include a processor and a memory. The memory stores computer program instructions, and the processor can invoke the computer program instructions in the memory to execute all or part of the steps of the method described in any embodiment of this application. The computer program instructions in the memory described above can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0081] Furthermore, this application also provides a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, which, when connected to a computer device, is executed by one or more processors of the computer device, is capable of performing all or part of the steps of the method described in any embodiment of this application.

[0082] Furthermore, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which can be executed by one or more processors to perform all or part of the steps of the method described in any embodiment of this application.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that the various embodiments of this application can be implemented by means of software or software combined with necessary general-purpose hardware platforms, and of course, can also be implemented by hardware functions. Based on this understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device, such as, but not limited to, a personal computer, a server, or a network device, to execute all or part of the steps of the method described in any embodiment of this application. The aforementioned storage medium may include: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and other media capable of storing computer program code.

[0084] The foregoing description of exemplary embodiments of this application should be understood as not limiting, but illustrative, and the scope of protection of this application is not limited thereto. It should be understood that those skilled in the art can make modifications and variations to the embodiments of this application without departing from the spirit and scope of this application, and such modifications and variations should be within the scope of protection of this application.

Claims

1. A method for detecting a violation based on video analysis, characterized in that, The method comprises: extracting a 3D feature map containing continuous multiple frames from a real-time collected video stream through a video backbone network, and cutting a center key frame and front and rear key frames located at a preset step length before and after the center key frame from the 3D feature map; extracting a main feature matrix corresponding to the center key frame and a reference feature matrix corresponding to the front and rear key frames respectively, calculating a first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map to obtain a key frame feature matrix; identifying detection bounding boxes of personnel, key equipment and personal protective equipment from the key frame feature matrix through a target detection classifier to obtain a binary tuple set of personal protective equipment associated with each personnel; extracting a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, calculating a second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix; performing action scene classification on the scene feature matrix through an action scene classifier to output an action scene classification result of each personnel; matching the binary tuple set of personal protective equipment associated with each personnel with a rule set of personal protective equipment corresponding to the action scene classification result of each personnel to identify whether each personnel has a violation behavior.

2. The method of claim 1, wherein, The method comprises: splicing and fusing a preceding feature matrix extracted from a preceding key frame in the front and rear key frames and a following feature matrix extracted from a following key frame in the front and rear key frames to obtain the reference feature matrix; calculating a first linear transformation of the main feature matrix to obtain a query matrix, calculating a second linear transformation of the reference feature matrix to obtain a key matrix, and calculating the first attention map based on a point product softmax function operation of the query matrix and the key matrix; calculating a third linear transformation of a point product of the first attention map and the reference feature matrix, and adding the main feature matrix to obtain the key frame feature matrix.

3. The method of rule violation detection based on video analysis according to claim 2, wherein, The method comprises: determining an association bounding box between each personnel and each key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, wherein the association bounding box at least contains each personnel and each key equipment; extracting a one-dimensional feature vector corresponding to the association bounding box between each personnel and each key equipment from the center key frame to obtain the association feature matrix between the personnel and the key equipment.

4. The method of rule violation detection based on video analysis according to claim 3, wherein, The method further comprises: The method further comprises:

5. The method of video analytics based violation detection as claimed in claim 4, wherein, The method further comprises: The method further comprises:

6. The method of rule violation detection based on video analysis according to claim 5, wherein, The method further comprises: The method further comprises: The method further comprises: The method further comprises:

7. The method of video analytics based misbehavior detection according to claim 6, characterized in that, The method further comprises: The method further comprises: The method further comprises:

8. The method of video analytics based misbehavior detection according to claim 7, wherein, The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further The rule bit mask and the reverse bit mask are logically ANDed to identify binary bits that are 1 in the result, thereby identifying the category of personal protective equipment involved in the rule violation of each person.

9. A video analysis-based misbehavior detection apparatus, characterized by, The device comprises: a key frame acquisition unit configured to extract a 3D feature map containing continuous multiple frames from a real-time video stream through a video backbone network, and to cut a center key frame and front and rear key frames located at a preset step before and after the center key frame from the 3D feature map; a key frame feature extraction unit configured to extract a main feature matrix corresponding to the center key frame and reference feature matrices corresponding to the front and rear key frames respectively, to calculate a first attention map based on the main feature matrix and the reference feature matrices, and to aggregate the main feature matrix based on the first attention map to obtain a key frame feature matrix; a target detection unit configured to identify detection bounding boxes of persons, key equipment and personal protective equipment from the key frame feature matrix through a target detection classifier to obtain a binary tuple set of personal protective equipment associated with each person; a scene feature extraction unit configured to extract a person feature matrix corresponding to a person and an association feature matrix between the person and the key equipment from the center key frame according to the detection bounding boxes of the person and the key equipment, to calculate a second attention map based on the person feature matrix and the association feature matrix, and to aggregate the person feature matrix based on the second attention map to obtain a scene feature matrix; a scene classification unit configured to perform action scene classification on the scene feature matrix through an action scene classifier to output an action scene classification result of each person; a rule violation identification unit configured to match the binary tuple set of personal protective equipment associated with each person with a rule set of personal protective equipment corresponding to the action scene classification result of each person to identify whether each person has a rule violation.

10. An edge computing device, comprising: comprise: one or more processors; a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the steps of the method of any one of claims 1 to 8.

Images