Method, apparatus and device for detecting violation behavior based on video analysis

Abstract

The application discloses a method, device and equipment for detecting rule violation based on video analysis, and belongs to the technical field of video analysis. The method extracts a main feature matrix corresponding to a center key frame and a reference feature matrix corresponding to front and back key frames respectively, aggregates the main feature matrix based on the main feature matrix and the reference feature matrix to obtain a key frame feature matrix by calculating a first attention graph, extracts a personnel feature matrix and an association feature matrix from the center key frame, obtains a scene feature matrix based on the personnel feature matrix and the association feature matrix to perform action scene classification by calculating a second attention graph, matches a binary tuple set of personal protective equipment associated with each personnel with a rule set of personal protective equipment corresponding to the action scene classification result of each personnel, and identifies whether each personnel has a rule violation. The application can improve the accuracy of personnel action scene classification and rule violation detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video image analysis of artificial intelligence, in particular to a method and device for detecting illegal behavior based on video analysis. BACKGROUND

[0002] With the development of deep learning-based computer vision technology, the current industrial safety monitoring in the fields of energy, chemical industry, and power industry relies on worker illegal behavior detection based on video analysis, such as personal protective equipment (safety helmet, protective gloves, welding mask, etc.) detection based on video intelligent analysis to identify whether workers have safety illegal behavior in different work scenarios. In the prior art, a pre-trained deep learning-based target detection model (such as a YOLO model) is usually used to directly identify whether workers in the factory area wear personal protective equipment such as safety helmets, protective gloves, welding masks, and whether there are other illegal behaviors that violate safety regulations. However, this traditional detection scheme has significant limitations, as it cannot distinguish between the special wearing requirements of personal protective equipment in different work scenarios, which can lead to a large number of false positives in the system. For example, the system may incorrectly determine that a worker not wearing a welding mask or protective gloves while walking is an illegal behavior, which seriously affects the credibility of the system detection.

[0003] To reduce system false positives, existing technologies can also use a deep learning model for action behavior detection, such as the SlowFast model, to identify the action scenario of the worker's current work. For example, the existing patent document CN119091360B relates to a posture recognition management system and method for factory workers, which preprocesses video data in each key area based on the YOLOv8 algorithm, identifies relevant information of target workers and trajectory information of automated equipment in the target factory, and outputs the relevant information and trajectory information. At the same time, the SlowFast model is used to process the video data in each key area simultaneously, analyze the actions and postures of the workers, and trigger the recognition of the interactive behavior of the workers under the condition of interaction between the workers and the automated equipment, to comprehensively judge whether the workers have illegal operations.

[0004] Then, this scheme simultaneously uses the YOLOv8 model and the SlowFast model, which has a large amount of calculation and high requirements for hardware computing resources, making it difficult to ensure real-time performance and unsuitable for deployment on edge computing devices in a factory environment. Moreover, in the actual complex factory environment with multi-target visual monitoring using a wide-angle camera, due to image distortion, non-uniform resolution, and complex background of the video captured by the wide-angle camera, the existing scheme has low recognition accuracy in identifying the action scenario of the workers and detecting small-sized personal protective equipment. Therefore, there is an urgent need for an improved technical solution to solve the above technical problems. SUMMARY

[0005] Therefore, the application provides a method, device and equipment for detecting illegal behavior based on video analysis to improve the accuracy of identifying the action scene and target object detection of workers in a factory environment, and thus effectively identify the illegal behavior of each worker in a specific action scene.

[0006] In a first aspect, the application provides a method for detecting illegal behavior based on video analysis, which comprises:

[0007] extracting a 3D feature map containing continuous multiple frames from a real-time collected video stream through a video backbone network, intercepting a center key frame from the 3D feature map according to a certain time interval, and extracting front and rear key frames located at a preset step length before and after the center key frame respectively according to the preset step length;

[0008] extracting a main feature matrix corresponding to the center key frame and a reference feature matrix corresponding to the front and rear key frames respectively, calculating a first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map to obtain a key frame feature matrix;

[0009] identifying the detection bounding box of personnel, key equipment and personal protective equipment from the key frame feature matrix through a target detection classifier to obtain a binary tuple set of personal protective equipment associated with each personnel;

[0010] extracting a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment from the center key frame according to the detection bounding box of the personnel and the key equipment, calculating a second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix;

[0011] classifying the scene feature matrix through an action scene classifier to output an action scene classification result of each personnel;

[0012] matching the binary tuple set of personal protective equipment associated with each personnel with a rule set of personal protective equipment corresponding to the action scene classification result of each personnel to identify whether each personnel has illegal behavior.

[0013] In an optional implementation, the extracting a main feature matrix corresponding to the center key frame and a reference feature matrix corresponding to the front and rear key frames respectively, calculating a first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map to obtain a key frame feature matrix comprises:

[0014] splicing and fusing a preceding feature matrix extracted from a preceding key frame in the preceding and following key frames and a following feature matrix extracted from a following key frame in the preceding and following key frames to obtain the reference feature matrix;

[0015] calculating a first linear transformation of the main feature matrix to obtain a query matrix and a second linear transformation of the reference feature matrix to obtain a key matrix, and calculating the first attention map based on a point product softmax function operation of the query matrix and the key matrix;

[0016] calculating a third linear transformation of a point product of the first attention map and the reference feature matrix, and adding the main feature matrix to obtain the key frame feature matrix.

[0017] In an optional implementation, the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment are extracted from the center key frame according to the detection bounding boxes of the personnel and the key equipment, including:

[0018] determining an association bounding box between each personnel and each key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, the association bounding box containing each personnel and each key equipment at least;

[0019] extracting a one-dimensional feature vector corresponding to the association bounding box between each personnel and each key equipment from the center key frame to obtain the association feature matrix between the personnel and the key equipment.

[0020] In an optional implementation, the association bounding box between each personnel and each key equipment is determined from the center key frame according to the detection bounding boxes of the personnel and the key equipment, the association bounding box containing each personnel and each key equipment at least, including:

[0021] determining the association bounding box between each personnel and each key equipment according to the minimum value of the upper left corner coordinates and the maximum value of the lower right corner coordinates of the detection bounding boxes of each personnel and each key equipment.

[0022] In an optional implementation, the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment are extracted from the center key frame according to the detection bounding boxes of the personnel and the key equipment, including:

[0023] calculating the distance between the centers of the detection bounding boxes of each personnel and each key equipment, and determining the association bounding box between each personnel and each key equipment when the distance between the centers of the detection bounding boxes of each personnel and each key equipment from the center key frame is less than or equal to a preset distance threshold.

[0024] In an optional implementation, the method further comprises:

[0025] aligning dimensions of the personnel feature matrix, copying and filling elements in each row in the row dimension to obtain a personnel feature transformation matrix with the same dimensions as the association feature matrix;

[0026] calculating a softmax function operation of a Hadamard product of the personnel feature transformation matrix and the association feature matrix to obtain the second attention map;

[0027] calculating a fourth linear transformation of a dot product between the second attention map and the association feature matrix, and adding the personnel feature matrix to obtain the scene feature matrix.

[0028] In an optional implementation, the method further comprises:

[0029] for the front and rear key frames, respectively performing the steps of extracting the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment, calculating a second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix; and performing the steps of classifying the scene feature matrix by an action scene classifier and outputting an action scene classification result of each personnel;

[0030] performing multiplicative fusion calculation on the confidence of the action scene classification result of each personnel output by the center key frame, the front key frame, and the rear key frame to obtain a final action scene classification and fusion confidence of each personnel.

[0031] In an optional implementation, the method further comprises:

[0032] According to the index order of each category of personal protective equipment, mapping the identifier of whether each category of personal protective equipment contained in the rule set must be worn to a binary rule bit mask, mapping the identifier of whether each category of personal protective equipment contained in the binary tuple set is detected to a binary wearing state bit mask with the same number of bits, and performing an inversion operation on the wearing state bit mask to obtain an inverse bit mask corresponding to the wearing state bit mask;

[0033] performing a logical AND operation on the rule bit mask and the reverse bit mask, identifying binary bits that are 1 in the operation result, and thereby identifying the category of personal protective equipment involved in the rule violation behavior of each person.

[0034] In a second aspect, the present application further provides a rule violation behavior detection device based on video analysis, comprising:

[0035] a key frame collection unit configured to extract a 3D feature map containing continuous multiple frames from a real-time collected video stream through a video backbone network, to intercept a center key frame from the 3D feature map at a certain time interval, and to extract a front key frame and a rear key frame located at a preset step length before and after the center key frame respectively according to the preset step length;

[0036] a key frame feature extraction unit configured to extract a main feature matrix corresponding to the center key frame and a reference feature matrix corresponding to the front and rear key frames respectively, to calculate a first attention map based on the main feature matrix and the reference feature matrix, and to aggregate the main feature matrix based on the first attention map to obtain a key frame feature matrix;

[0037] a target detection unit configured to identify detection bounding boxes of personnel, key equipment and personal protective equipment from the key frame feature matrix through a target detection classifier, and to obtain a binary tuple set of personal protective equipment associated with each person;

[0038] a scene feature extraction unit configured to extract a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, to calculate a second attention map based on the personnel feature matrix and the association feature matrix, and to aggregate the personnel feature matrix based on the second attention map to obtain a scene feature matrix;

[0039] a scene classification unit configured to perform action scene classification on the scene feature matrix through an action scene classifier, and to output an action scene classification result of each person;

[0040] a rule violation behavior identification unit configured to match the binary tuple set of personal protective equipment associated with each person with a rule set of personal protective equipment corresponding to the action scene classification result of each person, and to identify whether each person has a rule violation behavior.

[0041] In a third aspect, the present application further provides an edge computing device, comprising:

[0042] one or more processors;

[0043] a memory storing one or more programs that, when executed by the one or more processors, cause the one or more processors to perform the steps of the method as described in any of the preceding embodiments.

[0044] The present application can achieve the following beneficial effects:

[0045] In one aspect of the present application, the center key frame and the front and rear key frames located a preset step before and after the center key frame are intercepted from the 3D feature map output by the video backbone network, the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames are extracted respectively, the first attention map is calculated based on the main feature matrix and the reference feature matrix, and the main feature matrix is enhanced based on the first attention map to obtain an enhanced key frame feature matrix. The detection bounding box of personnel, key equipment and personal protective equipment and the binary set of personal protective equipment associated with each personnel are identified from the enhanced key frame feature matrix through a target detection classifier. Thus, the detection error of the target object in the image frame, especially the small-size target object such as the personal protective equipment worn by the personnel, can be effectively improved, and the accuracy of target detection can be improved, in view of the image distortion, non-uniform resolution and complex background of the original video captured by the wide-angle camera, and the like, which leads to the occlusion and blur between objects in the original frame.

[0046] On the other hand, the personnel feature matrix corresponding to the personnel and the associated feature matrix between the personnel and the key equipment are extracted from the center key frame according to the detection bounding box of the personnel and the key equipment, the second attention map is calculated based on the personnel feature matrix and the associated feature matrix, and the personnel feature matrix is aggregated based on the second attention map to obtain a scene feature matrix. The action scene classification result of each personnel is output by classifying the scene feature matrix through an action scene classifier. Thus, the action scene in which the personnel is located in the factory environment can be quickly and accurately identified in view of the image distortion, non-uniform resolution and complex background of the original video captured by the wide-angle camera, and the like, which leads to the occlusion and blur between objects in the original frame. Thus, the violation behavior of each personnel in the specific action scene can be effectively identified.

[0047] In addition, the lightweight model used in the present application can effectively reduce the consumption of hardware computing power and computing resources, and can be deployed in the edge computing device in the factory environment to realize real-time visual safety behavior monitoring in the complex environment of the factory. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and should not be regarded as a limitation on the scope of the present application.

[0049] Figure 1 is a system architecture diagram suitable for the method of detecting violation behavior based on video analysis according to an embodiment of the present application;

[0050] Figure 2 is a flow diagram of the method of detecting violation behavior based on video analysis according to a first embodiment of the present application;

[0051] Figure 3 is a partial flow diagram of the method of detecting violation behavior based on video analysis according to a second embodiment of the present application;

[0052] Figure 4 is a partial flow diagram of the method of detecting violation behavior based on video analysis according to a third embodiment of the present application;

[0053] Figure 5 is a partial flow diagram of the method of detecting violation behavior based on video analysis according to a fourth embodiment of the present application;

[0054] Figure 6 is a partial flow diagram of the method of detecting violation behavior based on video analysis according to a fifth embodiment of the present application;

[0055] Figure 7 is a partial flow diagram of the method of detecting violation behavior based on video analysis according to a sixth embodiment of the present application;

[0056] Figure 8 is a structure diagram of the device for detecting violation behavior based on video analysis according to a first embodiment of the present application;

[0057] Figure 9 is a partial structure diagram of the device for detecting violation behavior based on video analysis according to a second embodiment of the present application;

[0058] Figure 10 is a partial structure diagram of the device for detecting violation behavior based on video analysis according to a third embodiment of the present application;

[0059] Figure 11 is a partial structure diagram of the device for detecting violation behavior based on video analysis according to a fourth embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. However, it should be understood that the described embodiments are only some exemplary embodiments of the present application, but not all the embodiments, and therefore the following detailed description of the embodiments of the present application is not intended to limit the scope of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0061] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application are only used to distinguish similar objects, and are not used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance.

[0062] As described above, the prior art solutions usually have high requirements for hardware computing resources, and it is difficult to guarantee real-time performance, which is not suitable for deployment on edge computing devices in factory environments. Moreover, in the actual complex factory environment of visual monitoring of multiple targets by using a wide-angle camera, due to the image distortion, non-uniform resolution and complex background of the video collected by the wide-angle camera, etc., the recognition accuracy of the existing technical solutions for recognizing the action scene of workers and detecting small-sized personal protective equipment is low. Therefore, the present application proposes a method, device and equipment for detecting illegal behavior based on video analysis to improve the accuracy of recognizing the action scene of workers and target object detection in a factory environment, and thus the illegal behavior of each worker in a specific action scene can be effectively identified.

[0063] Figure 1 is a system architecture diagram applicable to the method for detecting illegal behavior based on video analysis according to an embodiment of the present application. The system collects video streams in real time in a factory environment through a video stream acquisition module 10, extracts frames through a video backbone network 20, and obtains a 3D feature map of continuous multiple frames. Then, on the one hand, a feature extraction module 30 intercepts a group of key frames from the 3D feature map output by the video backbone network 20, the group of key frames including a center key frame and front and rear key frames located at a preset step length, and outputs a main feature matrix and a reference feature matrix of the group of key frames. The main feature matrix and the reference feature matrix of the group of key frames are input into a multi-frame feature aggregation module 40 for time sequence feature aggregation. The multi-frame feature aggregation module 40 calculates a first attention map based on the main feature matrix and the reference feature matrix of the group of key frames, and aggregates the main feature matrix based on the first attention map to obtain an enhanced key frame feature matrix. Subsequently, a target detection classifier 50 identifies the detection bounding boxes and categories of personnel, key equipment and personal protective equipment worn by personnel based on the key frame feature matrix, and simultaneously outputs a binary tuple set of personal protective equipment associated with each person.

[0064] On the other hand, the detection bounding boxes of the personnel and the key equipment output by the target detection classifier 50 are fed back to the feature extraction module 30, and the feature extraction module 30 extracts the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment from the set of key frames according to the detection bounding boxes of the personnel and the key equipment, inputs the association feature aggregation module 60, and the association feature aggregation module 60 calculates a second attention map based on the personnel feature matrix and the association feature matrix, and aggregates the personnel feature matrix based on the second attention map to obtain an enhanced scene feature matrix. Subsequently, the action scene classifier 70 performs action scene classification prediction on the aggregated scene feature matrix, and outputs the action scene classification result of each personnel.

[0065] Finally, the violation behavior judgment module 80 matches the binary tuple set of the personal protective equipment associated with each personnel with the rule set of the personal protective equipment corresponding to the action scene classification result of each personnel, and identifies whether each personnel has a violation behavior.

[0066] Figure 2 is a flow diagram of a violation behavior detection method based on video analysis according to the first embodiment of the present application. As shown in Figure 2 , the method comprises the following steps:

[0067] Step 110, extracting a 3D feature map containing continuous multiple frames from a real-time collected video stream through a video backbone network, and cutting a center key frame and front and rear key frames located in front and rear of the center key frame from the 3D feature map;

[0068] Step 120, respectively extracting a main feature matrix corresponding to the center key frame and a reference feature matrix corresponding to the front and rear key frames, calculating a first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map to obtain a key frame feature matrix;

[0069] Step 130, identifying detection bounding boxes of personnel, key equipment and personal protective equipment from the key frame feature matrix through a target detection classifier, and obtaining a binary tuple set of personal protective equipment associated with each personnel;

[0070] Step 140, extracting a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, calculating a second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix;

[0071] In step 150, the action scene classifier is used to classify the scene feature matrix, and the action scene classification result of each person is output.

[0072] In step 160, the binary set of personal protective equipment associated with each person is matched with the rule set of personal protective equipment corresponding to the action scene classification result of each person, and it is determined whether there is a violation of each person.

[0073] In step 110 of the embodiment, the 3D feature map containing continuous multiple frames is extracted from the real-time collected video stream by the video backbone network. The video backbone network can be implemented based on existing lightweight video backbone network, such as MobileNet, EfficientNet, Video Swim Transformer, etc. The 3D feature map containing continuous multiple frames can be represented as T represents the number of frames, H represents the height of the image frame, and W represents the width of the image frame.

[0074] In the embodiment, a group of key frames is first extracted from the 3D feature map containing continuous multiple frames, including a center key frame and a front key frame and a rear key frame located in front and behind the center key frame by a preset step length. Specifically, the front key frame and the rear key frame located in front and behind the center key frame by a preset step length can include at least one front key frame and at least one rear key frame located in front and behind the center key frame by a preset step length.

[0075] As one of the examples, the center key frame can be extracted from the 3D feature map at a certain time interval , and according to a preset time step t, a front key frame and a rear key frame located in front and behind the center key frame by a preset step length t are extracted, respectively; or according to a preset time step t, a plurality of front key frames 、 and a plurality of rear key frames 、 located in front and behind the center key frame by a preset step length t are extracted, respectively. For ease of description, only the implementation of the front key frame and the rear key frame located in front and behind the center key frame by a preset step length t is described in the following implementation, and those skilled in the art can understand that the steps in the embodiment can be extended to the implementation of multiple front key frames and multiple rear key frames.

[0076] In step 120 of the embodiment, after the aforementioned step of intercepting the center key frame and the front and rear key frames located a preset step before and after the center key frame from the 3D feature map, the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames can be further extracted, the first attention map is calculated based on the main feature matrix and the reference feature matrix through the transformer model, and the main feature matrix is aggregated based on the first attention map to obtain the key frame feature matrix for feature aggregation between multiple key frames.

[0077] The main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames can be obtained by feature extraction based on the RPN (Region Proposal Network) region candidate network and the ROI pooling operation. In this step, due to the image distortion, non-uniform resolution and complex background of the original video captured by the wide-angle camera, the occlusion and blur between objects in the original frame, the detection of target objects in the image frame, especially small-size target objects such as personal protective equipment worn by personnel, will have large errors. By extracting the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames, calculating the first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map, the key frame feature matrix for feature aggregation and enhancement between multiple key frames is obtained, which can significantly reduce the detection error of target objects in the image frame, especially small-size target objects such as personal protective equipment, caused by the occlusion and blur between objects in the original frame.

[0078] In step 130 of the embodiment, after obtaining the key frame feature matrix in the foregoing step, the detection bounding box of personnel, key equipment and personal protective equipment can be further identified from the key frame feature matrix through a target detection classifier (such as a fully connected layer based on a CNN network) to obtain a binary tuple set of personal protective equipment associated with each personnel. Personnel are workers in the video images collected in the current factory environment, key equipment can include electrical equipment, mechanical equipment, work tools and other key equipment that interact or are associated with personnel in different work scenarios in the current factory environment, key equipment that interacts or is associated with personnel can be used to identify the action scene classification of personnel, personal protective equipment represents personal protective equipment worn by personnel, such as safety helmets, protective gloves, safety shoes, welding masks, reflective clothing, safety ropes, etc., in different work scenarios, there are specific safety regulations for personal protective equipment that personnel need to wear, one of the purposes of the embodiment is to accurately identify whether there is a violation of not wearing personal protective equipment of a specified category for personnel in different work scenarios, so that industrial safety warning can be performed in time.

[0079] In this step, the set of detection bounding boxes of personnel can represent , and the set of detection bounding boxes of key equipment can represent The personal protective equipment associated with each personnel represents the subset of personal protective equipment currently worn by the personnel, such as the detection bounding box The personal protective equipment associated with the corresponding personnel is represented as a set of binary tuples , where n and m are the number of personnel and key equipment, respectively, and s is the number of categories of personal protective equipment involved for each personnel, is the jth category of personal protective equipment, is the detection bounding box The identification of whether the corresponding personnel detects the jth category of personal protective equipment can be identified as 0 or 1, 0 for not detected, and 1 for detected. As an example, the categories of personal protective equipment can include: : safety helmet, : protective gloves, : safety shoes, : welding mask, : reflective clothing, : safety rope, and many other categories.

[0080] In step 140 of this embodiment, the personnel feature matrix corresponding to the personnel and the associated feature matrix between the personnel and the key equipment are further extracted from the center key frame based on the detection bounding boxes of the personnel and the key equipment. The second attention map is calculated based on the personnel feature matrix and the associated feature matrix, and the personnel feature matrix is aggregated based on the second attention map to obtain a scene feature matrix. The personnel feature matrix corresponding to the personnel is a set of one-dimensional feature vectors extracted by the feature extraction module 30 from the center key frame through ROI pooling operation based on the detection bounding boxes of the personnel. The associated feature matrix between the personnel and the key equipment is a feature matrix representing the association relationship between each personnel and the key equipment, which can be obtained based on the associated bounding boxes between the personnel and the key equipment.

[0081] In this step, the second attention map can be calculated based on the personnel feature matrix and the associated feature matrix by a transformer model, and the personnel feature matrix is aggregated based on the second attention map, thereby obtaining a scene feature matrix after aggregating the features of the personnel and the key equipment associated therewith. This scene feature matrix enhances the features of the interaction and association between the personnel and the key equipment. Subsequently, in step 150 of this embodiment, the aggregated scene feature vector can be further predicted by a pre-trained action scene classifier to obtain the action scene classification result of each personnel.

[0082] Finally, step 160 of the embodiment matches the binary tuple set of personal protective equipment associated with each person with the rule set of personal protective equipment corresponding to the action scene classification result of each person, to identify whether each person has a violation.

[0083] In this step, each action scene classification of the personnel is associated with a preset rule set of personal protective equipment, The rule set means that the gth action scene classification is set for each category of personal protective equipment whether it must be worn, The jth category of personal protective equipment set for the gth action scene classification, The jth category of personal protective equipment set for the gth action scene classification whether it must be worn, which can be identified as 0 or 1, 0 indicating no need to wear, and 1 indicating need to wear.

[0084] Therefore, the embodiment can quickly identify the category of personal protective equipment involved in the violation of each person based on the set matching between the binary tuple set of personal protective equipment associated with each person and the rule set of personal protective equipment corresponding to the action scene classification result of each person, and determine whether each person has a violation of not wearing a certain category of personal protective device that does not comply with the rule set of personal protective equipment.

[0085] In the embodiment of the application, on the one hand, the center key frame and the front and rear key frames located a preset step before and after the center key frame are intercepted from the 3D feature map output from the video backbone network, the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames are extracted respectively, the first attention map is calculated based on the main feature matrix and the reference feature matrix, and the main feature matrix is aggregated and enhanced based on the first attention map to obtain an enhanced key frame feature matrix. The detection bounding box of personnel, key equipment and personal protective equipment and the binary tuple set of personal protective equipment associated with each person are identified from the enhanced key frame feature matrix by a target detection classifier, so that the detection error of the target object in the image frame, especially the small size target object such as personal protective equipment worn by personnel, can be effectively improved, and the accuracy of target detection can be improved. The problem of object occlusion and blur in the original frame caused by image distortion, non-uniform resolution and complex background of the wide-angle camera captured original video.

[0086] On the other hand, based on the detection bounding boxes of personnel and key equipment, the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and key equipment are extracted from the central key frame. A second attention map is calculated based on the personnel feature matrix and the association feature matrix, and the personnel feature matrix is ​​aggregated based on the second attention map to obtain the scene feature matrix. The scene feature matrix is ​​then classified into action scenes using an action scene classifier, and the action scene classification result for each person is output. This can quickly and accurately identify specific action scenes of personnel in the factory environment, which can be addressed by addressing issues such as image distortion, uneven resolution, and complex backgrounds in the original video captured by the wide-angle camera, which can lead to occlusion and blurring between objects in the original frame. This can effectively identify the violations of each person in specific action scenes.

[0087] In one implementation, such as Figure 3 As shown, in step 120, the main feature matrix corresponding to the central keyframe and the reference feature matrix corresponding to the preceding and following keyframes are extracted respectively. A first attention map is calculated based on the main feature matrix and the reference feature matrix, and the main feature matrix is ​​aggregated based on the first attention map to obtain the keyframe feature matrix. This may include the following steps:

[0088] Step 210: The preceding feature matrix extracted from the preceding keyframe and the following feature matrix extracted from the following keyframe are concatenated and fused to obtain the reference feature matrix.

[0089] Step 220: Calculate the first linear transformation of the main feature matrix using the converter model to obtain the query matrix, calculate the second linear transformation of the reference feature matrix to obtain the key matrix, and calculate the first attention map based on the softmax function operation of the dot product of the query matrix and the key matrix.

[0090] Step 230: Calculate the third linear transformation of the dot product of the first attention map and the reference feature matrix, and add it to the main feature matrix to obtain the keyframe feature matrix.

[0091] In this embodiment, firstly, the central keyframes are respectively... In the previous keyframe and in later keyframes Feature extraction is performed using the Region Proposal Network (RPN) and ROI pooling operations. This allows for the identification of feature vectors containing N candidate bounding boxes from each keyframe, starting from the central keyframe. The extracted feature vector set is the principal feature matrix, represented as: From the previous keyframe The extracted preceding feature matrix is represented as The extracted succeeding feature matrix is represented as The extracted succeeding feature matrix is represented as Wherein, the target objects involved in the N candidate boundary boxes can include personnel detected in each frame, key equipment, and personal protective equipment worn by each personnel. The personnel are workers in the video images collected in the current factory environment, the key equipment is the key equipment in the current factory environment, such as electrical equipment, mechanical equipment, and work tools that interact or are associated with personnel in different work scenes, and the key equipment that interacts or is associated with personnel can be used to identify the current action scene classification of the personnel, and the personal protective equipment represents the personal protective equipment worn by each personnel.

[0092] Subsequently, the preceding feature matrix and the succeeding feature matrix can be spliced and fused, for example, the preceding feature matrix and the succeeding feature matrix can be added or averaged to obtain a fused feature matrix as a reference feature matrix, and the reference feature matrix is represented as .

[0093] Due to the image distortion, non-uniform resolution, and complex background of the original video collected by the wide-angle camera, and the occlusion and blur between objects in the original frame, the detection of target objects in the image frame, especially small-size target objects such as personal protective equipment worn by personnel, will have a large error. In this step, the first linear transformation of the main feature matrix can be calculated by the transformer model to obtain a query matrix, and the second linear transformation of the reference feature matrix can be calculated to obtain a key matrix, so that the first attention map can be calculated based on the point product softmax function operation of the query matrix and the key matrix.

[0094] Wherein, the query matrix is calculated as follows: The key matrix is calculated as follows: . And are linear transformation operations in the transformer model.

[0095] The calculation formula of the first attention map is as follows: Wherein represents the transpose operation of the key matrix, and d is the dimension of the main feature matrix and the reference feature matrix. The function is used to convert the matrix operation result into a probability distribution with a sum of 1 as the attention weight.

[0096] Subsequently, taking the reference feature matrix as the value matrix, a third linear transformation of the dot product of the first attention map and the reference feature matrix is calculated, and the main feature matrix of the center key frame is added to obtain an enhanced key frame feature matrix, which can be realized by a feedforward neural network layer of the transformer model. The enhanced key frame feature matrix can be represented as . is a linear transformation operation in the transformer model.

[0097] In one embodiment, as shown in Figure 4 , in the step 140, the personnel feature matrix corresponding to each person and the association feature matrix between each person and each key equipment are extracted from the center key frame according to the detection bounding box of each person and each key equipment, which can include:

[0098] In step 310, the association bounding box between each person and each key equipment is determined from the center key frame according to the detection bounding box of each person and each key equipment, and the association bounding box contains each person and each key equipment.

[0099] In step 320, a one-dimensional feature vector corresponding to the association bounding box between each person and each key equipment is extracted from the center key frame to obtain the association feature matrix between each person and each key equipment.

[0100] In this embodiment, the association bounding box between each person and each key equipment can be determined from the center key frame according to the detection bounding box of each person and each key equipment, so that a one-dimensional feature vector corresponding to the association bounding box between each person and each key equipment can be extracted from the center key frame according to the association bounding box between each person and each key equipment, and the association feature matrix between each person and each key equipment is obtained.

[0101] The association bounding box between each person and each key equipment represents the detection bounding box containing each person and each key equipment.

[0102] In one embodiment, the association bounding box between each person and each key equipment can be determined according to the minimum value of the top-left corner coordinates and the maximum value of the bottom-right corner coordinates of the detection bounding box of each person and each key equipment. Assuming that the coordinates of the detection bounding box of the ith person are represented as , and the coordinates of the detection bounding box of the hth key equipment are represented as ; wherein represents the top-left corner coordinates of the detection bounding box of the ith person, represents the bottom-right corner coordinates of the detection bounding box of the ith person, represents the top-left corner coordinates of the detection bounding box of the hth key equipment, represents the bottom-right coordinate of the detection bounding box of the h-th key device. Wherein , .

[0103] Then, the coordinates of the association bounding box between the i-th person and the h-th key device can be represented as , i.e. , , , .min represents the minimum function, and max represents the maximum function.

[0104] In an embodiment, as shown in Figure 5 , in the step 140, the calculation of the second attention map based on the person feature matrix and the association feature matrix, and the aggregation of the person feature matrix based on the second attention map to obtain the scene feature matrix, can include:

[0105] Step 410, performing a dimension alignment operation on the person feature matrix, and copying and filling each row element in the row dimension to obtain a person feature transformation matrix with the same dimension as the association feature matrix;

[0106] Step 420, calculating the softmax function operation of the Hadamard product of the person feature transformation matrix and the association feature matrix to obtain the second attention map;

[0107] Step 430, calculating the fourth linear transformation of the dot product between the second attention map and the association feature matrix, and adding the person feature matrix to obtain the scene feature matrix.

[0108] In this embodiment, according to the detection bounding box of the person, a one-dimensional feature vector set of the person, i.e., a person feature matrix, can be extracted from the center key frame through an ROI pooling operation. The person feature matrix can be represented as .

[0109] Subsequently, a one-dimensional feature vector corresponding to the association bounding box between the i-th person and the h-th key device is extracted from the center key frame using a convolution layer operation with zero padding and an aligned ROI pooling operation. Thus, an association feature matrix , corresponding to the association bounding box between each person and each key device can be obtained, which is an n m-dimensional matrix. n and m are respectively the number of persons and key devices detected from the center key frame, i.e., the number of detection bounding boxes of persons and key devices.

[0110] In an embodiment, in order to focus on the association between the action scene of the personnel and the key equipment within a certain range of the personnel's surroundings, the association bounding box between the personnel and the key equipment can be limited within a certain distance threshold, that is, the distance between the center of the detection bounding box of each personnel and each key equipment is first calculated, and only the association bounding box between each personnel and each key equipment when the distance between the center of the detection bounding box of each personnel and each key equipment is less than or equal to the preset distance threshold is extracted. At this time, the association feature matrix corresponding to the association bounding box between the target personnel and the key equipment when the distance between the center of the detection bounding box of each personnel and each key equipment is greater than the preset distance threshold can be set to a zero vector.

[0111] Subsequently, the Hadamard product between the association feature matrix and the personnel feature matrix can be calculated by the transformer model to obtain a second attention map. Specifically, first, the personnel feature matrix is subjected to a dimension alignment operation, and each row element is copied and filled according to the row dimension to obtain an n m-dimensional personnel feature transformation matrix with the same dimension as the association feature matrix . The personnel feature transformation matrix can be expressed as: .

[0112] The second attention map can be calculated as: . Wherein, represents the Hadamard product between matrices. The function is used to convert the matrix operation result into a probability distribution with a sum of 1 as the attention weight.

[0113] Subsequently, the fourth linear transformation of the dot product between the second attention map and the association feature matrix is calculated by the transformer model, the dot product operation performs weighted summation on the association feature matrix according to the attention weight in the row dimension, and finally adds the personnel feature matrix of the target personnel to obtain an enhanced scene feature matrix, . is a linear transformation operation in the transformer model.

[0114] The scene feature matrix enhances the features of the interaction and association between the personnel and the key equipment, and finally the action scene classifier is used to classify and predict the scene feature matrix, so that the action scene classification result of each personnel can be accurately obtained. As an example, the action scene classification can include: welding work, factory walking, operating equipment, high-altitude work, and other categories.

[0115] In an embodiment, as Figure 6As shown, the method further comprises:

[0116] Step 510, for the front and rear key frames, respectively, performing the extraction of the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and the key equipment, calculating a second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix; and performing action scene classification on the scene feature matrix by an action scene classifier to output an action scene classification result for each personnel;

[0117] Step 520, performing multiplicative fusion calculation on the confidence of the action scene classification result for each personnel respectively output by the center key frame, the front key frame and the rear key frame to obtain the final action scene classification and fusion confidence of each personnel.

[0118] In this embodiment, the action scene classification steps in the foregoing embodiments can also be performed for the preceding key frame and the following key frame. The action scene classification result for each personnel respectively predicted by the center key frame, the preceding key frame and the following key frame is determined based on the multiplicative fusion calculation of the confidence, so that the advantage of this processing is that the influence of individual key frame misclassification can be reduced. If the prediction probability of a key frame for a certain action scene category is very low, then the confidence after multiplicative fusion will also be very low. Only when the confidence of the classification prediction result in each key frame tends to be a high probability value, the fusion confidence obtained after multiplicative fusion calculation will exceed a preset threshold value, and at this time the final action scene classification will be more reliable, further reducing the error of action scene classification prediction.

[0119] Suppose that the confidence (prediction probability) of the gth action scene classification of the ith personnel for the center key frame can be represented as The confidence (prediction probability) of the gth action scene classification of the ith personnel for the preceding key frame can be represented as The confidence (prediction probability) of the gth action scene classification of the ith personnel for the following key frame can be represented as Then, the fusion confidence of the gth action scene classification of the ith personnel can be calculated as: .

[0120] In one embodiment, as shown in Figure 7 the step 160, the matching of the binary tuple set of the personal protective equipment associated with each personnel with the rule set of the personal protective equipment corresponding to the action scene classification result of each personnel to identify whether each personnel has a rule violation behavior can further include:

[0121] Step 610, according to the index order of each category of personal protective equipment, mapping the identification of whether each category of personal protective equipment contained in the rule set must be worn to a binary rule bit mask, mapping the identification of whether each category of personal protective equipment contained in the binary tuple set is detected to a binary wearing state bit mask of the same bit number, and performing an inverse operation on the wearing state bit mask to obtain the reverse bit mask corresponding to the wearing state bit mask;

[0122] Step 620, performing a logical AND operation on the rule bit mask and the reverse bit mask, identifying the binary bits that are 1 in the operation result, and thereby identifying the category of personal protective equipment involved in the violation behavior of each person.

[0123] In this embodiment, in order to reduce the consumption of computing resources and improve the processing performance of violation behavior identification, the identification of whether each category of personal protective equipment contained in the rule set must be worn can be mapped to a binary rule bit mask according to the index order of each category of personal protective equipment. Similarly, the identification of whether each category of personal protective equipment contained in the binary tuple set associated with each person is detected is also mapped to a binary wearing state bit mask of the same bit number, and an inverse operation is performed on the wearing state bit mask to obtain the reverse bit mask corresponding to the wearing state bit mask. Subsequently, a logical AND operation is performed on the rule bit mask and the reverse bit mask. If the binary bits that are 1 in the operation result are the bits corresponding to the personal protective equipment that is not worn in violation, the category of personal protective equipment involved in the violation behavior of each person is quickly identified by converting the set to a bit operation, and it is determined whether each person has a violation behavior of not wearing a certain category of personal protective device that does not comply with the rule set of personal protective equipment.

[0124] Figure 8 is a structural schematic diagram of a violation behavior detection device based on video analysis according to the first embodiment of the present application. As shown in Figure 8 , the device includes the following units:

[0125] The key frame acquisition unit 710 is configured to extract a 3D feature map containing continuous multiple frames from a real-time video stream through a video backbone network, and cut a center key frame and front and rear key frames located at a preset step before and after the center key frame from the 3D feature map.

[0126] The key frame feature extraction unit 720 is configured to extract a main feature matrix corresponding to the center key frame and reference feature matrices corresponding to the front and rear key frames, respectively, calculate a first attention map based on the main feature matrix and the reference feature matrices, and aggregate the main feature matrix based on the first attention map to obtain a key frame feature matrix.

[0127] The target detection unit 730 is configured to identify, by a target detection classifier, detection bounding boxes of personnel, key equipment, and personal protective equipment from the key frame feature matrix, to obtain a set of two-tuples of personal protective equipment associated with each piece of personnel;

[0128] The scene feature extraction unit 740 is configured to extract, from the central key frame, a personnel feature matrix corresponding to personnel and an association feature matrix between personnel and key equipment according to the detection bounding boxes of the personnel and the key equipment, to calculate a second attention map based on the personnel feature matrix and the association feature matrix, and to aggregate the personnel feature matrix based on the second attention map to obtain a scene feature matrix.

[0129] The scene classification unit 750 is configured to perform action scene classification on the scene feature matrix by an action scene classifier, and to output an action scene classification result of each piece of personnel.

[0130] The violation behavior identification unit 760 is configured to match the set of two-tuples of personal protective equipment associated with each piece of personnel with a set of rules of personal protective equipment corresponding to the action scene classification result of each piece of personnel, and to identify whether each piece of personnel has a violation behavior.

[0131] In one embodiment, as shown in FIG. 7, the key frame feature extraction unit 720 can further include: Figure 9

[0132] The feature splicing sub-unit 721 is configured to splice and fuse a preceding feature matrix extracted from a preceding key frame in the preceding and following key frames and a following feature matrix extracted from a following key frame in the preceding and following key frames to obtain the reference feature matrix.

[0133] The first calculation sub-unit 722 is configured to calculate a first linear transformation of the main feature matrix to obtain a query matrix, to calculate a second linear transformation of the reference feature matrix to obtain a key matrix, and to calculate the first attention map based on a point product of the query matrix and the key matrix.

[0134] The second calculation sub-unit 723 is configured to calculate a third linear transformation of a point product of the first attention map and the reference feature matrix, and to add the main feature matrix to obtain the key frame feature matrix.

[0135] In one embodiment, as shown in FIG. 7, the scene feature extraction unit 740 can include: Figure 10

[0136] ​​The association bounding box determination sub-unit 741 is configured to determine, according to the detection bounding boxes of the personnel and the key equipment, an association bounding box between each piece of personnel and each piece of key equipment from the center key frame, the association bounding box containing each piece of personnel and the each piece of key equipment.

[0137] The association feature extraction sub-unit 742 is configured to extract, from the center key frame, a one-dimensional feature vector corresponding to the association bounding box between each piece of personnel and each piece of key equipment, to obtain the association feature matrix between the personnel and the key equipment.

[0138] In one embodiment, as shown in FIG. 7, the scene feature extraction unit 740 can include: Figure 11

[0139] The feature transformation sub-unit 743 is configured to perform a dimension alignment operation on the personnel feature matrix, and copy and fill each row element in the row dimension, to obtain a personnel feature transformation matrix with the same dimension as the association feature matrix.

[0140] The third calculation sub-unit 744 is configured to perform a softmax function operation on a Hadamard product of the personnel feature transformation matrix and the association feature matrix, to obtain the second attention map.

[0141] The fourth calculation sub-unit 745 is configured to perform a fourth linear transformation on a dot product between the second attention map and the association feature matrix, and add the personnel feature matrix, to obtain the scene feature matrix.

[0142] In one embodiment, the scene feature extraction unit 740 is further configured to: according to the detection bounding boxes of the personnel and the key equipment, extract, from the front key frame and the rear key frame, respectively, a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment, calculate a second attention map based on the personnel feature matrix and the association feature matrix, and aggregate the personnel feature matrix based on the second attention map, to obtain a scene feature matrix.

[0143] The scene classification unit 750 is further configured to: for the scene feature matrix obtained for the front key frame and the rear key frame, perform action scene classification on the scene feature matrix and output an action scene classification result of each piece of personnel.

[0144] The scene classification unit 750 is further configured to: perform multiplicative fusion calculation on the confidence of the action scene classification result of each piece of personnel output by the center key frame, the front key frame and the rear key frame respectively, to obtain a final action scene classification and a fusion confidence of each piece of personnel.

[0145] In one embodiment, the violation behavior recognition unit 760 is further configured to: ​

[0146] According to the index order of each category of personal protective equipment, the identification of whether each category of personal protective equipment contained in the rule set must be worn is mapped to a binary rule bit mask, the identification of whether each category of personal protective equipment contained in the binary tuple set is detected is mapped to a binary wearing state bit mask of the same number of bits, and an inverse operation is performed on the wearing state bit mask to obtain an inverse bit mask corresponding to the wearing state bit mask;

[0147] The rule bit mask and the inverse bit mask are subjected to a logical AND operation, and binary bits that are 1 in the operation result are identified, thereby identifying the category of personal protective equipment involved in the violation behavior of each person.

[0148] The method and device for detecting violation behavior based on video analysis proposed in the embodiments of the present application, on the one hand, intercept the center key frame and the front and rear key frames located a preset step before and after the center key frame from the 3D feature map output by the video backbone network, extract the main feature matrix corresponding to the center key frame and the reference feature matrix corresponding to the front and rear key frames respectively, calculate a first attention map based on the main feature matrix and the reference feature matrix, and aggregate and enhance the main feature matrix based on the first attention map to obtain an enhanced key frame feature matrix, and identify the detection bounding box of personnel, key equipment and personal protective equipment and the binary tuple set of personal protective equipment associated with each person from the enhanced key frame feature matrix through a target detection classifier, so as to effectively improve the detection error of the target object in the image frame, especially the small-size target object such as personal protective equipment worn by personnel, and improve the accuracy of target detection, in view of the image distortion, non-uniform resolution and complex background of the original video captured by the wide-angle camera, and other problems such as occlusion and blur between objects in the original frame.

[0149] On the other hand, the personnel feature matrix corresponding to the personnel and the associated feature matrix between the personnel and the key equipment are extracted from the center key frame according to the detection bounding box of the personnel and the key equipment, a second attention map is calculated based on the personnel feature matrix and the associated feature matrix, the personnel feature matrix is aggregated based on the second attention map to obtain a scene feature matrix, and the scene feature matrix is classified by an action scene classifier to output the action scene classification result of each person, so that the specific action scene of the personnel in the factory environment can be quickly and accurately identified in view of the image distortion, non-uniform resolution and complex background of the original video captured by the wide-angle camera, and other problems such as occlusion and blur between objects in the original frame, and then the violation behavior of each person in the specific action scene can be effectively identified.

[0150] In addition, the lightweight model used in the embodiments of the present application can effectively reduce the consumption of hardware computing power and computing resources, and can be deployed in edge computing devices in a factory environment to realize real-time visual safety behavior monitoring in a complex factory environment.

[0151] It should be noted that the different embodiments described by the method embodiments of the present application and their explanations and achieved technical effects are also applicable to the device embodiments of the present application, which will not be described here.

[0152] Further, the embodiments of the present application also propose an edge computing device, which can include a processor and a memory. The memory stores computer program instructions, and the processor can invoke the computer program instructions in the memory to execute all or part of the steps of the method described in any embodiment of the present application. The computer program instructions in the memory described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. The edge computing device can be deployed in a complex factory environment to realize real-time visual safety behavior monitoring in a complex factory environment.

[0153] Further, the embodiments of the present application also propose an electronic device, which can include a processor and a memory. The memory stores computer program instructions, and the processor can invoke the computer program instructions in the memory to execute all or part of the steps of the method described in any embodiment of the present application. The computer program instructions in the memory described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium.

[0154] Further, the present application also provides a computer program product, which includes a non-transitory computer readable storage medium storing a computer program. When the computer readable storage medium is connected to a computer device, the computer program is executed by one or more processors of the computer device, and the computer program can execute all or part of the steps of the method described in any embodiment of the present application.

[0155] Further, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, which can be executed by one or more processors to execute all or part of the steps of the method described in any embodiment of the present application.

[0156] Those skilled in the art can clearly understand the implementation of the embodiments of the present application by the description of the above embodiments, which can be realized by software or software combined with necessary general hardware platform, and of course can also be realized by hardware function. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art, which is stored in a storage medium, and includes a plurality of instructions for causing a computer device, such as but not limited to a personal computer, a server, or a network device, to perform all or part of the steps of the method described in any embodiment of the present application. The foregoing storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store computer program codes.

[0157] The above describes the exemplary embodiments of the present application, and it should be understood that the above exemplary embodiments are not restrictive but illustrative, and the protection scope of the present application is not limited thereto. It should be understood that those skilled in the art can modify and change the embodiments of the present application without departing from the spirit and scope of the present application, and these modifications and changes should be within the protection scope of the present application.

Claims

1. A method for detecting a violation based on video analysis, characterized in that, The method comprises: extracting a 3D feature map containing continuous multiple frames from a real-time collected video stream through a video backbone network, and extracting a center key frame from the 3D feature map at a certain time interval, and extracting a front key frame and a rear key frame located at a preset step length before and after the center key frame respectively; extracting a main feature matrix corresponding to the center key frame and a reference feature matrix corresponding to the front and rear key frames respectively, calculating a first attention map based on the main feature matrix and the reference feature matrix, and aggregating the main feature matrix based on the first attention map to obtain a key frame feature matrix; identifying detection bounding boxes of personnel, key equipment and personal protective equipment from the key frame feature matrix through a target detection classifier to obtain a binary tuple set of personal protective equipment associated with each personnel; extracting a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, calculating a second attention map based on the personnel feature matrix and the association feature matrix, and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix; performing action scene classification on the scene feature matrix through an action scene classifier to output an action scene classification result of each personnel; matching the binary tuple set of personal protective equipment associated with each personnel with a rule set of personal protective equipment corresponding to the action scene classification result of each personnel to identify whether each personnel has a violation behavior.

2. The method of claim 1, wherein, The method comprises: splicing and fusing a front feature matrix extracted from a front key frame in the front and rear key frames and a rear feature matrix extracted from a rear key frame in the front and rear key frames to obtain the reference feature matrix; calculating a first linear transformation of the main feature matrix to obtain a query matrix, calculating a second linear transformation of the reference feature matrix to obtain a key matrix, and calculating the first attention map based on a point product softmax function operation of the query matrix and the key matrix; calculating a third linear transformation of a point product of the first attention map and the reference feature matrix, and adding the main feature matrix to obtain the key frame feature matrix.

3. The method of rule violation detection based on video analysis according to claim 2, wherein, The method comprises: determining an association bounding box between each personnel and each key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, wherein the association bounding box at least contains each personnel and each key equipment; extracting a one-dimensional feature vector corresponding to the association bounding box between each personnel and each key equipment from the center key frame to obtain the association feature matrix between the personnel and the key equipment.

4. The method of rule violation detection based on video analysis according to claim 3, wherein, The method involves determining the associated bounding box between each person and each key device from the central keyframe based on the detected bounding boxes of the personnel and key devices. The associated bounding box at a minimum includes each person and each key device, including: The associated bounding box between each person and each key device is determined based on the minimum value of the upper left corner coordinate and the maximum value of the lower right corner coordinate of the detection bounding box of each person and each key device.

5. The method of video analytics based violation detection as claimed in claim 4, wherein, The step of extracting the personnel feature matrix and the association feature matrix between personnel and key equipment from the central keyframe based on the detection bounding boxes of the personnel and key equipment includes: Calculate the distance between the centers of the detection bounding boxes of each person and each key device, and determine the associated bounding boxes between each person and each key device when the distance between the centers of the detection bounding boxes of each person and each key device is less than or equal to a preset distance threshold from the center key frame.

6. The method of rule violation detection based on video analysis according to claim 5, wherein, The step of calculating a second attention map based on the personnel feature matrix and the associated feature matrix, and then aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix, includes: Perform a dimension alignment operation on the personnel feature matrix, and copy and fill each row element according to the row dimension to obtain a personnel feature transformation matrix with the same dimension as the associated feature matrix; The second attention map is obtained by performing a softmax function operation on the Hadamard product of the personnel feature transformation matrix and the associated feature matrix; Calculate the fourth linear transformation of the dot product between the second attention map and the associated feature matrix, and add it to the personnel feature matrix to obtain the scene feature matrix.

7. The method of video analytics based misbehavior detection according to claim 6, characterized in that, The method further includes: For the preceding and following keyframes, the steps of extracting the personnel feature matrix corresponding to the personnel and the association feature matrix between the personnel and key equipment are executed respectively; calculating a second attention map based on the personnel feature matrix and the association feature matrix; and aggregating the personnel feature matrix based on the second attention map to obtain a scene feature matrix; and classifying the scene feature matrix into action scenes using an action scene classifier and outputting the action scene classification result for each personnel. The confidence scores of the action scene classification results for each person output from the central keyframe and the preceding and following keyframes are multiplied and fused to obtain the final action scene classification and fusion confidence score for each person.

8. The method of video analytics based misbehavior detection according to claim 7, wherein, The step of matching the set of binary pairs of personal protective equipment associated with each person with the set of rules for personal protective equipment corresponding to the action scenario classification results of each person to identify whether each person has violated regulations includes: Based on the index order of each category of personal protective equipment, the identifier of whether each category of personal protective equipment in the rule set must be worn is mapped to a binary rule bitmask. The identifier of whether each category of personal protective equipment in the tuple set is detected is mapped to a binary wearing status bitmask of the same number of bits. The wearing status bitmask is then inverted to obtain the inverted bitmask corresponding to the wearing status bitmask. The rule bit mask and the reverse bit mask are logically ANDed to identify binary bits that are 1 in the result, thereby identifying the category of personal protective equipment involved in the rule violation of each person.

9. A video analysis-based misbehavior detection apparatus, characterized by, The device comprises: a key frame acquisition unit configured to extract a 3D feature map containing continuous multiple frames from a real-time video stream through a video backbone network, to cut a center key frame from the 3D feature map at a certain time interval, and to extract a front key frame and a rear key frame located at a preset step length before and after the center key frame respectively according to the preset step length; a key frame feature extraction unit configured to extract a main feature matrix corresponding to the center key frame and reference feature matrices corresponding to the front key frame and the rear key frame respectively, to calculate a first attention map based on the main feature matrix and the reference feature matrices, and to aggregate the main feature matrix based on the first attention map to obtain a key frame feature matrix; a target detection unit configured to identify detection bounding boxes of personnel, key equipment and personal protective equipment from the key frame feature matrix through a target detection classifier to obtain a binary tuple set of personal protective equipment associated with each person; a scene feature extraction unit configured to extract a personnel feature matrix corresponding to the personnel and an association feature matrix between the personnel and the key equipment from the center key frame according to the detection bounding boxes of the personnel and the key equipment, to calculate a second attention map based on the personnel feature matrix and the association feature matrix, and to aggregate the personnel feature matrix based on the second attention map to obtain a scene feature matrix; a scene classification unit configured to perform action scene classification on the scene feature matrix through an action scene classifier to output an action scene classification result of each person; a rule violation identification unit configured to match the binary tuple set of personal protective equipment associated with each person with a rule set of personal protective equipment corresponding to the action scene classification result of each person to identify whether each person has a rule violation.

10. An edge computing device, comprising: comprise: one or more processors; a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the steps of the method of any one of claims 1 to 8.

Images