Multimodal based global-local perception misbehavior identification method and system

Abstract

The application discloses a global-local perception violation behavior recognition method and system based on multi-modal, and relates to the field of computer vision.The method comprises the following steps: obtaining to-be-recognized video data; the to-be-recognized video data comprises to-be-recognized oil well operation video and corresponding human body video frames; obtaining mask features and action features corresponding to the human body video frames; obtaining multi-scale global features corresponding to the to-be-recognized oil well operation video; performing data alignment on the mask features and the action features to obtain aligned mask features and action features; obtaining local fusion features; obtaining global fusion features; and performing recognition on the global fusion features to obtain a prediction result of a violation behavior category.The application can comprehensively represent content and semantic information in a video, enhance the expression ability of features, provide more abundant information for the recognition of violation behaviors, and significantly improve the accuracy and efficiency of the recognition of violation behaviors, thereby providing strong support for behavior analysis in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a method and system for identifying violations based on multimodal global-local perception. Background Technology

[0002] In the petroleum industry's safety production management, the identification of violations, monitoring of personnel movement compliance, and multi-source data fusion analysis are core technical requirements for ensuring operational safety. In drilling operations, human motion recognition is one of the important means of detecting violations. Human motion recognition technology can accurately determine whether there are any dangerous or unsafe operations by analyzing the movement trajectories, posture changes, and behavioral sequences of workers.

[0003] Existing human motion recognition methods mainly include traditional feature-based methods and deep learning-based methods. However, existing methods cannot simultaneously consider both global and local information. Global information includes the spatial relationships between objects and between people and objects in the global scene, while local information focuses on the relationship between a person and surrounding objects. However, in drilling scenarios, workers are often in harsh environments such as high noise, low light, and confined spaces. Single video data is easily affected by occlusion and changes in lighting. Relying solely on single-modal data (such as video or images) for violation detection often suffers from insufficient information and significant environmental interference, making it difficult to achieve high accuracy in complex and ever-changing drilling environments. Traditional methods typically rely on manually extracted features, such as HOG (Histogram of Oriented Gradients), optical flow features, and pose estimation, which are insufficient to cope with the complex and ever-changing drilling environment. Relying on manual inspection or single-sensor monitoring results in low efficiency, high false alarm rates, and poor adaptability to complex scenarios. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, the present invention provides a multimodal global-local perception violation recognition method and system that solves the problems of insufficient recognition accuracy and recognition results being limited by single-modal data.

[0005] To achieve the aforementioned objectives, the technical solution adopted by this invention is: a multimodal global-local perception violation recognition method, comprising:

[0006] Acquire the video data to be identified; the video data to be identified includes the oil well operation video to be identified and its corresponding human video frames;

[0007] Obtain the mask features and motion features corresponding to human video frames;

[0008] Obtain multi-scale global features corresponding to the oil well operation video to be identified;

[0009] align the mask feature and the action feature to obtain an aligned mask feature and an aligned action feature;

[0010] fuse the aligned mask feature and the aligned action feature by using a fusion architecture based on a Transformer to obtain a local fusion feature;

[0011] fuse the local fusion feature and a multi-scale global feature by using a global interaction perception architecture based on an attention mechanism to obtain a global fusion feature;

[0012] identify the global fusion feature to obtain a prediction result of a rule violation behavior category.

[0013] The application further provides a global-local perception rule violation behavior identification system based on multi-modalities, comprising:

[0014] a data loading module configured to obtain to-be-identified video data;

[0015] a mask feature extraction module configured to obtain a mask feature corresponding to the to-be-identified video data; the mask feature extraction module comprises a segmentation large model SAM and a time sequence feature extraction network;

[0016] an action feature extraction module configured to obtain an action feature corresponding to the to-be-identified video data; the action feature extraction module comprises a human body action detection network FiDiP and a novel spatio-temporal graph convolution network ST-GCN;

[0017] a multi-scale global feature extraction module configured to obtain a multi-scale global feature corresponding to the to-be-identified video data; the multi-scale global feature extraction module comprises a 3D-CNN network;

[0018] a feature consistency constraint module configured to align the mask feature and the action feature;

[0019] a local feature fusion module configured to fuse the aligned mask feature and the aligned action feature by using a fusion architecture based on a Transformer to obtain a local fusion feature;

[0020] a global interaction module and an inference module configured to fuse the local fusion feature and a multi-scale global feature by using a global interaction perception architecture based on an attention mechanism to obtain a global fusion feature; and identify the global fusion feature to obtain a prediction result of a rule violation behavior category.

[0021] The application has the following beneficial effects:

[0022] 1. By acquiring multi-scale global features of video frames, the spatial layout information in the video and the association information between the human body and the environment are represented, which can comprehensively represent the content and semantic information in the image, consider different features of the human body at different scales, enhance the expression ability of the features, provide more abundant information for the identification of illegal behavior, and thus significantly improve the accuracy and efficiency of the identification of illegal behavior, and provide strong support for behavior analysis in complex scenes.

[0023] 2. By designing a time sequence feature extraction network, the mask frames segmented from the video frames are modeled in time and space, and by focusing on the mutual relationship between different time points and spatial positions, the complex time and space dependence in the mask sequence can be captured, thereby further enhancing the expression ability of the features and making the identification of illegal behavior more accurate.

[0024] 3. By multi-modal fusion of mask features and action features to obtain local fusion features, the diversity and complexity of the data are increased, which helps the illegal behavior recognition system to learn more general feature representation, so as to make more accurate and reasonable logical reasoning for the illegal behavior, thereby improving the generalization ability, the recognition efficiency and the accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 a flowchart of the method of the present application;

[0026] Figure 2 a structure diagram of the time sequence feature extraction network;

[0027] Figure 3 a structure diagram of the local feature fusion module;

[0028] Figure 4 a structure diagram of the global interaction perception architecture based on the attention mechanism;

[0029] Figure 5 a flowchart of the identification process of the illegal behavior in the embodiment. DETAILED DESCRIPTION

[0030] The specific embodiments of the present application are described below to facilitate understanding of the present application by those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all applications utilizing the concept of the present application are within the scope of protection.

[0031] As Figure 1 , Figure 5As shown, in one embodiment of the present invention, a multimodal global-local perception violation recognition method includes:

[0032] S1. Obtain the video data to be identified using the data loading module. This embodiment does not limit the specific method of acquiring the video; the key is to ensure that the video content accurately reflects various violations in oil well operations. The video data to be identified includes the oil well operation video and human video frames. In this embodiment, the pre-trained human detection network YOLOv7 is used to detect people in each frame of the oil well operation video, thus obtaining the human video frames. .

[0033] S2. Obtain the mask features corresponding to the human video frames using the mask feature extraction module. Specifically, input the human video frames into the segmentation large model (SAM) to obtain the mask video frames. Combine the same mask in the masked video frames into a single mask frame to obtain multiple sets of mask frames. The Temporal Feature Extraction Network (TFNet) inputs multiple sets of masked frames as multiple features to model the spatiotemporal relationships of each masked frame segmented from the video, ultimately obtaining a set of masked features with spatiotemporal relationship information. ,in i Indicates the first i A mask, Indicates the first i This method generates high-dimensional feature representations containing rich spatiotemporal information. These mask feature representations not only cover the static features of objects in the video, but also the dynamic features of objects changing over time and their interactions.

[0034] like Figure 2As shown, the time sequence feature extraction network (TFNet) includes an input convolutional layer, a first one-dimensional convolutional neural network layer, a first time block (Time Block), a second time block, a third time block, a second one-dimensional convolutional neural network layer, and an output convolutional layer connected in sequence. The TFNet is mainly composed of a 1D time CNN layer and a Time Block, and a 1D time CNN is added on the basis of the input conv layer, aiming to capture local features in the time dimension. The convolution kernel sliding on the time axis is used to preliminarily encode the input mask sequence, and the basic feature representation in the time dimension is extracted. Then, the TFNet further models the preliminary extracted basic features through three layers of Time Block composed of 3D CNN and cross-attention (Cross-Attention) mechanism. The 3D CNN layer simultaneously performs convolution operation in the time and space dimensions, which can capture the complex spatio-temporal dependence relationship in the mask sequence, and the cross-attention mechanism further enhances the expression ability of the features, which captures long distances by focusing on the mutual relationship between different time points and spatial positions.

[0035] The action feature extraction module is used to obtain the action features corresponding to the human video frame. The specific method is as follows: inputting the human video frame into the human action detection network FiDiP to obtain a human pose node graph; inputting the human pose node graph into the action feature extraction network ST-GCN to obtain the action feature The action feature extraction module is used to obtain the action features corresponding to the human video frame. The specific method is as follows: inputting the human video frame into the human action detection network FiDiP to obtain a human pose node graph; inputting the human pose node graph into the action feature extraction network ST-GCN to obtain the action feature , The action feature extraction module is used to obtain the action features corresponding to the human video frame. The specific method is as follows: inputting the human video frame into the human action detection network FiDiP to obtain a human pose node graph; inputting the human pose node graph into the action feature extraction network ST-GCN to obtain the action feature i The action feature extraction module is used to obtain the action features corresponding to the human video frame. The specific method is as follows: inputting the human video frame into the human action detection network FiDiP to obtain a human pose node graph; inputting the human pose node graph into the action feature extraction network ST-GCN to obtain the action feature The action feature extraction module is used to obtain the action features corresponding to the human video frame. The specific method is as follows: inputting the human video frame into the human action detection network FiDiP to obtain a human pose node graph; inputting the human pose node graph into the action feature extraction network ST-GCN to obtain the action feature

[0036]

[0037]

[0038] wherein, represents normalization.

[0039] S3, the multi-scale global feature extraction module is used to input the oil well operation video into the 3D-CNN network, and the convolution operation is performed in the time and space dimensions to capture the complex spatio-temporal dependence relationship in the mask sequence, thereby obtaining the multi-scale global feature The multi-scale global feature represents the spatial layout information and the context information between instances and the environment in the video, enhances the feature expression capability, and significantly improves the accuracy and efficiency of the violation behavior recognition, and provides strong support for behavior analysis in complex scenes.

[0040] Segmentation mask feature And action feature Respectively from different modalities, each in an independent feature space. Due to the distance between these feature spaces, the correlation between features is difficult to capture directly, which brings difficulties to subsequent fusion and reasoning. Directly operating on these features not only increases the complexity of fusion, but also may affect the accuracy and efficiency of reasoning. Therefore, in order to realize effective feature fusion and reasoning, measures need to be taken to narrow the gap between feature spaces.

[0041] S4, using a feature consistency constraint module to align the mask feature and the action feature, this feature alignment mechanism can effectively solve the scale difference problem in multi-modal feature fusion, improve the accuracy and efficiency of feature fusion, and realize accurate identification and real-time response of violation behavior. The alignment loss function is used to constrain the consistency of the two in the feature space, and the specific formula is as follows:

[0042]

[0043] Among them, is the alignment loss value, represents and The cosine similarity between them.

[0044] By minimizing the alignment loss function, the feature consistency constraint module is used to unify the two modal features into the same feature space, and the consistency of different modal features in semantics and scale is enhanced, thereby providing more effective feature representation for subsequent fusion and reasoning.

[0045] S5, using the local feature fusion module based on the fusion architecture of the Transformer to fuse the aligned mask feature and action feature, to obtain the local fusion feature, as shown in Figure 3 The specific method is:

[0046] Initialize the query vector, and its expression is:

[0047]

[0048] Among them is the query vector, represents the corresponding processing of the multi-layer perception, represents the connection, represents the self-attention mechanism, to the category-aware embedding vector;

[0049] The local fusion feature is obtained by a cross-attention mechanism, and its expression is:

[0050]

[0051] wherein, represents the local fusion feature, represents the cross-attention mechanism.

[0052] S6, the local fusion feature cannot pay attention to the global information, i.e. the relationship between people and objects in the video, and objects, so the embodiment utilizes the global interaction perception architecture based on the attention mechanism in the global interaction module and the reasoning module to fuse the local fusion feature and the multi-scale global feature to obtain the global fusion feature. The structure of the global interaction perception architecture based on the attention mechanism is shown in Figure 4 It contains three layers of global interaction fusion modules (Global Interaction Block) connected in turn and the same, and a multi-layer perception (MLP).

[0053] The specific method for obtaining the global fusion feature is:

[0054] The three layers of global interaction fusion modules (Global Interaction Block) connected in turn and the same are used to fuse respectively, and its expression is:

[0055]

[0056]

[0057]

[0058] wherein, is the first layer fusion feature, is the second layer fusion feature, is the third layer fusion feature, represents the global interaction fusion processing;

[0059] The third layer fusion feature is input into the multi-layer perception to obtain the global fusion feature, and its expression is:

[0060]

[0061] wherein, is the global fusion feature, represents the corresponding processing of the multi-layer perception;

[0062] The fusion process of the first layer global interaction fusion module is:

[0063] The unique local fusion features are strengthened by a self-attention mechanism to obtain strengthened local fusion features , and the expression is

[0064]

[0065] wherein, is a function, represents the dimension, and the superscript T represents transposition;

[0066] The global features are fused with the local features by a cross-attention mechanism to obtain features , and the expression is

[0067]

[0068] wherein, represents an attention mechanism;

[0069] The global features are fused with the local features twice by a cross-attention mechanism to obtain first-layer fusion features , and the expression is

[0070]

[0071] The first-layer fusion features are represented as:

[0072] .

[0073] The fusion processing procedures of the second-layer global interaction fusion module and the third-layer global interaction fusion module are the same as those of the first-layer global interaction fusion module.

[0074] S7, after completing the global interaction feature fusion, the global fusion features are input into a multi-layer perception (MLP) in a global interaction module and an inference module for nonlinear transformation to obtain a violation behavior prediction result. The global fusion features are identified to obtain a prediction result of a violation behavior category. The prediction result includes “not wearing a safety helmet”, “smoking in the workplace”, “not wearing safety glasses as required”, and the like.

[0075] The implementation of the above steps depends on a global-local perception violation behavior recognition system based on multiple modalities provided by the application, which is a system obtained after training. During the training process, a cross-entropy loss function is used, and the expression is

[0076]

[0077] wherein, is a cross-entropy loss value, is a number of violation behavior categories, i denotes a violation behavior category index, is a true value, is a predicted value, denotes a natural logarithm.

[0078] The present application comprehensively represents the content and semantic information in the image by fusing multi-modal and multi-scale features, comprehensively considers different features of the human body at different scales, enhances the expression ability of the features, provides more abundant information for the identification of the violation behavior, and thus significantly improves the precision and efficiency of the violation behavior identification, and provides strong support for the behavior analysis in a complex scene.

Claims

1. A multimodal based global-local perception violation behavior recognition method, characterized in that, The method comprises the following steps: obtaining to-be-recognized video data; the to-be-recognized video data comprises to-be-recognized oil well operation video and corresponding human body video frames; obtaining mask features and action features corresponding to the human body video frames; obtaining multi-scale global features corresponding to the to-be-recognized oil well operation video; aligning the mask features and the action features to obtain aligned mask features and action features; fusing the aligned mask features and the action features by using a fusion architecture based on a Transformer to obtain local fusion features; fusing the local fusion features and the multi-scale global features by using a global interaction perception architecture based on an attention mechanism to obtain global fusion features; recognizing the global fusion features to obtain a prediction result of a type of violation behavior; wherein the specific method for obtaining the mask features corresponding to the human body video frames comprises the following steps: inputting the human body video frames into a segmentation large model SAM to obtain mask video frames; combining the same mask in the mask video frames into a mask frame to obtain multiple groups of mask frames; inputting the multiple groups of mask frames into a time sequence feature extraction network as multiple feature inputs to obtain mask features; wherein the time sequence feature extraction network comprises an input convolutional layer, a first one-dimensional convolutional neural network layer, a first time block, a second time block, a third time block, a second one-dimensional convolutional neural network layer and an output convolutional layer connected in sequence; the time block is composed of a 3D CNN and a cross-attention mechanism; the specific method for obtaining the action features corresponding to the human body video frames comprises the following steps: inputting the human body video frames into a human body action detection network FiDiP to obtain a human body pose node graph; inputting the human body pose node graph into a novel spatio-temporal graph convolution network ST-GCN to analyze the spatial and temporal relationships in the action node graph to obtain action features.

2. The method of claim 1, wherein the method is a multi-modal based global-local perception violation identification method. The alignment of the mask features and the action features is realized by using a minimum alignment loss function, and the expression of the alignment loss function is: wherein, is an alignment loss value, denotes a cosine similarity between and denotes an action feature, denotes a mask feature.

3. The method of claim 2, wherein the method is a multi-modal based global-local perception violation identification method. the specific method for fusing the aligned mask features and the action features by using a fusion architecture based on a Transformer to obtain local fusion features comprises the following steps: initializing a query vector, and the expression of the query vector is: wherein is a query vector, denotes a corresponding processing of a multi-layer perceptron, denotes a concatenation, denotes a self-attention mechanism, is a class-aware embedding vector; obtaining the local fusion features by using a cross-attention mechanism, and the expression of the local fusion features is: wherein, denotes a local fusion feature, denotes a cross-attention mechanism.

4. The multi-modal based global-local perception violation behavior identification method of claim 3, wherein, the multi-scale global features comprise three scales, and the expression of the multi-scale global features is: wherein, is a multi-scale global feature, are global features at different scales, respectively.

5. The multi-modal based global-local perception violation behavior identification method of claim 4, wherein, the specific method for fusing the local fusion features and the multi-scale global features by using a global interaction perception architecture based on an attention mechanism to obtain global fusion features comprises the following steps: Through three interconnected and identical global interaction fusion modules, respectively... The expression for fusion is: wherein, is a first layer fusion feature, is a second layer fusion feature, is a third layer fusion feature, denotes a global interaction fusion process; inputting the third layer fusion features into a multi-layer perception machine to obtain the global fusion features, and the expression of the global fusion features is: wherein, is a global fusion feature, denotes a corresponding processing of a multi-layer perceptron; the fusion process of the first layer global interaction fusion module is: The unique property of the local fusion feature is strengthened by a self-attention mechanism to obtain a strengthened local fusion feature The expression is: wherein is a function, denotes dimension, superscript T denotes the transpose; The global feature is fused with the local feature through a cross attention mechanism to obtain a feature with an expression as follows: ​​ wherein, denotes the attention mechanism; Global features are combined using a cross-attention mechanism. and A second fusion is performed to obtain the first layer of fusion features. Its expression is: fusing the first layer of features is represented as: 。 6. The multi-modal based global-local perception violation behavior identification method of claim 5, wherein, the specific method for recognizing the global fusion features to obtain a prediction result of a type of violation behavior comprises the following steps: The global fusion features are input into a multi-layer perception machine for nonlinear transformation to obtain a violation behavior prediction result.

7. A system based on any one of claims 1-6, wherein the system is a multi-modal based global-local perception violation behavior recognition method. The method comprises the following steps: a data loading module configured to obtain to-be-recognized video data; a mask feature extraction module configured to obtain mask features corresponding to the to-be-recognized video data; comprising a segmentation large model SAM and a time sequence feature extraction network; an action feature extraction module configured to obtain action features corresponding to the to-be-recognized video data; comprising a human body action detection network FiDiP and a novel spatio-temporal graph convolution network ST-GCN; The multi-scale global feature extraction module is configured to obtain multi-scale global features corresponding to the video data to be recognized; The 3D-CNN network is included; The feature consistency constraint module is configured to perform data alignment on the mask features and the action features; The local feature fusion module is configured to fuse the aligned mask features and the action features by using a fusion architecture based on a Transformer to obtain local fusion features; The global interaction module and the reasoning module are configured to fuse the local fusion features and the multi-scale global features by using a global interaction perception architecture based on an attention mechanism to obtain global fusion features; The global fusion features are recognized to obtain a prediction result of the type of the violation behavior.

8. The system of claim 7, wherein, The system is a system obtained after training, and a cross-entropy loss function is used in the training process, and the expression is: wherein, is a cross-entropy loss value, is a number of violation behavior categories, i denotes a violation behavior category index, is a true value, is a predicted value, denotes a natural logarithm.

Images