Attention-based multi-modal well site video safety analysis method

Abstract

The invention discloses a multi-mode well site video safety analysis method based on attention, and relates to the field of intelligent video monitoring analysis, and the method comprises the steps: obtaining a low-level fusion feature corresponding to a to-be-analyzed well site operation video; obtaining video modal global spatial-temporal features corresponding to the low-level fusion features; acquiring pixel-level fine-grained local features of a single-frame image mode; inputting the global spatial-temporal features of the video modality and the pixel-level fine-grained local features of the single-frame image modality into a multi-modal feature progressive fusion module to obtain multi-modal fusion features; and inputting the multi-modal fusion features into a multi-task decoder module based on the multi-modal features to obtain a safety analysis result of the well site operation video. According to the invention, the safety analysis of the well site video can be accurately and effectively realized, and the accuracy and real-time performance of safety monitoring are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent video monitoring and analysis, and in particular to an attention-based multi-modal well site video safety analysis method. Background Art

[0002] As an important place for oil production, oil well sites are particularly important for safety management and efficient operation. However, the operating environment of oil well sites is complex and changeable, involving a large number of mechanical equipment, flammable and explosive materials, and high-risk operating processes, with many safety hazards. Traditional oil well site safety management methods mainly rely on manual inspections and monitoring, which have problems such as management blind spots and slow response speed, and are difficult to meet the safety management needs of modern oil well sites.

[0003] In recent years, with the rapid development of the Internet of Things (IoT), artificial intelligence (AI) and big data technology, intelligent oil well scene video monitoring systems have emerged. The system deploys high-definition cameras and edge computing devices at the oil well site, combined with advanced video image analysis algorithms, to identify and analyze violations and hidden dangers, and achieve all-round, all-weather monitoring of the oil well site. The existing well site video image safety analysis algorithms are mainly based on two aspects: based on target recognition and based on behavior and posture recognition. Methods based on target recognition, such as the YOLO algorithm, identify and locate workers and other key targets in the video based on single-frame images of the well site monitoring video, and analyze the current safety hazards based on static information such as the existence and location of the target. Methods based on behavior and posture recognition, such as the temporal convolution algorithm, use tracking and positioning technology in multiple frames of images to extract rich dynamic information from workers in the video and identify dangerous actions and behaviors of workers.

[0004] However, the existing safety analysis algorithms have many problems and are difficult to meet the needs of the complex environment of the actual well site. The target recognition-based method only uses a single frame image, so the algorithm cannot recognize dynamic information and cannot identify dangerous actions and behaviors. In addition, due to the information limitations brought by the single frame image, the algorithm has a high probability of misjudgment, which will lead to the neglect of safety hazards and excessive alarms. Based on limited behaviors, the probability of misjudgment is high, and due to the singleness of the algorithm task, the safety hazards it focuses on are relatively limited, which cannot meet the needs of overall well site safety analysis. In addition, due to the complex well site environment, the above algorithms analyze the overall image, which easily leads to the omission of monitoring of small targets and small actions. The posture recognition method only focuses on the dynamic information of people and cannot judge the safety hazards of the environment. In addition, due to the need to process multiple frames, the algorithm has an inherent reaction time, which will cause a certain delay when facing sudden safety hazards. The above two types of algorithms do not make full use of the static information of a single frame of the video and the dynamic information of multiple frames to extract information. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides an attention-based multimodal well site video safety analysis method, which solves the problems of insufficient accuracy and low real-time performance of safety monitoring in the prior art.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a multimodal well site video safety analysis method based on attention, comprising the following steps: S1, sending the frame sequence in the well site operation video to be analyzed into the pre-trained image encoder to obtain the low-level fusion features of the video frame; S2. Input the obtained low-level fusion features into the video global feature extraction module based on the spatiotemporal attention mechanism to obtain the global spatiotemporal features of the video modality, namely, the temporal information and spatial information; S3, inputting the key frames in the frame sequence described in S1 into a feature extraction module based on multi-scale spatial attention to obtain pixel-level fine-grained local features of a single-frame image modality; S4, inputting the extracted global spatiotemporal features of the video modality and the pixel-level fine-grained local features of the single-frame image modality into the multimodal feature progressive fusion module for deep fusion to obtain multimodal fusion features; S5, inputting the multimodal fusion features into a multi-task decoder module based on multimodal features to obtain safety analysis results of the well site operation video, including safety hazard analysis results of three categories: workers themselves, workers and environment, and environment; The security analysis model is composed of an image encoder, a video global feature extraction module based on spatiotemporal attention mechanism, a multimodal feature progressive fusion module, a multi-task decoder module based on multimodal features, and a feature extraction module based on multi-scale spatial attention.

[0007] Further: the image encoder includes a pre-trained visual encoder based on SAM and a pre-trained visual encoder based on CLIP; the well site operation video frame to be analyzed is used as the input of the image encoder, and is input into the pre-trained visual encoder based on SAM and the pre-trained visual encoder based on CLIP respectively, to obtain two types of features about the well site operation video; and then the obtained two types of features are feature fused to obtain low-level fused features of all video frames, and the expression is:

[0008] in is the input of the image encoder; is the function representation of the CLIP visual encoder, are the parameters of the CLIP visual encoder, is the output of the CLIP visual encoder; is the function representation of the SAM visual encoder, are the parameters of the SAM visual encoder, is the output of the SAM visual encoder; H It is the low-level fusion feature of all video frames obtained in the end ， is the weight corresponding to the SAM visual encoder, yes CL The weights corresponding to the IP visual encoder.

[0009] Further: the video global feature extraction module based on the spatiotemporal attention mechanism includes a small-core deep convolution layer, a multi-head temporal attention mechanism unit, an expanded deep convolution layer, and a 1×1 convolution layer connected in sequence; the video global feature extraction module based on the spatiotemporal attention mechanism also includes a first temporal self-attention mechanism unit, a first average pooling layer, and a first fully connected layer connected in sequence; the low-level fusion features obtained in step S1 are respectively used as inputs of the small-core deep convolution unit and the first temporal self-attention mechanism unit; the output of the 1×1 convolution layer is subjected to a Kronecker product operation with the output of the first fully connected layer, and then subjected to a Hadamard product operation with the low-level fusion features obtained in step S1 to obtain the global spatiotemporal features of the video modality, and the calculation expression thereof is:

[0010] in, represents the output of the 1×1 convolutional layer, represents a 1×1 convolution, represents a depthwise convolution with dilation, represents multi-headed temporal attention processing, represents a small kernel depth convolution, is the low-level fusion feature obtained in step S1; represents the output of the first fully connected layer, Indicates full connection. represents global average pooling, represents temporal self-attention processing; is the output of the video global feature extraction module based on the spatiotemporal attention mechanism. represents the Kronecker product, represents the Hadamard product.

[0011] Further: in step S3, the feature extraction module based on multi-scale spatial attention performs feature extraction on the key frame image at five scales and uses fast normalization fusion to obtain multi-scale spatial attention features; the five scales are the original image scale of the key frame image, half of the original image scale, one quarter of the original image scale, one eighth of the original image scale, and one sixteenth of the original image scale.

[0012] Further: the feature extraction module based on multi-scale spatial attention includes a VIT unit, a 2D convolution unit, a hole convolution layer, a multi-head cross attention mechanism unit, a self-attention mechanism unit and a fast normalized feature fusion unit connected in sequence; the key frame image is the input of the feature extraction module based on multi-scale spatial attention, and the output of the fast normalized feature fusion unit is the pixel-level fine-grained local feature of the single-frame image modality; its calculation expression is:

[0013]

[0014]

[0015] in, For the i Key frame images at different scales, express deal with, represents 2D convolution, represents the dilated convolution, Indicates i Preliminary features extracted from keyframe images at different scales; represents multi-head cross-attention processing, represents self-attention processing, Represents the output of the self-attention mechanism unit; represents the output of the fast normalized feature fusion unit, express The corresponding weights, Represents the self-attention mechanism unit j The learning weights corresponding to the outputs are used to Perform normalization processing; Additional parameters to ensure the stability of the results.

[0016] Furthermore: the multimodal feature progressive fusion module includes a first cross-attention mechanism unit, a multi-layer perceptron, a second average pooling layer, and a separable convolution layer connected in sequence, and also includes a second cross-attention mechanism unit and a timing modeling unit connected in sequence; the output of the timing modeling unit serves as another input of the separable convolution layer; the input of the first cross-attention mechanism unit and the second cross-attention mechanism unit serves as the input of the multimodal feature progressive fusion module; the output of the separable convolution layer serves as the output of the multimodal feature progressive fusion module.

[0017] Further: the multi-task decoder module based on multimodal features includes a first linear layer group, a third average pooling layer, a flattening layer, a second linear layer group, and a second fully connected layer connected in sequence.

[0018] Further: The specific steps for training the security analysis model include: A1. Obtain the well site operation video and mark the corresponding safety hazards in the well site operation video to obtain a training set; A2. Send the videos in the training set to the safety analysis model to obtain the safety analysis results of the well site operation videos, including the safety hazard analysis results of the three categories of workers themselves, workers and environment, and environment; A3. Calculate the joint loss based on the safety analysis results of the well site operation video. If the joint loss no longer decreases or reaches the set training times, end the training; otherwise, return to step A2; The combined loss The calculation expression is:

[0019]

[0020]

[0021]

[0022] in, It means the loss caused by workers' violation of rules and regulations. represents the number of samples, Represents the total number of hidden dangers of workers violating regulations, Indicates Samples contain hidden dangers The probability of As the prediction index, when the safety analysis model outputs The potential hazard type labels in the samples and the real potential hazard type labels When consistent is 1, otherwise is 0; Indicates the loss of judgment of workers on the relationship between safety and the environment, D The total number of categories representing the relationship between workers and environmental safety; Indicates the loss of judgment on hidden dangers in the environment. G Represents the total number of types of hidden dangers in the environment; represents a hyperparameter used to weight various types of losses; ln represents the natural logarithm.

[0023] Further: in step S4, when the multimodal feature progressive fusion module is used to fuse the global spatiotemporal features of the video modality and the pixel-level fine-grained local features of the single-frame image modality, feature fusion is performed through a back projection scheme, that is, the output of the multimodal feature progressive fusion module is used as additional feature information, and is input into the first cross-attention mechanism unit and the second cross-attention mechanism unit in the multimodal feature progressive fusion module respectively; in the first cross-attention mechanism unit, the additional feature information is fused with the pixel-level fine-grained local features of the single-frame image modality; in the second cross-attention mechanism unit, the additional feature information is fused with the global spatiotemporal features of the video modality; the fused features are further processed through a separable convolutional layer to obtain fused features, and the obtained fused features are once again used as additional feature information; the above-mentioned back projection operation is repeated until the entire video frame sequence is input, the loop is terminated, and the multimodal fusion features are obtained.

[0024] The beneficial effects of the present invention are: 1. The present invention uses the pre-trained image encoder of the multimodal large model SAM and CLIP to effectively extract the texture details and category information of the video frame. This method not only reduces the cost of data processing, but also can more comprehensively capture the complex environmental information in the well site operation, thereby significantly improving the accuracy and reliability of the well site video safety analysis.

[0025] 2. The present invention can effectively learn and extract the temporal and spatial information in the well site video modality. This enables the system to maintain efficient recognition and analysis capabilities in dynamic scenes, accurately monitor continuous actions and complex operations that occur during well site operations, and further improve the timeliness and accuracy of video safety analysis.

[0026] 3. The multi-scale spatial attention feature extraction module adopted in the present invention performs deep extraction of pixel-level fine-grained local features of single-frame video image modalities, so that the system can accurately identify tiny movements and small targets in the well site video, avoiding missing key details, thereby improving the comprehensiveness and accuracy of safety analysis.

[0027] 4. The design of multimodal feature progressive fusion module and multi-task decoder module enables the system to adapt to different well site video safety analysis tasks. Whether it is detecting the operating status of well site equipment, identifying dangerous behaviors, or monitoring the operational compliance of personnel, this method can handle it efficiently and has strong adaptability.

[0028] 5. The present invention adopts modular design, each functional module can be independently optimized and upgraded, and the system has good scalability. With the changes in the well site environment and safety requirements, the system can flexibly integrate new feature extraction methods and analysis models to support future technology upgrades and application expansions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the process of the present invention; Figure 2 It is a structural diagram of the safety analysis model; Figure 3 This is a schematic diagram of the structure of the video global feature extraction module based on the spatiotemporal attention mechanism; Figure 4 It is a schematic diagram of the structure of the multi-scale spatial attention feature extraction module; Figure 5 It is a structural diagram of the multimodal feature progressive fusion module; Figure 6 It is a schematic diagram of the structure of a multi-task decoder module based on multimodal features; Figure 7 Schematic diagram of the safety analysis results obtained in the embodiment. DETAILED DESCRIPTION

[0030] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0031] In one embodiment of the present invention, Figure 1 As shown, the attention-based multimodal well site video safety analysis method proposed in the present invention comprises the following steps: S1. Send the frame sequence in the well site operation video to be analyzed into the pre-trained image encoder to obtain the low-level fusion features of the video frame. Here, the frame sequence is the frame sequence in a GOP of the video. The well site operation video can be obtained through the well site operation area monitoring equipment or other means. CLIP is good at semantic understanding, while SAM focuses on spatial understanding in segmentation tasks, so the image encoder is composed of a pre-trained visual encoder based on SAM and a pre-trained visual encoder based on CLIP; the frame sequence of the well site operation video to be analyzed is used as the input of the image encoder, and is input into the pre-trained visual encoder based on SAM and the pre-trained visual encoder based on CLIP respectively, to obtain two types of features about the well site operation video, including low-level features such as texture, category, semantic concept and object relationship of the video frame; then the two types of features are fused to obtain the low-level fusion features of all video frames, which are expressed as follows:

[0032] in is the input of the image encoder; is the function representation of the CLIP visual encoder, are the parameters of the CLIP visual encoder, is the output of the CLIP visual encoder; is the function representation of the SAM visual encoder, are the parameters of the SAM visual encoder, is the output of the SAM visual encoder; H It is the low-level fusion feature of all video frames obtained in the end ， is the weight corresponding to the SAM visual encoder, yes CL The weights corresponding to the IP visual encoder.

[0033] S2. Input the obtained low-level fusion features into the video global feature extraction module based on the spatiotemporal attention mechanism to obtain the global spatiotemporal features of the video modality, namely, the temporal information and spatial information. Figure 3 As shown, the video global feature extraction module based on the spatiotemporal attention mechanism includes a small-core deep convolution layer, a multi-head temporal attention mechanism unit, an expanded deep convolution layer, and a 1×1 convolution layer connected in sequence; the video global feature extraction module based on the spatiotemporal attention mechanism also includes a first temporal self-attention mechanism unit, a first average pooling layer, and a first fully connected layer connected in sequence; the low-level fusion features obtained in step S1 are respectively used as inputs of the small-core deep convolution unit and the first temporal self-attention mechanism unit; the output of the 1×1 convolution layer is subjected to a Kronecker product operation with the output of the first fully connected layer, and then subjected to a Hadamard product operation with the low-level fusion features obtained in step S1 to obtain the global spatiotemporal features of the video modality. In the process of extracting the global spatiotemporal features of the video modality, the final attention obtained is the product of dynamic attention and static attention. Using small-core deep convolution , multi-head temporal attention mechanism, dilated deep convolution and 1×1 convolution to model large kernel convolution to obtain dynamic attention between frames S ; Obtain static attention within the frame through temporal self-attention mechanism, average pooling, and full connection operation DA . Its calculation expression is:

[0034] in, represents the output of the 1×1 convolutional layer, represents a 1×1 convolution, represents a depthwise convolution with dilation, represents multi-headed temporal attention processing, represents a small kernel depth convolution, is the low-level fusion feature obtained in step S1; represents the output of the first fully connected layer, Indicates full connection. represents global average pooling, represents temporal self-attention processing; is the output of the video global feature extraction module based on the spatiotemporal attention mechanism. represents the Kronecker product, represents the Hadamard product.

[0035] S3, input the key frame (I frame) in the frame sequence described in S1 into a feature extraction module based on multi-scale spatial attention to obtain pixel-level fine-grained local features of a single-frame image modality. Figure 4 As shown in the figure, the feature extraction module based on multi-scale spatial attention includes a VIT unit, a 2D convolution unit, a hole convolution layer, a multi-head cross attention mechanism unit, a self-attention mechanism unit and a fast normalized feature fusion unit connected in sequence; the key frame image is the input of the feature extraction module based on multi-scale spatial attention, and the output of the fast normalized feature fusion unit is the pixel-level fine-grained local feature of the single-frame image modality.

[0036] When extracting features from key frames, we extract features at five scales and use fast normalization fusion to obtain multi-scale spatial attention features; the five scales are the original image scale of the key frame image, half of the original image scale, one quarter of the original image scale, one eighth of the original image scale, and one sixteenth of the original image scale. First, we use ViT to extract preliminary features from key frame images at different scales, and then use 2D convolution and dilated convolution to obtain features of different scales. Before fusing features of different scales, it is necessary to further capture the dependencies between features of different scales and multiple angles in a fine-grained manner, and perform top-down and bottom-up bidirectional feature fusion, which is achieved through multi-head cross-attention mechanism and self-attention mechanism. In order to achieve more efficient aggregation of image features at different resolutions in the network, Fast Normalized Fusion is used for adaptive feature fusion. Its calculation expression is:

[0037]

[0038]

[0039] in, For the i Key frame images at different scales, express deal with, represents 2D convolution, represents the dilated convolution, Indicates i Preliminary features extracted from keyframe images at different scales; represents multi-head cross-attention processing, represents self-attention processing, represents the output of the self-attention mechanism unit, represents the output of the fast normalized feature fusion unit, express The corresponding weight represents the contribution of the corresponding feature; Represents the self-attention mechanism unit j The learning weights corresponding to the outputs are used to Perform normalization processing; Additional parameters to ensure the stability of the results, .

[0040] Fast normalized fusion is used to fuse features of different scales. Features of different scales contribute differently to the total output features, so additional weights are added to features of each scale so that the security analysis model can learn the respective weights of features of different scales. Each normalized weight ranges from 0 to 1. In order to ensure , in each Then add a Relu activation function.

[0041] S4, input the extracted global spatiotemporal features of the video modality and the pixel-level fine-grained local features of the single-frame image modality into the multi-modal feature progressive fusion module for deep fusion to obtain multi-modal fusion features. Figure 5 As shown, the multimodal feature progressive fusion module includes a first cross-attention mechanism unit, a multi-layer perceptron, a second average pooling layer, and a separable convolution layer connected in sequence, and also includes a second cross-attention mechanism unit and a timing modeling unit connected in sequence; the output of the timing modeling unit serves as another input of the separable convolution layer; the input of the first cross-attention mechanism unit and the second cross-attention mechanism unit serves as the input of the multimodal feature progressive fusion module; the output of the separable convolution layer serves as the output of the multimodal feature progressive fusion module.

[0042] In order to avoid the problem of shallow feature loss that may occur after feature fusion, when fusing the global spatiotemporal features of the video modality and the pixel-level fine-grained local features of the single-frame image modality, feature fusion is performed through a reverse projection scheme, which helps to better realize multimodal feature fusion. The reverse projection is to use the output of the multimodal feature progressive fusion module as additional feature information, and input it into the first cross-attention mechanism unit and the second cross-attention mechanism unit in the multimodal feature progressive fusion module respectively; in the first cross-attention mechanism unit, the additional feature information is fused with the pixel-level fine-grained local features of the single-frame image modality; in the second cross-attention mechanism unit, the additional feature information is fused with the global spatiotemporal features of the video modality; the fused features are further processed by a separable convolutional layer to obtain fused features, and the obtained fused features are once again used as additional feature information; the above reverse projection operation is repeated until the entire video frame sequence is input, the loop ends, and the multimodal fusion features are obtained. The feature inputs of different modalities are represented as , Indicates The input features of the modality, in this embodiment , indicating that there are two modes; the final multimodal fusion feature is expressed as ,in Indicates additional feature information, Indicates feature fusion. In this way, connections are established between features at different levels, so that the additional feature information after feature fusion can be used by the shallow network, avoiding the problem of "either fusion or loss" in multimodal feature fusion.

[0043] S5. Input the multimodal fusion features into the multi-task decoder module based on multimodal features to obtain the safety analysis results of the well site operation video, such as Figure 7 As shown in the figure, the safety hazard analysis results of three categories are included: workers themselves (workers sleeping, operating mobile phones, slacking off), workers and environment (dangerous standing, dangerous leaning, dangerous walking), and environment (missing material funnel, unclosed cover, lack of warning signs). The multi-task learning method can learn different types of tasks and make result analysis, while avoiding the limitations of existing video analysis algorithms that can only target a single task and result in the singleness of the safety hazards concerned and the complex problems caused by multiple models. Figure 6 As shown, the multi-task decoder module based on multimodal features includes a first linear layer group, a third average pooling layer, a flattening layer, a second linear layer group, and a second fully connected layer connected in sequence.

[0044] like Figure 2As shown in the figure, the image encoder, the video global feature extraction module based on the spatiotemporal attention mechanism, the multimodal feature progressive fusion module, the multi-task decoder module based on multimodal features, and the feature extraction module based on multi-scale spatial attention constitute the security analysis model.

[0045] In this embodiment, the specific steps of training the security analysis model include: A1. Obtain the well site operation video and mark the corresponding safety hazards in the well site operation video to obtain a training set; A2. Send the videos in the training set to the safety analysis model to obtain the safety analysis results of the well site operation videos, including the safety hazard analysis results of the three categories of workers themselves, workers and environment, and environment; A3. Calculate the loss of each task according to the safety analysis results of the well site operation video, and optimize the model with the joint loss as the overall target loss. If the joint loss no longer decreases or reaches the set training times, end the training, otherwise return to step A2; The combined loss The calculation expression is:

[0046]

[0047]

[0048]

[0049] in, It means the loss caused by workers' violation of rules and regulations. represents the number of samples, Represents the total number of hidden dangers of workers violating regulations, Indicates Samples contain hidden dangers The probability of As the prediction index, when the safety analysis model outputs The potential hazard type labels in the samples and the real potential hazard type labels When consistent is 1, otherwise is 0; Indicates the loss of judgment of workers on the relationship between safety and the environment, D The total number of categories representing the relationship between workers and environmental safety; Indicates the loss of judgment on hidden dangers in the environment. G Represents the total number of types of hidden dangers in the environment; represents a hyperparameter used to weight various types of losses; ln represents the natural logarithm.

[0050] Through multi-task learning, the original feature extraction network can learn shared representations of different video analysis tasks, and utilize the complementarity of feature representations of different tasks to improve overall performance and maintain feature semantic consistency, so that the model can realize multiple functions under a unified framework.

[0051] In summary, the present invention has good timeliness and high precision, can handle diversified tasks, and has strong adaptability.

Claims

1. A multimodal well site video safety analysis method based on attention, characterized in that: The following steps are involved: S1, sending the frame sequence in the well site operation video to be analyzed into the pre-trained image encoder to obtain the low-level fusion features of the video frame; S2. Input the obtained low-level fusion features into the video global feature extraction module based on the spatiotemporal attention mechanism to obtain the global spatiotemporal features of the video modality, namely, the temporal information and spatial information; S3, inputting the key frames in the frame sequence described in S1 into a feature extraction module based on multi-scale spatial attention to obtain pixel-level fine-grained local features of a single-frame image modality; S4, inputting the extracted global spatiotemporal features of the video modality and the pixel-level fine-grained local features of the single-frame image modality into the multimodal feature progressive fusion module for deep fusion to obtain multimodal fusion features; S5, inputting the multimodal fusion features into a multi-task decoder module based on multimodal features to obtain safety analysis results of the well site operation video, including safety hazard analysis results of three categories: workers themselves, workers and environment, and environment; The security analysis model is composed of an image encoder, a video global feature extraction module based on spatiotemporal attention mechanism, a multimodal feature progressive fusion module, a multi-task decoder module based on multimodal features, and a feature extraction module based on multi-scale spatial attention.

2. The attention-based multimodal wellsite video safety analysis method according to claim 1, characterized in that: Image encoders include SAM-based pre-trained visual encoders and CLIP-based pre-trained visual encoders; The frame sequence in the well site operation video to be analyzed is used as the input of the image encoder, and is input into the pre-trained visual encoder based on SAM and the pre-trained visual encoder based on CLIP respectively, to obtain two types of features about the well site operation video; Then the two types of features are fused to obtain the low-level fusion features of all video frames, which are expressed as follows: in is the input of the image encoder; is the function representation of the CLIP visual encoder, are the parameters of the CLIP visual encoder, is the output of the CLIP visual encoder; is the function representation of the SAM visual encoder, are the parameters of the SAM visual encoder, is the output of the SAM visual encoder; H is the low-level fusion feature of all video frames finally obtained, is the weight corresponding to the SAM visual encoder, yes CL The weights corresponding to the IP visual encoder.

3. The attention-based multimodal wellsite video safety analysis method according to claim 1, characterized in that: The video global feature extraction module based on the spatiotemporal attention mechanism includes a small-core deep convolution layer, a multi-head temporal attention mechanism unit, an expanded deep convolution layer, and a 1×1 convolution layer connected in sequence; the video global feature extraction module based on the spatiotemporal attention mechanism also includes a first temporal self-attention mechanism unit, a first average pooling layer, and a first fully connected layer connected in sequence; the low-level fusion features obtained in step S1 are respectively used as inputs of the small-core deep convolution unit and the first temporal self-attention mechanism unit; the output of the 1×1 convolution layer is subjected to a Kronecker product operation with the output of the first fully connected layer, and then subjected to a Hadamard product operation with the low-level fusion features obtained in step S1 to obtain the global spatiotemporal features of the video modality, and the calculation expression thereof is: in, represents the output of the 1×1 convolutional layer, represents a 1×1 convolution, represents a depthwise convolution with dilation, represents multi-headed temporal attention processing, represents a small kernel depth convolution, is the low-level fusion feature obtained in step S1; represents the output of the first fully connected layer, Indicates full connection. represents global average pooling, represents temporal self-attention processing; is the output of the video global feature extraction module based on the spatiotemporal attention mechanism. represents the Kronecker product, Represents the Hadamard product.

4. The attention-based multimodal wellsite video safety analysis method according to claim 1, characterized in that: In step S3, a feature extraction module based on multi-scale spatial attention extracts features of five scales from the key frame image and uses fast normalization fusion to obtain multi-scale spatial attention features; The five scales are respectively the original image scale of the key frame image, half of the original image scale, one quarter of the original image scale, one eighth of the original image scale, and one sixteenth of the original image scale.

5. The attention-based multimodal wellsite video safety analysis method according to claim 4, characterized in that: The feature extraction module based on multi-scale spatial attention includes a VIT unit, a 2D convolution unit, a hole convolution layer, a multi-head cross attention mechanism unit, a self-attention mechanism unit and a fast normalized feature fusion unit connected in sequence; the key frame image is the input of the feature extraction module based on multi-scale spatial attention, and the output of the fast normalized feature fusion unit is the pixel-level fine-grained local feature of the single-frame image modality; Its calculation expression is: in, For the i Key frame images at different scales, express deal with, represents 2D convolution, represents the dilated convolution, Indicates i Preliminary features extracted from keyframe images at different scales; represents multi-head cross-attention processing, represents self-attention processing, Represents the output of the self-attention mechanism unit; represents the output of the fast normalized feature fusion unit, express The corresponding weight, Represents the self-attention mechanism unit j The learning weights of the outputs are used to Perform normalization processing; Additional parameters to ensure the stability of the results.

6. The attention-based multimodal wellsite video safety analysis method according to claim 1, characterized in that: The multimodal feature progressive fusion module includes a first cross-attention mechanism unit, a multi-layer perceptron, a second average pooling layer, and a separable convolution layer connected in sequence, and also includes a second cross-attention mechanism unit and a timing modeling unit connected in sequence; the output of the timing modeling unit serves as another input of the separable convolution layer; the input of the first cross-attention mechanism unit and the second cross-attention mechanism unit serves as the input of the multimodal feature progressive fusion module; the output of the separable convolution layer serves as the output of the multimodal feature progressive fusion module.

7. The attention-based multimodal wellsite video safety analysis method according to claim 1, characterized in that: The multi-task decoder module based on multimodal features includes a first linear layer group, a third average pooling layer, a flattening layer, a second linear layer group, and a second fully connected layer connected in sequence.

8. The attention-based multimodal wellsite video safety analysis method according to claim 1, characterized in that: The specific steps for training a security analysis model include: A1. Obtain the well site operation video and mark the corresponding safety hazards in the well site operation video to obtain a training set; A2. Send the videos in the training set to the safety analysis model to obtain the safety analysis results of the well site operation videos, including the safety hazard analysis results of the three categories of workers themselves, workers and environment, and environment; A3. Calculate the joint loss based on the safety analysis results of the well site operation video. If the joint loss no longer decreases or reaches the set training times, end the training; otherwise, return to step A2; The combined loss The calculation expression is: in, It means the loss caused by workers' violation of rules and regulations. represents the number of samples, Represents the total number of hidden dangers of workers violating regulations, Indicates Samples contain hidden dangers The probability of As the prediction index, when the safety analysis model outputs The potential hazard type labels in the samples and the real potential hazard type labels When consistent is 1, otherwise is 0; Indicates the loss of judgment of workers on the relationship between safety and the environment, D The total number of categories representing the relationship between workers and environmental safety; Indicates the loss of judgment on hidden dangers in the environment. G Represents the total number of types of hidden dangers in the environment; represents a hyperparameter used to weight various types of losses; ln represents the natural logarithm.

9. The attention-based multimodal wellsite video safety analysis method according to claim 5, characterized in that: When the multimodal feature progressive fusion module is used in step S4 to fuse the global spatiotemporal features of the video modality and the pixel-level fine-grained local features of the single-frame image modality, feature fusion is performed through a back-projection scheme, that is, the output of the multimodal feature progressive fusion module is used as additional feature information, and is input into the first cross-attention mechanism unit and the second cross-attention mechanism unit in the multimodal feature progressive fusion module respectively; in the first cross-attention mechanism unit, the additional feature information is fused with the pixel-level fine-grained local features of the single-frame image modality; in the second cross-attention mechanism unit, the additional feature information is fused with the global spatiotemporal features of the video modality; The fused features are further processed through a separable convolutional layer to obtain fused features, and the obtained fused features are once again used as additional feature information; the above-mentioned back projection operation is repeated until all video frame sequences are input, and the loop ends to obtain multimodal fusion features.

Images