Field operation behavior identification method and device based on improved feature fusion mechanism

Abstract

The invention provides a field operation behavior recognition method and device based on an improved feature fusion mechanism, and the method comprises the steps: collecting a plurality of video clips containing preset motion features for a target operation behavior, carrying out the target detection of the video clips, and recognizing a key space region of a person and equipment; intercepting a first feature image set formed by a first feature image set including a person and a device key space region, and performing multi-layer feature fusion on features of different layers in each first feature image to obtain a second feature image set; and taking a weighted average value of forward propagation and back propagation of the second feature image set as a final recognition image until the final recognition image sets corresponding to all video clips are output to form an image recognition set, and establishing a deep learning model to train the image recognition set to perform field operation behavior recognition on a field target operation behavior. According to the invention, real-time field operation automatic identification can be realized, false alarm and missing alarm are prevented, and automatic management and early warning are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of on-site operation behavior recognition, and in particular to an on-site operation behavior recognition method and device based on an improved feature fusion mechanism. Background Art

[0002] With the continuous advancement of science and technology, especially the rapid development in the fields of artificial intelligence and machine vision, the on-site operation behavior recognition technology of hydropower stations has gradually received attention and has been widely used. The introduction of this technology is mainly to solve many problems existing in traditional manual supervision in hydropower station operations, such as low efficiency and difficulty in ensuring safety. However, this technology still has the following problems:

[0003] 1. The problem of insufficient technical maturity. Although AI technology has achieved remarkable results in some fields, the technical maturity of identifying on-site operation behaviors in hydropower stations still needs to be improved. Especially in complex and changeable operating environments, how to ensure the accuracy and stability of recognition is still a major challenge currently faced.

[0004] 2. False alarm and missed alarm problems. Due to the complexity and uncertainty of the on-site environment, the AI ​​system may have false alarms and missed alarms when performing behavior recognition, which may lead to the neglect or overreaction of safety hazards and affect the safe production of the hydropower station.

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and device for identifying on-site operation behaviors based on an improved feature fusion mechanism, which helps to improve the real-time monitoring of sites, and timely discover and warn of potential safety hazards. Summary of the invention

[0006] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a method and device for identifying on-site operation behaviors based on an improved feature fusion mechanism, which can automatically identify on-site operations in real time, prevent false alarms and missed alarms, and facilitate automated management and early warning.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A method for identifying on-site operation behaviors based on an improved feature fusion mechanism, comprising:

[0009] Step S1: collecting multiple video clips containing preset action features for the target operation behavior;

[0010] Step S2: using the YOLOv8 algorithm to perform target detection on the video clip, identifying key spatial regions of people and equipment, intercepting multiple first feature images containing key spatial regions of people and equipment based on the segmentation attention network, and grouping the multiple first feature images into a first feature image set;

[0011] Step S3: performing multi-layer feature fusion on features at different levels in each first feature image to obtain a second feature image, and combining all second feature images into a second feature image set;

[0012] Step S4: Based on the Bi-LSTM network, a weighted average of the forward propagation and the backward propagation of the second feature image set is taken as the final recognition image;

[0013] Step S5: repeating steps S2 to S4 until the final recognition images corresponding to all video clips are output, and all the final recognition images are grouped into an image recognition set;

[0014] Step S6: Establish a deep learning model to perform model training on the image recognition set, and perform on-site operation behavior recognition on the target operation behavior on site based on the trained deep learning model.

[0015] Furthermore, the method for collecting multiple video clips containing preset action features for the target operation behavior is:

[0016] The historical video of the target operation behavior is shot, and the historical video is sparsely sampled to obtain multiple video clips containing preset action features.

[0017] Furthermore, the YOLOv8 algorithm is used to detect the target in the video clip, identify the key spatial areas of people and equipment, and intercept multiple first feature images containing the key spatial areas of people and equipment based on the segmentation attention network. The method of grouping the multiple first feature images into a first feature image set is as follows:

[0018] Step S201: identifying key spatial areas of people and equipment through YOLOv8 to obtain an initial recognition image set, wherein the initial recognition image set includes multiple initial recognition images;

[0019] Step S202: Establish a segmentation attention network, divide the initial recognition image as input into multiple basis arrays, and divide each basis array into multiple feature map groups;

[0020] Step S203: using the key spatial regions of characters and devices as full-text context information for each feature graph group to obtain the weight of each feature graph group in the channel dimension;

[0021] Step S204: performing weighted processing on all feature map groups in each basis array, and obtaining channel feature maps of all basis arrays based on channel dimension soft attention;

[0022] Step S205: short-circuit all channel feature atlas groups along the channel dimension to obtain the output of the segmentation attention network;

[0023] Step S206: short-circuiting the output of the segmentation attention network and the initial recognition image to generate a first feature image;

[0024] Step S207: repeating steps S202 to S206 until all first feature images are output, and all first feature images are grouped into a first feature image set.

[0025] Furthermore, the method of performing multi-layer feature fusion on the features at different levels in each first feature image to obtain the second feature image is:

[0026] Step S301: Establishing a ResNest backbone network;

[0027] Step S302: for a first feature image, a high-dimensional convolution kernel is replaced by a plurality of low-dimensional convolution kernels;

[0028] Step S303: Obtaining shallow features of the first feature image;

[0029] Step S304: obtaining deep features of the first feature image;

[0030] Step S305: fusing shallow features and deep features to obtain a second feature image;

[0031] Step S306: repeating steps S302 to S305 until all second feature images are output, and all second feature images are grouped into a second feature image set.

[0032] Furthermore, based on the Bi-LSTM network, the method of taking the weighted average of the forward propagation and the back propagation of the second feature image set as the final recognition image is:

[0033] Step S401: using global average pooling to expand the fused features in the second feature image into a one-dimensional vector;

[0034] Step S402: A Bi-LSTM network is established based on the forward LSTM neural network and the backward LSTM neural network, all second feature images are input into the Bi-LSTM network in chronological order, and a weighted average of the forward propagation and the backward propagation is output as the final recognition image.

[0035] Furthermore, in the deep learning model, the Softmax activation function is used to generate the category probability distribution, where the probability value of each category is between 0 and 1, and the sum of the probabilities of all categories is 1.

[0036] A field operation behavior recognition device based on an improved feature fusion mechanism, comprising:

[0037] A collection module, used to collect multiple video clips containing preset action features for the target operation behavior;

[0038] A first feature image set acquisition module is used to perform target detection on the video clip using the YOLOv8 algorithm, identify key spatial regions of people and equipment, intercept multiple first feature image sets containing key spatial regions of people and equipment into a first feature image set based on a segmentation attention network, and combine multiple first feature image sets into a first feature image set;

[0039] A second feature image set acquisition module, used for performing multi-layer feature fusion on features at different levels in each first feature image to acquire a second feature image, and to gather all second feature images into a second feature image set;

[0040] A final recognition image acquisition module is used to take the weighted average of the forward propagation and the backward propagation of the second feature image set as the final recognition image based on the Bi-LSTM network;

[0041] An image recognition set acquisition module, used to repeat the first feature image set acquisition module to the final recognition image acquisition module until the final recognition images corresponding to all video clips are output, and all the final recognition images are grouped into an image recognition set;

[0042] The deep learning model building module is used to build a deep learning model to perform model training on the image recognition set, and to identify the target operation behavior on site based on the trained deep learning model.

[0043] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned on-site operation behavior recognition method based on the improved feature fusion mechanism is implemented.

[0044] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned on-site operation behavior recognition method based on an improved feature fusion mechanism.

[0045] A computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned on-site operation behavior recognition method based on the improved feature fusion mechanism.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] The on-site operation behavior recognition method based on the improved feature fusion mechanism provided by the present invention is a behavior algorithm for extracting action features. The action features are the action behavior features of on-site workers during the inspection process, such as face recognition when entering the site, whether a helmet is worn in a standardized manner, whether there is smoking and other behaviors, etc. Multiple video clips containing preset action features are collected for the target operation behavior, and features are extracted from each video clip using a segmented attention network, multi-layer feature fusion and a Bi-LSTM network to obtain a final recognition image of each video clip, fully utilizing the spatiotemporal information of the video, and effectively realizing the modeling of long videos. The present invention combines all image recognition sets into an image recognition set, and then establishes a deep learning model to predict the final recognition images corresponding to these video clips respectively, so as to solve the problem that the system's automatic recognition and early warning rely too much on manual intervention, thereby improving the automation level of the system and reducing the workload of manual processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0049] Figure 1 It is a schematic diagram of the on-site operation behavior recognition method based on the improved feature fusion mechanism of the present invention;

[0050] Figure 2 It is a flow chart of the on-site operation behavior recognition algorithm based on the improved feature fusion mechanism of the present invention;

[0051] Figure 3 It is a schematic diagram of the on-site operation behavior recognition device based on the improved feature fusion mechanism of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0053] Example 1

[0054] Traditional behavior recognition technology is backward and relies on human intervention. Although AI technology can automatically identify and warn of unsafe behaviors, in some cases it still requires human intervention for confirmation and processing. This reduces the system's automation level to a certain extent and increases the workload of manual processing.

[0055] Traditional on-site operation behavior recognition algorithms characterize on-site operation actions by extracting behavioral features such as feature descriptions of moving targets to achieve the purpose of recognition. The operation behavior features extracted by traditional methods are limited by experience and the singleness of actions, and have weak robustness and migration capabilities.

[0056] This embodiment provides a method for identifying on-site operation behaviors based on an improved feature fusion mechanism. Figure 1 and Figure 2 As shown, including:

[0057] Step S1: collecting multiple video clips containing preset action features for the target operation behavior;

[0058] Step S2: using the YOLOv8 algorithm to perform target detection on the video clip, identifying key spatial regions of people and equipment, intercepting multiple first feature images containing key spatial regions of people and equipment based on the segmentation attention network, and grouping the multiple first feature images into a first feature image set;

[0059] Step S3: performing multi-layer feature fusion on features at different levels in each first feature image to obtain a second feature image, and combining all second feature images into a second feature image set;

[0060] Step S4: Based on the Bi-LSTM network, a weighted average of the forward propagation and the backward propagation of the second feature image set is taken as the final recognition image;

[0061] Step S5: repeating steps S2 to S4 until the final recognition images corresponding to all video clips are output, and all the final recognition images are grouped into an image recognition set;

[0062] Step S6: Establish a deep learning model to perform model training on the image recognition set, and perform on-site operation behavior recognition on the target operation behavior on site based on the trained deep learning model.

[0063] The on-site operation behavior recognition method based on the improved feature fusion mechanism provided in this embodiment is a behavior algorithm for extracting action features. The action features are the action behavior features of on-site workers during the inspection process, such as face recognition when entering the site, whether a helmet is worn properly, whether there is smoking, etc. Multiple video clips containing preset action features are collected for the target operation behavior, and a segmented attention network, multi-layer feature fusion and Bi-LSTM network are used to extract features for each video clip to obtain the final recognition image of each video clip, making full use of the spatiotemporal information of the video, and effectively realizing the modeling of long videos. In this embodiment, all image recognition sets are grouped into an image recognition set, and then a deep learning model is established to predict the final recognition images corresponding to these video clips respectively, so as to solve the problem that the system's automatic recognition and early warning rely too much on manual intervention, thereby improving the automation level of the system and reducing the workload of manual processing.

[0064] In step S1 of this embodiment, the method of collecting multiple video clips containing preset action features for the target operation behavior is:

[0065] The historical video of the target operation behavior is shot, and the historical video is sparsely sampled to obtain multiple video clips containing preset action features.

[0066] In step S2 of this embodiment, the method of using the YOLOv8 algorithm to detect objects in the video clips and identify key spatial areas of people and equipment is as follows:

[0067] The video clips are processed by frame extraction to obtain an image sequence containing key spatial areas of people and equipment. The image sequence is preprocessed using four image enhancement methods, including mosaic enhancement (Mosaic), spatial perturbation (Random Perspective), mixed enhancement (Mixup), and color perturbation (HSV augment).

[0068] (1) Mosaic enhancement: Multiple images of power field operations are stitched together into a large mosaic image. By randomly cropping and stitching the images, the model can better learn the location and features of targets in different scenarios, thereby improving the robustness of the model.

[0069] (2) Spatial perturbation (Random Perspective): Random perspective transformation is performed on the power site operation images to simulate shooting conditions at different angles and viewing angles. This helps the model better adapt to the changes in camera positions in actual power sites and improves the model’s detection performance in different scenarios.

[0070] (3) Mixup: Mixup can be used to perform mixed enhancement on different power field operation images to generate new training samples, increase the diversity of the data set, help improve the generalization ability of the model, and reduce the occurrence of overfitting.

[0071] (4) Color perturbation (HSV Augment): By applying color perturbation to power site operation images, the model can be better adapted to targets such as power equipment and signboards under different lighting conditions, thereby enhancing the robustness of the model.

[0072] In the YOLOv8 algorithm, the backbone feature extraction network adopts a modular structure, which consists of a CBS module, a C2f module (CSPBottleneck with 2convolutions) and a spatial pyramid pooling module (SPPF). In the backbone feature extraction network of the model, YOLOv8 also uses an improved Focus structure at the input end of the model.

[0073] The Focus structure is designed to reduce the dimensionality of the input image while retaining key image information to the maximum extent. Through slicing operations, Focus divides the original input image into four independent feature layers and stacks them, thereby quadrupling the dimension of the channel and retaining more original information. Although the Focus structure effectively alleviates the problem of information loss, it also brings about an increase in the amount of computation and increases the requirements for GPU hardware performance. Although this design has achieved remarkable results in retaining spatial information and increasing dimensionality, in some cases, there may be certain limitations due to the increased computational burden and high requirements for GPU performance.

[0074] In the YOLOv8 algorithm, the original Focus structure is cleverly replaced with a 3×3 convolution layer. This alternative not only achieves the same effect, but also makes the convolution kernel more friendly and efficient for some GPU hardware. This replacement strategy better adapts to the hardware environment while maintaining performance, providing a better balance for the practical application of the model.

[0075] The CBS module used in YOLOv8 consists of a convolutional layer, batch normalization (BatchNormalization, BN), and SiLU activation function (Sigmoid-weighted Linear Unit).

[0076] This module is designed to improve the expressiveness of the model, accelerate the training process, and avoid the problem of gradient vanishing to a certain extent. Compared with the CBL module in YOLOv5, which is composed of convolution (Conv), batch normalization (BN) and activation function LeakyRelu, the improved SiLU activation function in CBS has only a lower bound but no upper bound, and the smooth and non-monotonic design is more conducive to deep learning training. It is to comprehensively utilize the advantages of these three components to improve the expressiveness of the overall model. The convolution layer extracts features from the image through the convolution kernel, and the standardization ensures the standardized stability of the input data, while the SiLU activation function plays a key role in nonlinear transformation. This combination helps to overcome the common gradient vanishing problem in deep networks and improve the model's learning efficiency for complex features.

[0077] In YOLOv8, the C2f module first maps the input features to twice the number of hidden channels through a Conv layer with a 1x1 convolution kernel. Then, the features are processed through multiple Bottleneck modules in the branch, which include residual connections, group convolutions and other operations to better capture the nonlinear relationship of the features. Finally, the processed features are mapped to the final number of output channels through another Conv layer with a 1x1 convolution kernel. Unlike the traditional series structure, the C2f module introduces the Split operation in the branch, so that the features are spatially segmented and then cross-layered, enriching the expression of the feature map.

[0078] The C2f module successfully improves the integration and representation capabilities of features at different levels in YOLOv8, thereby effectively improving the target detection performance. In this module, the gradient flow of the model is enriched by fusing some features across layers, thereby enhancing the ability to express features. The purpose of this design is to enable the model to better capture feature information at different levels, thereby improving target detection performance.

[0079] At different levels of the network, the convolution operation gradually extracts features by adjusting the convolution kernel and step size, and at the same time adjusts the number of channels through the C2f module to increase the nonlinear representation of the network.

[0080] The SPPF module adjusts the original three different sizes of convolution kernels into three 5×5 convolution kernels. By connecting these small convolution kernels in series, it achieves a receptive field equivalent to that of a single large-size convolution kernel. Specifically, using two 5×5 convolution kernels in series is equivalent to using a 9×9 convolution kernel, while three 5×5 convolution kernels in series are equivalent to a 13×13 convolution kernel. This alternative reduces the amount of network computation while improving detection efficiency, providing a more flexible and efficient option for the practical application of the model.

[0081] The feature fusion layer is located between the backbone feature extraction network and the prediction network. It has the function of fusing shallow position texture information with deep strong semantic information. The YOLOv8 model uses PANet as the feature fusion layer.

[0082] The PANet structure introduces a bottom-up feature fusion mechanism based on FPN to form a bidirectional fusion network, which aims to make full use of the deep strong semantic information and shallow detailed texture information to improve the target detection performance. The specific calculation process is: First, the high-level semantic information of the deep layer is transferred to the shallow layer by upsampling, and fused with the shallow detailed texture information. This helps to enrich the semantic information of the shallow features and improve the accuracy of target detection. Secondly, the fused feature information is passed back to the deep layer through downsampling to achieve bidirectional information flow. Finally, the bidirectional fused features are sent to the prediction network for target detection prediction. This design effectively integrates feature information at different levels, making the network more comprehensive and accurate in perceiving targets.

[0083] The feature fusion layer of YOLOv8 achieves efficient integration of multi-level feature maps by cleverly combining key steps such as upsampling, feature map concatenation, channel conversion and convolution, thereby significantly improving the performance of target detection. Upsampling is used to accurately upsample the feature map and expand the spatial dimension. The upsampled feature map is concatenated with the feature map from the backbone through the Concat operation to ensure that the multi-level features are organically integrated. The number of channels is adjusted through the C2f operation to ensure the consistency of the feature map channels of each layer. Then, a 3x3 convolution is performed with a step size of 2 to integrate feature information at a deeper level. Finally, the convolution feature map is concatenated with feature maps of different levels through the Concat operation again to achieve the final feature fusion. This layer-by-layer operation not only makes full use of the multi-level feature information, but also improves the model's perception of the target, making the target detection significantly improved in terms of accuracy and robustness.

[0084] YOLOv8 continues the multi-scale prediction capability of the YOLO series and can effectively predict feature information of different scales. Among them, the 80×80 feature map contains rich detail texture information and is suitable for the detection of small targets; the 40×40 feature map retains certain semantic information while losing some detail texture, which is suitable for the detection of medium-sized targets; the 20×20 feature map has strong semantic information and is suitable for the detection of relatively large targets. Another important improvement of YOLOv8 in the network structure is that in the Head layer, the original coupled detection head (Coupled Head) is replaced with a decoupled detection head (Decoupled Head), and the anchor-based is changed to Anchor Free. The decoupled detection head decouples classification and regression into two independent branches. Experiments have shown that this design can accelerate convergence and improve detection accuracy. In addition, YOLOv8 uses a decoupled detection head to reduce the influence of K-means clustering caused by human factors. The anchor-free design reduces the number of detections, simplifies the prediction network, and effectively improves the inference speed. These improvements make YOLOv8 more efficient and accurate in target detection tasks.

[0085] YOLOv8 uses a specific loss function configuration, where the classification loss function is VFL (Varifocal Loss), and the regression loss is a combination of CIoU Loss and DFL (Distribution Focal Loss). The choice of this loss function is intended to improve the accuracy of the model's target classification and positioning. VFL focuses on optimizing the classification task, while CIoU Loss and DFL work together on the regression task, jointly building a more accurate and robust training framework.

[0086] (1) Compared with the traditional IoU (Intersection over Union), CIoU not only considers the differences in the position and size of the box, but also the differences in the aspect ratio and diagonal length of the box. This allows CIoU to more comprehensively capture the geometric relationship between target boxes and provide a more accurate regression loss.

[0087] The key spatial areas of people and equipment are identified through YOLOv8 to obtain an initial recognition image set, which includes multiple initial recognition images.

[0088] In step S2 of this embodiment, BASNet is a simple and powerful boundary-aware segmentation network, including a prediction-refinement architecture and a hybrid loss, for high-precision image segmentation. By introducing an attention mechanism, the model can focus more on the key areas in the image, thereby improving the accuracy of segmentation or detection. The present invention segments the key areas of people and equipment.

[0089] The YOLOv8 algorithm is used to detect the target in the video clip, identify the key spatial areas of people and equipment, and intercept multiple first feature images containing the key spatial areas of people and equipment based on the segmentation attention network. The method of grouping multiple first feature images into a first feature image set is as follows:

[0090] Step S201: identifying key spatial areas of people and equipment through YOLOv8 to obtain an initial recognition image set, wherein the initial recognition image set includes multiple initial recognition images;

[0091] Step S202: Establish a segmentation attention network, divide the initial recognition image as input into multiple basis arrays, and divide each basis array into multiple feature map groups.

[0092] Specifically, the input initial recognition image is first divided into k basis arrays, and then each basis array is divided into r feature map groups, so the total number of feature map groups can be recorded as G = kr. Channel dimension attention is used for each feature map group to improve the interactivity between channels. The feature granularity is improved by this method of refining the grouping and assigning channel weights.

[0093] Step S203: For each feature graph group, the key spatial regions of characters and devices are used as full-text context information to obtain the weight of each feature graph group in the channel dimension.

[0094] Specifically, the global context information S k It is the global statistical information embedded in the channel, which can describe the correlation between pixels in the video frame. In the segmentation attention network, by using the global context information S k Calculate the weights a of different feature map groups in the channel dimension i .

[0095] Step S204: Perform weighted processing on all feature map groups in each basis array, and obtain channel feature maps of all basis arrays based on channel dimension soft attention. The feature map of each channel is generated by weighted combination of different feature map groups.

[0096] Step S205: Short-circuit all channel feature graph sets along the channel dimension to obtain the output of the segmentation attention network. Connect each feature graph set along the channel dimension to obtain the output of the segmentation attention module.

[0097] Step S206: Short-circuit the output of the segmentation attention network and the initial recognition image to generate a first feature image.

[0098] Step S207: repeating steps S202 to S206 until all first feature images are output, and all first feature images are grouped into a first feature image set.

[0099] In step S3 of this embodiment, the method of performing multi-layer feature fusion on the features at different levels in each first feature image to obtain the second feature image is:

[0100] Step S301: Establishing a ResNest backbone network;

[0101] Step S302: for a first feature image, a high-dimensional convolution kernel is replaced by a plurality of low-dimensional convolution kernels;

[0102] Step S303: Obtaining shallow features of the first feature image;

[0103] Step S304: obtaining deep features of the first feature image;

[0104] Step S305: fusing shallow features and deep features to obtain a second feature image;

[0105] Step S306: repeating steps S302 to S305 until all second feature images are output, and all second feature images are grouped into a second feature image set.

[0106] Specifically, multi-layer feature fusion is used to remove redundant background and noise, retain spatial location information, and output feature maps to show whether operators comply with operating specifications when performing on-site operations.

[0107] Convolutional network is a neural network specially designed to process data with grid-like topological structure, such as image data. Its core feature is the convolution operation, which allows the network to learn local features of input data and greatly reduces the number of model parameters due to weight sharing. Multi-layer feature fusion is an important technology in convolutional networks, which effectively combines feature information from different layers of the network to improve the performance of the model. The main purpose is to effectively integrate feature information from different levels of the network to make full use of feature information at different levels. Low-level features usually contain more details and local information, while high-level features contain more global and abstract information.

[0108] Multi-layer feature fusion considers the differences between features at different levels of convolutional neural networks, and fuses the two through operations such as average pooling and splicing to improve the interaction of information, thereby improving the accuracy of the on-site operation behavior recognition algorithm. The present invention uses ResNest as the backbone network and introduces a feature fusion mechanism in the network to improve the representation ability of CNN features. The ResNest network is mainly divided into 5 modules.

[0109] In this embodiment, three 3*3 convolution kernels are used in the Conv_custom layer to replace the 7*7 convolution kernel for convolution, which reduces the amount of calculation without changing the receptive field. At the end of the Conv_custom layer, maximum pooling is used to output a feature map of size 128*56*56. After a series of convolution and pooling operations, a feature map of 2048*7*7 is output at the Layer4 layer. At the end of the forward convolution, the ResNeSi network reduces the size of the output of the Laver1 layer to 7*7 through an average pooling operation, and fuses it with the Layer4 layer as a shallow feature. The Layer1 layer output is selected here as the shallow feature because the Layer1 layer has undergone more convolution operations than the Conv_custom layer, removing redundant background and noise, and retaining more spatial position information relative to the Layer2 and Layer3 layers. The shallow features after pooling are superimposed (concatenation) with the deep features of the Layer4 layer in the channel dimension to achieve the purpose of fusion features.

[0110] In step S4 of this embodiment, the method of taking the weighted average of the forward propagation and the backward propagation of the second feature image set as the final recognition image based on the Bi-LSTM network is:

[0111] Step S401: using global average pooling to expand the fused features in the second feature image into a one-dimensional vector;

[0112] Step S402: A Bi-LSTM network is established based on the forward LSTM neural network and the backward LSTM neural network, all second feature images are input into the Bi-LSTM network in chronological order, and a weighted average of the forward propagation and the backward propagation is output as the final recognition image.

[0113] Specifically, global average pooling is used again to expand the fused features to 1*1*2048 and transmit the one-dimensional vector to the Bi-LSTM network.

[0114] The LSTM network introduces a "gate" structure to update the "cell state", which effectively alleviates the gradient vanishing problem. Figure 3 shown.

[0115] LSTM networks are very suitable for processing sequence data, but their unidirectional structure does not utilize the future information of sequence data. Bi-LSTM networks modify them by adding a reverse transmission structure while forward transmission. This structure can not only utilize previous video frames when processing video data, but also rely on subsequent video frames, making full use of the before-after correlation characteristics of on-site operation actions.

[0116] In the deep learning model of step S5 of this embodiment, the Softmax activation function is used to generate a category probability distribution, where the probability value of each category is between 0 and 1, and the sum of the probabilities of all categories is 1.

[0117] Through the Softmax classifier, a probability score can be generated for each category in the video, indicating the possibility that the video belongs to that category.

[0118] By combining deep features, Bi-LSTM networks, and Softmax classifiers, we can build a powerful action recognition system that can accurately identify various actions in videos.

[0119] Example 2

[0120] Embodiment 2 provides a field operation behavior recognition device based on an improved feature fusion mechanism, such as Figure 3 As shown, including:

[0121] A collection module, used to collect multiple video clips containing preset action features for the target operation behavior;

[0122] A first feature image set acquisition module is used to perform target detection on the video clip using the YOLOv8 algorithm, identify key spatial regions of people and equipment, intercept multiple first feature image sets containing key spatial regions of people and equipment into a first feature image set based on a segmentation attention network, and combine multiple first feature image sets into a first feature image set;

[0123] A second feature image set acquisition module, used for performing multi-layer feature fusion on features at different levels in each first feature image to acquire a second feature image, and to gather all second feature images into a second feature image set;

[0124] A final recognition image acquisition module is used to take the weighted average of the forward propagation and the backward propagation of the second feature image set as the final recognition image based on the Bi-LSTM network;

[0125] An image recognition set acquisition module, used to repeat the first feature image set acquisition module to the final recognition image acquisition module until the final recognition images corresponding to all video clips are output, and all the final recognition images are grouped into an image recognition set;

[0126] The deep learning model building module is used to build a deep learning model to perform model training on the image recognition set, and to identify the target operation behavior on site based on the trained deep learning model.

[0127] Example 3

[0128] Embodiment 3 provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned on-site operation behavior recognition method based on the improved feature fusion mechanism when executing the program.

[0129] Example 4

[0130] Embodiment 4 provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for identifying on-site operation behaviors based on the improved feature fusion mechanism is implemented.

[0131] Example 5

[0132] Embodiment 5 A computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned on-site operation behavior identification method based on the improved feature fusion mechanism.

[0133] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for identifying on-site operation behaviors based on an improved feature fusion mechanism, characterized in that: include: Step S1: collecting multiple video clips containing preset action features for the target operation behavior; Step S2: using the YOLOv8 algorithm to perform target detection on the video clip, identifying key spatial regions of people and equipment, intercepting multiple first feature images containing key spatial regions of people and equipment based on the segmentation attention network, and grouping the multiple first feature images into a first feature image set; Step S3: performing multi-layer feature fusion on features at different levels in each first feature image to obtain a second feature image, and combining all second feature images into a second feature image set; Step S4: Based on the Bi-LSTM network, a weighted average of the forward propagation and the backward propagation of the second feature image set is taken as the final recognition image; Step S5: repeating steps S2 to S4 until the final recognition images corresponding to all video clips are output, and all the final recognition images are grouped into an image recognition set; Step S6: Establish a deep learning model to perform model training on the image recognition set, and perform on-site operation behavior recognition on the target operation behavior on site based on the trained deep learning model.

2. The on-site operation behavior recognition method based on the improved feature fusion mechanism according to claim 1 is characterized in that: The method for collecting multiple video clips containing preset action features for the target operation behavior is: The historical video of the target operation behavior is shot, and the historical video is sparsely sampled to obtain multiple video clips containing preset action features.

3. The on-site operation behavior recognition method based on the improved feature fusion mechanism according to claim 1 is characterized in that: The YOLOv8 algorithm is used to detect targets in video clips, identify key spatial areas of people and equipment, and intercept multiple first feature images containing key spatial areas of people and equipment based on the segmentation attention network. The method of grouping multiple first feature images into a first feature image set is as follows: Step S201: identifying key spatial areas of people and equipment through YOLOv8 to obtain an initial recognition image set, wherein the initial recognition image set includes multiple initial recognition images; Step S202: Establish a segmentation attention network, divide the initial recognition image as input into multiple basis arrays, and divide each basis array into multiple feature map groups; Step S203: using the key spatial regions of characters and devices as full-text context information for each feature graph group to obtain the weight of each feature graph group in the channel dimension; Step S204: performing weighted processing on all feature map groups in each basis array, and obtaining channel feature maps of all basis arrays based on channel dimension soft attention; Step S205: short-circuit all channel feature atlas groups along the channel dimension to obtain the output of the segmentation attention network; Step S206: short-circuiting the output of the segmentation attention network and the initial recognition image to generate a first feature image; Step S207: repeating steps S202 to S206 until all first feature images are output, and all first feature images are grouped into a first feature image set.

4. The on-site operation behavior recognition method based on the improved feature fusion mechanism according to claim 1 is characterized in that: The method of performing multi-layer feature fusion on the features at different levels in each first feature image to obtain the second feature image is: Step S301: Establishing a ResNest backbone network; Step S302: for a first feature image, a high-dimensional convolution kernel is replaced by a plurality of low-dimensional convolution kernels; Step S303: Obtaining shallow features of the first feature image; Step S304: obtaining deep features of the first feature image; Step S305: fusing shallow features and deep features to obtain a second feature image; Step S306: repeating steps S302 to S305 until all second feature images are output, and all second feature images are grouped into a second feature image set.

5. The on-site operation behavior recognition method based on the improved feature fusion mechanism according to claim 1 is characterized in that: The method of taking the weighted average of the forward propagation and the back propagation of the second feature image set as the final recognition image based on the Bi-LSTM network is: Step S401: using global average pooling to expand the fused features in the second feature image into a one-dimensional vector; Step S402: A Bi-LSTM network is established based on the forward LSTM neural network and the backward LSTM neural network, all second feature images are input into the Bi-LSTM network in chronological order, and a weighted average of the forward propagation and the backward propagation is output as the final recognition image.

6. The method for identifying on-site operation behaviors based on an improved feature fusion mechanism according to claim 1, characterized in that: In deep learning models, the Softmax activation function is used to generate category probability distribution, where the probability value of each category is between 0 and 1, and the sum of the probabilities of all categories is 1.

7. A field operation behavior recognition device based on an improved feature fusion mechanism, characterized in that: include: A collection module, used to collect multiple video clips containing preset action features for the target operation behavior; A first feature image set acquisition module is used to perform target detection on the video clip using the YOLOv8 algorithm, identify key spatial regions of people and equipment, intercept multiple first feature image sets containing key spatial regions of people and equipment into a first feature image set based on a segmentation attention network, and combine multiple first feature image sets into a first feature image set; A second feature image set acquisition module, used for performing multi-layer feature fusion on features at different levels in each first feature image to acquire a second feature image, and to gather all second feature images into a second feature image set; A final recognition image acquisition module is used to take the weighted average of the forward propagation and the backward propagation of the second feature image set as the final recognition image based on the Bi-LSTM network; An image recognition set acquisition module, used to repeat the first feature image set acquisition module to the final recognition image acquisition module until the final recognition images corresponding to all video clips are output, and all the final recognition images are grouped into an image recognition set; The deep learning model building module is used to build a deep learning model to perform model training on the image recognition set, and to identify the target operation behavior on site based on the trained deep learning model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the on-site operation behavior recognition method based on the improved feature fusion mechanism as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying on-site operation behaviors based on an improved feature fusion mechanism as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for identifying on-site operation behaviors based on an improved feature fusion mechanism as described in any one of claims 1 to 6 is implemented.

Images