Behavior detection method and system and electronic equipment

Abstract

The invention discloses a behavior detection method and system and electronic equipment. The method comprises the following steps: acquiring a collected image; the method comprises the following steps: inputting an image into a target detection model, performing feature extraction on the image by the target detection model, segmenting features extracted from each frame of image into a plurality of paths, performing parallel processing on the plurality of paths, and fusing the obtained features to obtain a global feature vector of each frame of image; determining a key feature based on the feature of each frame of image in the continuous multiple frames of images, fusing the global feature vector and the key feature, and performing prediction by using the fused feature to obtain a prediction result; and determining whether the collected image detects the target behavior based on the prediction result.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a behavior detection method, system, and electronic device. Background Art

[0002] When displaying video streams in video surveillance or video conferencing, it is usually impossible to accurately and quickly detect various behaviors in the image. Summary of the Invention

[0003] In view of this, the present application provides a behavior detection method, system, and electronic device, the specific solutions of which are as follows:

[0004] A behavior detection method, comprising:

[0005] obtaining collected images;

[0006] Inputting the image into a target detection model, the target detection model performs feature extraction on the image, divides the features extracted from each frame of the image into multiple paths, fuses the features obtained after processing the multiple paths in parallel to obtain a global feature vector for each frame of the image, determines key features based on the features of each frame of the continuous multiple frames of the image, fuses the global feature vector and the key features, and performs prediction using the fused features to obtain a prediction result;

[0007] Based on the prediction result, it is determined whether the target behavior is detected in the collected image.

[0008] Furthermore, the target detection model extracts features from the image, divides the features extracted from each frame into multiple paths, fuses the features obtained after processing the multiple paths in parallel to obtain a global feature vector for each frame, determines key features based on the features of each frame in multiple consecutive frames, fuses the global feature vector and the key features, and performs prediction using the fused features to obtain a prediction result, including:

[0009] Performing two-dimensional feature extraction on the image using the target detection model, the two-dimensional feature extraction at least comprising: performing feature extraction on each frame of the image, dividing the extracted features into multiple paths, fusing the features obtained after parallel processing of the multiple paths, and obtaining a global feature vector for each frame of the image;

[0010] Performing three-dimensional feature extraction on the image using the target detection model, the three-dimensional feature extraction at least comprising: determining weight data of features at the same spatial position in different frames of image based on features of each frame of image in a plurality of consecutive frames, and determining key features based on the weight data of features at the same spatial position in the different frames of image;

[0011] The target detection model is used to perform feature fusion on the global feature vector and the key features, and the feature fusion at least includes: fusing the global feature vector and the key features at the channel level, and analyzing the spatial information of the global feature vector and the key features.

[0012] Furthermore, the extracting two-dimensional features of the image using the target detection model includes:

[0013] Extracting features from each frame of the image, dividing the extracted features into multiple paths, and fusing the obtained features after processing the multiple paths in parallel;

[0014] The fused features are pooled at different scales to obtain the global feature vector of each frame image.

[0015] Furthermore, the extracting two-dimensional features of the image using the target detection model further includes:

[0016] The initial global feature vector of each frame image obtained after the pooling operation is used to calculate channel attention and spatial attention to obtain all feature vectors in each frame image after attention weighting.

[0017] Furthermore, the extracting three-dimensional features of the image using the target detection model includes:

[0018] Performing a group convolution operation on the features of each frame of the continuous multiple frames in the image to obtain each group of convolution output results;

[0019] For each group of the convolution output results, weight data of features at the same spatial position in different frame images are determined to determine the key features in each group of the convolution output results.

[0020] Furthermore, determining weight data of features at the same spatial position in different frame images for each group of the convolution output results to determine key features in each group of the convolution output results includes:

[0021] Performing global average pooling on each group of convolution output results to obtain global features of each group;

[0022] Obtaining the importance coefficient of each feature in each group by dot product calculation based on the global features of each group;

[0023] Normalizing the importance coefficient of each feature in each group to determine the normalized weight coefficient of each feature in each group;

[0024] An enhanced feature vector corresponding to each feature is determined based on the activation function and the normalized weight coefficient of each feature in each group.

[0025] Furthermore, the utilizing the target detection model to perform feature fusion on the global feature vector and the key feature includes:

[0026] Fusing the global feature vector and the key feature at the channel level to obtain a fused feature;

[0027] The spatial information of the fusion feature is analyzed to obtain the fusion feature obtained after weight distribution using the spatial information.

[0028] Furthermore, the analyzing of spatial information of the fused features to obtain fused features obtained by weighting the spatial information includes:

[0029] Determining initial weights of features at different spatial positions in the fused features;

[0030] Feature processing is performed using the fusion features set with the initial weights to obtain fusion features after adjusting the initial weights of the different spatial positions.

[0031] A behavior detection system, comprising:

[0032] an acquisition unit, configured to acquire an acquired image;

[0033] an input unit, configured to input the image into a target detection model, wherein the target detection model performs feature extraction on the image, divides the features extracted from each frame of the image into multiple paths, fuses the features obtained after processing the multiple paths in parallel to obtain a global feature vector for each frame of the image, determines key features based on the features of each frame of the continuous multiple frames of the image, fuses the global feature vector and the key features, and performs prediction using the fused features to obtain a prediction result;

[0034] A determination unit is configured to determine whether a target behavior is detected in the collected image based on the prediction result.

[0035] An electronic device, comprising:

[0036] An image acquisition device, used for acquiring images;

[0037] A processor is used to obtain a captured image, input the image into a target detection model, the target detection model performs feature extraction on the image, divides the features extracted from each frame of the image into multiple paths, fuses the obtained features after parallel processing of the multiple paths to obtain a global feature vector for each frame of the image, determines key features based on the features of each frame of the continuous multiple frames of the image, fuses the global feature vector and the key features, and uses the fused features to perform prediction to obtain a prediction result; and determines whether target behavior is detected in the captured image based on the prediction result.

[0038] A computer storage medium carries one or more computer programs, which, when executed by an electronic device, can enable the electronic device to implement any of the above behavior detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of a behavior detection method disclosed in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a behavior detection method disclosed in an embodiment of the present application;

[0042] Figure 3 A flowchart of a behavior detection method disclosed in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of a behavior detection method disclosed in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of a C2f backbone network processing input features disclosed in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of a process of processing a feature graph in a dimension processing unit disclosed in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of a process of inputting a feature map into a C2f convolution structure and processing the feature map in the C2f convolution structure disclosed in an embodiment of the present application;

[0047] Figure 8A schematic diagram of a feature map disclosed in an embodiment of the present application being input into an SPPF structure for a pooling operation;

[0048] Figure 9 A schematic diagram of a processing process after an input feature is input into a CBAM module disclosed in an embodiment of the present application;

[0049] Figure 10 A schematic diagram of a feature processing flow in a C2f backbone network disclosed in an embodiment of the present application;

[0050] Figure 11 A flowchart of a behavior detection method disclosed in an embodiment of the present application;

[0051] Figure 12 Schematic diagram of a feature processing process in a ResNeXt Block structure including a spatial grouping enhanced attention network SGE disclosed in an embodiment of the present application;

[0052] Figure 13 A schematic diagram of a feature processing process of a spatial grouping enhanced attention network SGE disclosed in an embodiment of the present application;

[0053] Figure 14 A flowchart of a behavior detection method disclosed in an embodiment of the present application;

[0054] Figure 15 A schematic diagram of a process of fusing global feature vectors and key features by a feature fusion network disclosed in an embodiment of the present application;

[0055] Figure 16 A schematic diagram of a process of processing features by a self-attention module disclosed in an embodiment of the present application;

[0056] Figure 17 A schematic diagram of the completion process of a behavior detection method disclosed in an embodiment of the present application;

[0057] Figure 18 A schematic diagram of the structure of a behavior detection system disclosed in an embodiment of the present application;

[0058] Figure 19 This is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0060] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0061] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] This application discloses a behavior detection method, the flow chart of which is as follows: Figure 1 Shown, including:

[0063] Step S11, obtaining the collected image;

[0064] Step S12: Input the image into the target detection model, which performs feature extraction on the image, divides the features extracted from each frame into multiple paths, fuses the features obtained after processing the multiple paths in parallel to obtain a global feature vector for each frame, determines key features based on the features of each frame in the continuous multiple frames, fuses the global feature vector and the key features, and uses the fused features to perform prediction to obtain a prediction result;

[0065] Step S13: Determine whether the target behavior is detected in the captured image based on the prediction result.

[0066] Various behaviors may occur during video surveillance or video conferencing, such as: someone raising their hand to speak, someone writing on a whiteboard, someone pointing, group discussion, etc. When the above behaviors occur, it is usually necessary to adjust the image acquisition device so that the image acquisition device can capture the relevant behaviors. For example, in the mode of shooting a panoramic view of the conference room, when someone is detected raising their hand to speak, the image acquisition device needs to be controlled to focus on the speaker and capture the speaker's image so that the speaker's image can be highlighted; if someone is detected writing on the whiteboard, the image acquisition device needs to be controlled to focus on the whiteboard and the writer so that the images of the whiteboard and the writer can be highlighted; if someone is detected pointing, the image acquisition device needs to be controlled to capture the person performing the pointing action and the person or object they are pointing to for highlighting; if a group discussion is detected, the image acquisition device needs to be controlled to capture the global image, etc.

[0067] However, when displaying video streams in video surveillance or video conferencing, it is usually impossible to accurately and quickly detect various behaviors in the image.

[0068] Based on this, in this solution, after obtaining the image that needs to be detected for behavior, the image is input into the target detection model, and the target detection model is used to make predictions in order to determine whether the target behavior exists in the image. After the image is input into the target detection model, the target detection model extracts features from the image, and divides the features extracted from each frame of the image into multiple paths. After parallel processing, feature fusion is performed to obtain a global feature vector for each frame of the image, and the features of each frame of the continuous multiple frames of the image are used to determine the key features. By processing the features of multiple paths in parallel, the processing efficiency is improved while the diversified features can be captured more accurately, thereby ensuring the accuracy of behavior detection. The global feature vector and the key feature vector are fused to obtain the prediction result, so that the prediction result refers to both the global features and the key features, further improving the accuracy of the behavior detection result.

[0069] During video surveillance or video conferencing, the images captured by the image capture device are sent in real time to the behavior detection device corresponding to the behavior detection method disclosed in this embodiment, or directly to the target detection model.

[0070] The image acquisition device acquires images in real time. The image acquisition device may optionally support a frame rate of 30 FPS or higher and a resolution of 1080p or higher to ensure high image quality for subsequent accurate behavior recognition.

[0071] Before inputting the collected image into the target detection model for detection, or before inputting the collected image into the target detection model, the image can first be denoised, such as by mean filtering or Gaussian filtering, to reduce the impact of environmental noise on the accuracy of behavior recognition; in addition, in the process of processing the image, in order to reduce the impact of vibration or jitter caused by the image acquisition device during image acquisition, image stabilization processing can be added to ensure image stability.

[0072] The target detection model may be a YOWOv2 behavior recognition model, which can efficiently process video stream data, detect and recognize various behaviors appearing in the video, so that when a target behavior is detected, a response can be made based on the target behavior.

[0073] After obtaining the input image, the target detection model performs feature extraction and then segments the extracted features so that the segmented images can be processed separately and simultaneously using multiple paths to improve the efficiency of image processing while ensuring that as many features as possible are extracted from the image. After processing the features in multiple paths in parallel, the features obtained by processing the multiple paths are fused to obtain the global feature vector of each frame of the image, that is, to obtain the global information of each frame of the image, so as to ensure that the features extracted from each frame of the image are more comprehensive.

[0074] In addition, key features are determined based on the features of each frame in multiple consecutive frames of images to achieve the purpose of suppressing irrelevant features.

[0075] Afterwards, the global feature vector and the key features are fused to obtain the fused features. The fused features not only ensure the comprehensiveness of the features, but also avoid the interference of non-key features. Based on the fused features, the behaviors included in the image can be identified more accurately, thereby determining whether the target behavior exists.

[0076] Responding based on the target behavior may specifically include adjusting the image acquisition area of ​​the image acquisition device based on the target behavior.

[0077] For example: when the target detection model identifies target behaviors related to speaking, such as raising hands and standing, the image acquisition area of ​​the image acquisition device is controlled to match the position of the speaker; for another example: when the target detection model detects target behaviors related to the whiteboard, the image acquisition area of ​​the image acquisition device is controlled to be divided into at least two parts, one part is the picture of the whiteboard, and the other part is the image of the operator writing or performing other operations on the whiteboard; for another example: when the target detection model determines that an object performs a related target behavior such as a pointing operation, the pointing direction is detected, the image acquisition device is controlled to capture the picture of the object performing the pointing operation, and switches along the direction of the pointing operation to the image corresponding to the target corresponding to the pointing operation, etc.

[0078] Specifically, the complete behavior detection method disclosed in this embodiment can be as follows: Figure 2As shown, it includes: image acquisition device acquisition; obtaining video stream; inputting the video stream into the target detection model; the target detection model analyzes to obtain target behavior, and the target behavior may include at least one of the following: speaking behavior, writing behavior, pointing behavior, collective behavior, other behaviors, etc.; for speaking behavior, it is necessary to determine the speaker's image and display it; for writing behavior, it is necessary to determine the images of the writer and the whiteboard and display them on multiple screens; for pointing behavior, at least the image of the pointed target needs to be determined and displayed; for collective behavior, if the collective behavior is a discussion behavior, multiple groups of discussion images need to be displayed on multiple screens; if the collective behavior is a silent behavior, it can be directly displayed globally; for other behaviors, behavior statistics are directly performed.

[0079] It should be noted that when the image acquisition device is controlled to simultaneously acquire images of multiple different areas based on the detected target behavior, the acquired images of multiple areas can be displayed on the display screen at the same time. Alternatively, when it is necessary to display images of multiple different areas at the same time, the priority order of the target images can be set, and the images of multiple different areas can be displayed in sequence according to the priority order. Alternatively, the images of multiple different areas can be displayed at the same time, but the display positions or display area sizes on the display screen are allocated to the images of different areas according to the priority order, such as the priority of the whiteboard image is higher than the priority of the pointed target image, and the priority of the pointed target image is higher than the priority of the speaker image, etc., so as to highlight or display images with higher priority based on the priority order.

[0080] The behavior detection method disclosed in this embodiment, when it is necessary to detect whether the target behavior exists, inputs the collected image into the target detection model, uses the target detection model to extract features from the image, divides the extracted features into multiple paths and processes them in parallel, fuses the features after parallel processing to obtain a global feature vector for each frame of the image, and determines key features based on the features of each frame in multiple consecutive frames of images. The global feature vector and the key features are then fused, and prediction is performed using the fused features to obtain a prediction result, thereby determining whether the target behavior is detected in the collected image. This solution can more accurately capture diverse features by dividing the image features into multiple paths and processing them in parallel, and then fusing the results, thereby improving the accuracy of behavior detection in the image and facilitating accurate adjustment of the picture content.

[0081] This embodiment discloses a behavior detection method, the flow chart of which is as follows: Figure 3 Shown, including:

[0082] Step S31, obtaining a captured image;

[0083] Step S32: Input the image into the target detection model and use the target detection model to perform two-dimensional feature extraction on the image. The two-dimensional feature extraction at least includes: extracting features from each frame of the image, dividing the extracted features into multiple paths, fusing the features obtained after parallel processing of the multiple paths, and obtaining a global feature vector for each frame of the image;

[0084] Step S33: performing three-dimensional feature extraction on the image using the object detection model, wherein the three-dimensional feature extraction at least includes: determining weight data of features at the same spatial position in different frames based on the features of each frame in the continuous multiple frames, and determining key features based on the weight data of features at the same spatial position in different frames;

[0085] Step S34: using the target detection model to perform feature fusion on the global feature vector and the key features, wherein the feature fusion at least includes: fusing the global feature vector and the key features at the channel level, and analyzing the spatial information of the global feature vector and the key features;

[0086] Step S35: Use the fused features to perform prediction and obtain a prediction result to determine whether the target behavior is detected in the collected image.

[0087] After the image to be detected for target behavior is input into the target detection model, the target detection model performs feature extraction, feature fusion and prediction on the image.

[0088] Specifically, in the feature extraction stage, it can include two-dimensional feature extraction and three-dimensional feature extraction. The two-dimensional feature extraction can extract the global feature vector of each frame image, and the three-dimensional feature extraction can extract the key features in multiple consecutive frames of images, so as to use the global feature vector of each frame image and the key features in multiple consecutive frames of images to perform feature fusion, so as to ensure that the features obtained after fusion include both the features in each frame image and the key features determined based on adjacent frame images, thereby ensuring the integrity of the fused features and avoiding missing key features.

[0089] Among them, two-dimensional feature extraction can be achieved through the C2f backbone network in the target detection model. The C2f backbone network is a convolutional feature fusion network, which can not only perform convolution operations to reduce the amount of calculation, but also segment the feature map and process the segmented features in parallel through multiple paths and then fuse them to obtain the global feature vector of each frame image, thereby improving the quality and quantity of the detected features, thereby achieving the purpose of improving the accuracy of target detection.

[0090] The C2f backbone network is mainly used to extract features from each individual frame of the image to obtain a global feature vector. When the input to the target detection model is a continuous multi-frame image, the C2f backbone network needs to extract a global feature network for each frame in the continuous multi-frame image.

[0091] In addition, the extraction of three-dimensional features can be achieved through the grouped residual network in the target detection model. The grouped residual network can be specifically a ResNeXt-101 network, which can extract spatiotemporal features from video clips (i.e., continuous multi-frame images), that is, extract key features from continuous multi-frame images. The key features can be: features that change in continuous multi-frame images.

[0092] When extracting key features from multiple consecutive frames of images, the ResNeXt-101 network can not only process all the frame images as a whole to extract the spatiotemporal features in the continuous multiple frames, but also perform differentiated feature extraction based on the differences between different frames to ensure that the key features extracted by the ResNeXt-101 network are more accurate and comprehensive, thereby optimizing the target detection model's ability to recognize different targets in the scene, and also improving the target detection model's ability to extract detailed features in the image and related features in different frame images.

[0093] Of course, the grouped residual network can be a ResNeXt-101 network or other networks, and is not specifically limited here.

[0094] The feature fusion network in the target detection model can be used to fuse the global feature vector and key features. The feature fusion network is used to fuse the global feature vector and key features to the level, and the spatial information is analyzed to obtain the fused features. The analysis of spatial information effectively improves the ability to accurately locate the target in the image and improves the performance of the target detection model in identifying target behavior.

[0095] The behavior detection method disclosed in this embodiment, after obtaining the collected image and inputting it into the target detection model, uses the target detection model to extract two-dimensional features to obtain a global feature vector for each frame of the image, extracts three-dimensional features to obtain key features, and performs channel-level and spatial-level feature fusion on the global feature vector obtained by the two-dimensional feature extraction and the key features obtained by the three-dimensional feature extraction to enhance the ability to capture spatial information. It can more accurately capture important features in the image, thereby improving the accuracy of behavior detection in the image and facilitating accurate adjustment of the picture content.

[0096] This embodiment discloses a behavior detection method, the flow chart of which is as follows: Figure 4 Shown, including:

[0097] Step S41: obtaining a captured image;

[0098] Step S42: Input the image into the target detection model, use the target detection model to extract features from each frame of the image, divide the extracted features into multiple paths, and fuse the features obtained after processing the multiple paths in parallel;

[0099] Step S43: Using the target detection model, the fused features are pooled at different scales to obtain a global feature vector for each frame of the image.

[0100] Step S44: performing three-dimensional feature extraction on the image using the target detection model, wherein the three-dimensional feature extraction at least includes: determining weight data of features at the same spatial position in different frames based on the features of each frame in the continuous multiple frames, and determining key features based on the weight data of features at the same spatial position in different frames;

[0101] Step S45: using the target detection model to perform feature fusion on the global feature vector and the key features, wherein the feature fusion at least includes: fusing the global feature vector and the key features at the channel level, and analyzing the spatial information of the global feature vector and the key features;

[0102] Step S46: Use the fused features to perform prediction and obtain a prediction result to determine whether the target behavior is detected in the collected image.

[0103] In the behavior detection method disclosed in this embodiment, a C2f backbone network is used to extract two-dimensional features of an image. The C2f backbone network may specifically include: a C2f convolution structure and an SPPF structure.

[0104] Specifically, the input features are input to the C2f backbone network, and the schematic diagram of the C2f backbone network processing the input features can be shown as follows: Figure 5 As shown, it includes: first performing a convolution operation, inputting the features obtained after the convolution operation into the C2f convolution structure for feature segmentation, parallel processing and fusion of multiple paths, performing a convolution operation again, and inputting the features obtained after the convolution operation into the C2f convolution structure again. After multiple convolutions and processing of the C2f convolution structure, the results are input into the SPPF structure for pooling operation to obtain the final global feature vector.

[0105] Among them, the C2f convolution structure is a convolution structure used in the C2f backbone network to realize convolution, feature segmentation, parallel processing and fusion of multiple paths.

[0106] Feature segmentation can be achieved through the split operation Split, which divides the input features into different parts along the channel dimension, and extracts features from the divided parts separately by different processing paths. This segmentation is conducive to parallel processing within the network, so that each path can extract a certain type of feature, and different paths extract different types of features, so as to achieve the purpose of improving feature extraction accuracy.

[0107] Furthermore, the C2f convolution structure in the target detection model in this embodiment may also include: a dimensionality processing unit, such as a Bottleneck unit, which can reduce the amount of computation by reducing the number of channels of the feature map (such as dimensionality reduction through 1×1 convolution), and then increase the dimensionality through 1×1 convolution. The dimensionality processing unit can reduce the computational overhead while maintaining network performance.

[0108] like Figure 6 As shown in the figure, it is a schematic diagram of the processing process of the feature map in the dimensional processing unit. The feature is input into the dimensional processing unit and reaches the first convolution conv in the dimensional processing unit, which can be a 1×1 convolution. The first convolution conv is used for dimensionality reduction, that is, the number of channels is compressed, such as reducing the number of channels from C to C / 2 to reduce the subsequent calculation amount; after dimensionality reduction, it is input into the second convolution conv, and feature extraction is performed at the second convolution conv, that is, features are extracted in low-dimensional space, retaining core information, and after extracting features at the second convolution conv, a dimensionality increase operation is performed to restore the number of channels from C / 2 to C; then a residual connection is performed, and the input features of the first convolution layer and the output features of the second convolution layer are added element by element and output. The number of channels of the final output feature map is consistent with the number of channels of the input feature map. At the same time, the output feature map contains multi-scale features.

[0109] In addition, after using multiple paths to process the segmented features in parallel and obtain the processed features respectively, feature fusion is required, that is, the features after convolution processing, the features after segmentation, and the features after dimension processing are spliced ​​together. At this time, the splicing structure Concat can be used to implement it, which can realize the splicing of the features obtained after processing each path in multiple paths together to obtain the fused image.

[0110] Specifically, the feature map disclosed in this embodiment is input into the C2f convolution structure, and the processing process of the feature map in the C2f convolution structure can be as follows: Figure 7 As shown, the feature map needs to go through the first layer of convolution conv, the segmentation operation Split, the dimensional processing performed by n dimensional processing units Bottleneck, the feature fusion Concat and the second layer of convolution conv, where n is a positive integer.

[0111] After the feature map is processed by the C2f convolution structure, the effect of feature extraction can be enhanced. Specifically, by dividing the input features into multiple paths, performing parallel processing, and fusing the results, diversified features can be obtained more accurately. The split operation Split in the C2f convolution structure and the dimensional processing operation of the dimensional processing unit Bottleneck can achieve more efficient feature expression.

[0112] After the feature map is processed by the C2f convolution structure, the processed features can be input into the SPPF structure, and pooling operations of different scales can be performed through the SPPF structure.

[0113] The SPPF structure is a spatial pyramid pooling operation, which is mainly used for target detection and computer vision tasks. It can improve the efficiency of feature extraction while maintaining the ability of multi-scale feature fusion.

[0114] The pooling operation is usually performed after the convolution layer. It can slide a window of fixed size on the feature map and calculate the pixel values ​​in each window to obtain a feature map with a smaller dimension. The feature map retains the main information of the feature map input to the SPPF structure, while reducing the amount of data and calculation. The feature map obtained by the window is used for feature fusion operation, so that the features output by the SPPF structure include not only features of different scales, but also global and local detail features.

[0115] The schematic diagram of the feature map input to the SPPF structure for pooling operation can be shown as follows Figure 8 As shown in the figure, it includes: first, a convolution operation is performed through the first convolution layer Conv, then multiple maximum pooling Max Pools are performed, and then the feature maps after multiple maximum pooling are fused Concat, and finally the fused features are convolved through the second convolution layer Conv to output the SPPF structure.

[0116] The behavior detection method disclosed in this embodiment inputs an image into a target detection model. When the target detection model is used to extract two-dimensional features from the image, feature extraction is first performed on each frame of the image. The extracted features are then divided into multiple paths. After processing the multiple paths in parallel, the obtained features are fused. The fused features are pooled at different scales to obtain a global feature vector for each frame of the image, thereby realizing the process of extracting two-dimensional features from the image. In this embodiment, the process of extracting two-dimensional features from the image can more accurately capture diverse features by dividing the features into multiple paths and processing them in parallel before fusing the features. In addition, the pooling of the fused features at different scales can enable the target detection model to obtain both global and local features simultaneously, thereby improving the recognition accuracy of the target detection model.

[0117] Furthermore, in the behavior detection method disclosed in this embodiment, when extracting two-dimensional features from an image, the following steps may be further included:

[0118] The initial global feature vector of each frame obtained after the pooling operation is used to calculate the channel attention and spatial attention to obtain all the feature vectors in each frame after attention weighting.

[0119] That is, the C2f backbone network can also include a convolutional attention module, which uses the convolutional attention module to calculate the channel attention, i.e., spatial attention, of the initial global feature vector to obtain all feature vectors, i.e., the final global feature vector.

[0120] The convolutional attention module is used to enhance the feature extraction capability of the C2f backbone network. The convolutional attention module can specifically adopt the CBAM module (Convolutional Block Attention Module, a lightweight attention mechanism module), which combines channel attention and spatial attention. By dynamically adjusting the feature weights in the channel and spatial dimensions, it highlights key information and suppresses redundant information to enhance the network's feature expression capability.

[0121] The CBAM module can be composed of two parts, namely the Channel Attention Module (CAM) and the Spatial Attention Module (SAM), as shown in Figure 9 Figure 2 shows the processing of input features after they are fed into the CBAM module. By combining modeling channels and spatial attention, the network's ability to perceive important features is enhanced, thereby strengthening both the feature representation capability and the network's generalization capabilities. For example, when an image is fed into an object detection model, its C2f network architecture can identify a blurred person in the image. This image is then fed into the CBAM module, where it is processed to identify a clear portrait of the person.

[0122] If the size of the input feature F input to the CBAM module is C×H×W, where C is the number of channels, H is the feature height, and W is the feature width, the channel attention module CAM first calculates the channel attention of the input feature F to obtain the feature F'. In the channel attention module, each channel of the feature map is the result after feature extraction. Therefore, the calculation of channel attention can enhance important features and suppress unimportant features, thereby accurately identifying key information and adjusting the feature weights so that the feature weights can conform to the current input features to improve the performance of the model.

[0123] Within the channel attention module, channel-level statistical features can be first extracted through global average pooling and global maximum pooling, and then the pooling results are input into the multi-layer perceptron to generate a channel weight vector. Finally, the activation function is used to normalize the channel weights to the target range, such as [0, 1], to enhance important channels and suppress noisy channels; then the channel weights are feature-weighted with the feature map to obtain F'.

[0124] After the feature F' obtained by the channel attention module CAM is input into the spatial attention module SAM, the spatial attention module SAM calculates the spatial attention of the feature F' to obtain the output feature F''. The size of the output feature F'' is the same as the input feature F, which is still C×H×W.

[0125] Inside the spatial attention module, the channel-weighted feature F' needs to be average pooled and max-pooled along the channel dimension to obtain two feature maps. After splicing the two feature maps, a spatial weight map is generated through convolution, and the spatial weight is normalized using an activation function. Finally, the spatial weight is weighted with the channel-weighted feature F' to obtain the output feature F''.

[0126] In the spatial attention module, the spatial correlation of features is used to generate a spatial weight map. For each prominent important channel, the spatial attention mechanism is used to extract the important features in the channel, so as to extract a special local spatial feature in the whole, so that the model can pay more attention to this feature when processing the image.

[0127] Among them, it should be noted that the process of using the CBAM module to calculate the channel attention and spatial attention of the features can be executed after the C2f backbone network performs a pooling operation to obtain the initial global feature vector, so as to update the initial global feature vector and obtain the final global feature vector; it can also be executed in the process of the C2f backbone network processing the features, that is, in the process of the C2f backbone network processing the features, it will undergo multiple C2f convolution structure processing. After each C2f convolution structure outputs the features, the channel attention and spatial attention calculation process is executed once. After the last processing of the C2f convolution structure, the pooling operation is first performed, and after the pooling operation, the channel attention and spatial attention are calculated. After the calculation is completed, the final global feature vector is obtained.

[0128] The specific processing process can be as follows Figure 10As shown in the figure, the input feature is input to the C2f backbone network, and the convolution operation is performed first. After that, it is processed by the C2f convolution structure, and then the channel attention and spatial attention are calculated by the CBAM module. After the calculation is completed, the convolution operation and C2f convolution structure processing are performed again. After that, the channel attention and spatial attention are calculated by the CBAM module again. After the calculation is completed, the convolution operation and C2f convolution structure processing are continued, and the channel attention and spatial attention are calculated by the CBAM module again. After the calculation is completed, the convolution operation and C2f convolution structure processing are continued, and the channel attention and spatial attention are calculated by the CBAM module. After the calculation is completed, the convolution operation and C2f convolution structure processing are continued, and the pooling operation is performed through the SPPF structure to obtain the final global feature vector.

[0129] This embodiment discloses a behavior detection method, the flow chart of which is as follows: Figure 11 Shown, including:

[0130] Step S111, obtaining a captured image;

[0131] Step S112: Input the image into the target detection model and use the target detection model to extract two-dimensional features of the image. The two-dimensional feature extraction at least includes: extracting features from each frame of the image, dividing the extracted features into multiple paths, fusing the features obtained after parallel processing of the multiple paths, and obtaining a global feature vector for each frame of the image.

[0132] Step S113: using the target detection model to perform group convolution operations on the features of each frame of the continuous multiple frames in the image to obtain each group of convolution output results;

[0133] Step S114: using the target detection model to determine weight data of features at the same spatial position in different frame images for each set of convolution output results, so as to determine key features in each set of convolution output results;

[0134] Step S115: using the target detection model to perform feature fusion on the global feature vector and the key features, wherein the feature fusion at least includes: fusing the global feature vector and the key features at the channel level, and analyzing the spatial information of the global feature vector and the key features;

[0135] Step S116: Use the fused features to perform prediction and obtain a prediction result to determine whether the target behavior is detected in the collected image.

[0136] In the behavior detection method disclosed in this embodiment, a grouped residual network is used to extract the three-dimensional features of the image. The grouped residual network can be specifically a ResNeXt-101 network. The grouped residual network may include: a basic residual network structure and a spatial grouping enhanced attention network. The basic residual network structure is used to perform group convolution operations on the features of each frame image in multiple consecutive frames in the image to obtain each group of convolution output results; the spatial grouping enhanced attention network is used to determine the weight data of the features in the same spatial position in different frame images for each group of convolution output results to determine the key features in each group of convolution output results.

[0137] The basic residual network structure can be composed of multiple ResNeXt Block structures. Each ResNeXt Block structure contains 32 branches. Each branch contains a convolution layer, a normalization layer, and an activation layer. The ResNeXt Block structure enhances feature expression capabilities while maintaining computational efficiency through the synergy of grouped convolution and residual connections.

[0138] Grouped convolution is to divide the feature image into 32 branches, each branch as a group, and each group performs convolution operation independently, which expands the diversity of features by increasing the grouping method; before and after the grouped convolution, dimensionality reduction and dimensionality increase can be achieved through 1×1 convolution, that is, dimensionality reduction is performed first, then feature extraction, and then dimensionality increase. Dimensionality reduction is used to reduce the amount of computation and avoid high-order feature redundancy. Dimension increase is used to restore the number of channels to match the input dimension of the residual connection, ensuring lossless information transmission.

[0139] Each ResNeXt Block structure directly superimposes the input to the output through a residual connection, retaining the original feature details and solving the gradient disappearance problem of deep networks.

[0140] As for the Spatial Group-wise Enhance (SGE) attention network, it can assign differentiated weights to features at the same spatial position in different video frames, allowing the model to focus on key features at a specific scale while effectively suppressing irrelevant features, improving the model's understanding and utilization of multi-scale information, and enhancing the model's ability to handle complex scenes.

[0141] Since the grouped residual network includes: a basic residual network structure and a spatial grouping enhanced attention network, the basic residual network structure is composed of multiple ResNeXt Block structures, each ResNeXt Block structure contains 32 branches, and each branch can correspond to a spatial grouping enhanced attention network SGE. The feature processing process in a ResNeXt Block structure containing a spatial grouping enhanced attention network SGE can be as follows: Figure 12 As shown, in Figure 12In the example, a 256-dimensional image is input and divided into 32 branches, with each column as a branch. In each branch, the image can be divided into blocks with a dimension of 256, a size of 1×1, and a channel of 4, that is, a "256, 1×1, 4" block; then dimensionality reduction is performed, that is, the 256 dimension is reduced to 4 dimensions, and the block size is adjusted to "3×3", that is, the amount of information in each block is increased, that is, a "4, 3×3, 4" block is obtained; then, the block size is adjusted back to "1×1", that is, a "4, 1×1, 256" block is obtained; the "4, 1×1, 256" block obtained from each branch is input into the corresponding spatial grouping enhanced attention network SGE, and differentiated weights are configured; finally, all features are fused twice, where the first feature fusion is to fuse the features output by all branches, and the second fusion is to fuse the features after the first fusion with the features of the input image, and finally a feature map with a dimension of 256 is output.

[0142] Specifically, the spatial grouping enhanced attention network SGE determines the weight coefficients of the features at the same spatial position in different frame images for each set of convolution output results to determine the key features in each set of convolution output results, which can be specifically as follows:

[0143] Perform global average pooling on each group of convolution output results to obtain the global features of each group; obtain the importance coefficient of each feature in each group through dot product calculation based on the global features of each group; normalize the importance coefficient of each feature in each group to determine the normalized weight coefficient of each feature in each group; determine the enhanced feature vector corresponding to each feature based on the activation function and the normalized weight coefficient of each feature in each group.

[0144] Take the feature map of size C×H×W as an example, where C is the number of channels, H is the feature height, and W is the feature width. It is divided into G groups along the channel dimension, and the position of each group in space is represented by a vector, which is , where C / G represents the dimension of each channel (i.e., the length of each feature vector), R is a real number set, and m represents the input size, i.e., the value of H×W. If it is a 256×256 image, then m=256×256, that is, decomposing the image into 256×256 small units, x i Represents each small unit.

[0145] Ke Ru Figure 13As shown in the figure, it is a schematic diagram of the feature processing process of the spatial grouping enhanced attention network SGE. The feature map is input to the spatial grouping enhanced attention network SGE, and the size of the input feature map is C×H×W. Inside the spatial grouping enhanced attention network SGE, the input feature map is firstly global average pooled using the spatial average function Φ(·) (in Figure 13 In the example, G is used to integrate the information of the entire group space, so as to deepen the learning of the meaning features in the key areas and reduce the noise interference. The formula of global average pooling can be:

[0146]

[0147] Among them, g is the global feature.

[0148] After obtaining the global feature g, the importance coefficient corresponding to each feature is obtained by dot product calculation, that is, the global feature and each feature are dot-producted, which is equivalent to m small units performing dot product with the global feature to obtain the importance coefficient of each small unit (that is, the degree of similarity between the global and the local), then each position has , where c i is the importance coefficient; then, the importance coefficient c is spatially i Normalization is performed to avoid the influence of coefficient deviation between different samples. The normalization formula can be: , where γ and β are used to scale and shift the normalized values ​​to get x i The normalized representation of a i , is the normalized weight coefficient. After obtaining the weight coefficient, it is necessary to process the weight coefficient based on the activation function to obtain the enhanced feature vector corresponding to each enhanced feature. Finally, the feature group enhanced by the spatial grouping enhanced attention network SGE can be expressed as: .

[0149] The behavior detection model disclosed in this embodiment inputs an image into a target detection model. When the target detection model extracts three-dimensional features from the image, it performs a grouped convolution operation on the features of each of the multiple consecutive frames in the image to obtain each set of convolution output results. For each set of convolution output results, weight data for features at the same spatial location in different frames is determined to identify the key features in each set of convolution output results. By assigning differentiated weights to features at the same spatial location in different frames, this solution emphasizes important features and suppresses irrelevant features, thereby enhancing the ability to identify key features and improving the ability to recognize different targets in the image.

[0150] This embodiment discloses a behavior detection method, the flow chart of which is as follows: Figure 14 Shown, including:

[0151] Step S141: obtaining a captured image;

[0152] Step S142: Input the image into the target detection model and use the target detection model to perform two-dimensional feature extraction on the image. The two-dimensional feature extraction at least includes: performing feature extraction on each frame of the image, dividing the extracted features into multiple paths, fusing the features obtained after parallel processing of the multiple paths, and obtaining a global feature vector for each frame of the image;

[0153] Step S143: performing three-dimensional feature extraction on the image using the object detection model, wherein the three-dimensional feature extraction at least includes: determining weight data of features at the same spatial position in different frames based on the features of each frame in the continuous multiple frames, and determining key features based on the weight data of features at the same spatial position in different frames;

[0154] Step S144: using the target detection model to fuse the global feature vector and key features at the channel level to obtain fused features;

[0155] Step S145: Analyze the spatial information of the fused features using the target detection model to obtain fused features obtained by weighting the spatial information;

[0156] Step S146: Use the fused features to perform predictions and obtain prediction results to determine whether the target behavior is detected in the captured image.

[0157] When an image is input into a target detection model and the target detection model is used to determine whether there is target behavior in the image, the target detection model extracts two-dimensional features from the image to obtain a global feature vector. At the same time, the target detection model extracts three-dimensional features from the image to obtain key features. Afterwards, the global feature vector is fused with the key features so that the fused features can be used for prediction to determine whether the target behavior is detected in the image.

[0158] Specifically, when the feature fusion network in the target detection model is used to fuse the global feature vector with the key features, the channel fusion module and attention module in the feature fusion network can be used to perform channel-level fusion; and the self-attention module SA in the feature fusion network can be used to analyze the spatial information to obtain the final fused features.

[0159] The schematic diagram of the process of feature fusion network fusing global feature vectors and key features can be shown as follows: Figure 15 As shown, determine the global eigenvector and set the global eigenvector to , determine the key features, set the key features as , the global feature vector and key features are input to the channel fusion module, and the channels can be aligned first, such as adjusting the number of channels of the global feature vector and key features through 1×1 convolution to make the number of channels of the two the same, and then performing feature fusion, which can be spliced ​​along the channel dimension, and finally outputting the fused features. If a specific number of channels needs to be output The features can be adjusted through 1×1 convolution to achieve the purpose of dimensionality reduction or dimensionality increase, so that the final channel fusion module outputs a channel number of The characteristics of ;

[0160] Afterwards, the features output by the channel fusion module Input to the attention module, so that the key information of the features fused by the channel fusion module can be enhanced through the attention module to achieve the purpose of suppressing redundancy or noise. The features output by the channel fusion module After being enhanced by the attention module, the output features ;

[0161] Attention module output features To the self-attention module, analyze the spatial information, assign an initial weight to each spatial position of the feature map, and in the process of feature processing by the self-attention module, the initial weight changes to obtain the feature after weight distribution based on spatial information , which is the fused feature.

[0162] That is, the initial weights of the features at different spatial positions in the fusion feature are determined; the fusion feature with the initial weights is used to perform feature processing to obtain the fusion feature after adjusting the initial weights at different spatial positions.

[0163] Specifically, the schematic diagram of the process of processing features by the self-attention module disclosed in this embodiment can be shown as follows: Figure 16As shown in the figure, when the input feature of the self-attention module is C×H×W, the input feature can first be convolved with 7×7, and normalized and activated. The Relu activation function can activate important information and suppress useless information, thereby obtaining a feature map with a channel number of C / r, that is, C / r×H×W, where r is the decay rate; then, the feature map with a channel number of C / r is convolved with 7×7, normalized and function, and the number of channels is restored to C. At this time, the feature map obtained is C×H×W; finally, the Sigmoid activation function is applied to activate and multiply the feature with the feature in the C×H×W feature map obtained after the second 7×7 convolution, normalization and function to obtain the final output feature. The weights of different spatial positions between the output features of the self-attention module and the input features input to the self-attention module are different. The above method can adjust the initial weights of different spatial positions in the input features input to the self-attention module. That is, the self-attention module calculates the attention weight based on the existing feature map, improving the model's ability to capture spatial information without significantly increasing the model complexity and number of parameters.

[0164] The behavior detection method disclosed in this embodiment has a complete process diagram as shown in FIG. Figure 17 As shown, each frame of the image is input into the convolutional feature fusion network for two-dimensional feature extraction to obtain a global feature vector, and the category feature and regression feature can be obtained based on the global feature vector; multiple consecutive frames of images are input into the grouped residual network for three-dimensional feature extraction to obtain key features; the category features, regression features and key features are input into the feature fusion network for feature fusion; the fused features are used for prediction, and category prediction can be performed, such as determining whether the target in a certain area of ​​the image is a person or an object; regression prediction can be performed, such as determining the position of a certain box in the image; and confidence prediction can also be performed, such as predicting that the probability of the target in the box at a certain position being a person is 90%.

[0165] The behavior detection method disclosed in this embodiment inputs an image into a target detection model. After using the target detection model to extract two-dimensional and three-dimensional features from the image, the extracted global feature vectors and key features are fused at the channel level to obtain fused features. The fused features are then analyzed for spatial information to obtain fused features after weighting using spatial information. The fused features can then be used to predict whether the target behavior exists in the image. This solution enhances the learning of spatial information by fusing global feature vectors and key features at the channel level while fully utilizing spatial information, thereby improving the target detection model's ability to identify important areas in the image and thereby achieving improved behavior detection accuracy.

[0166] This embodiment discloses a behavior detection system, and its structural diagram is shown in FIG. Figure 18 Shown, including:

[0167] Acquisition unit 181, input unit 182 and determination unit 183.

[0168] The obtaining unit 181 is used to obtain the collected image;

[0169] The input unit 182 is used to input the image into the target detection model. The target detection model extracts features from the image, divides the features extracted from each frame into multiple paths, fuses the features obtained after processing the multiple paths in parallel to obtain a global feature vector for each frame, determines key features based on the features of each frame in the continuous multiple frames, fuses the global feature vector and the key features, and uses the fused features to perform prediction to obtain a prediction result.

[0170] The determination unit 183 is configured to determine whether the target behavior is detected in the captured image based on the prediction result.

[0171] The behavior detection system disclosed in this embodiment is implemented based on the behavior detection method disclosed in the above embodiments, and will not be described in detail here.

[0172] The behavior detection system disclosed in this embodiment, when it is necessary to detect whether the target behavior exists, inputs the collected image into the target detection model, uses the target detection model to extract features from the image, divides the extracted features into multiple paths and processes them in parallel, fuses the features after parallel processing to obtain a global feature vector for each frame of the image, and determines key features based on the features of each frame in multiple consecutive frames of images. The global feature vector and the key features are then fused, and prediction is performed using the fused features to obtain a prediction result, thereby determining whether the target behavior is detected in the collected image. By dividing the image features into multiple paths and processing them in parallel, and then fusing the results, this solution can more accurately capture diverse features, thereby improving the accuracy of behavior detection in the image and facilitating accurate adjustment of the image content.

[0173] This embodiment discloses an electronic device, the structural diagram of which is shown in FIG. Figure 19 Shown, including:

[0174] Image acquisition device 191 and processor 192.

[0175] The image acquisition device 191 is used to acquire images;

[0176] Processor 192 is used to obtain the collected image, input the image into the target detection model, and the target detection model extracts features from the image, divides the features extracted from each frame of the image into multiple paths, and fuses the obtained features after processing the multiple paths in parallel to obtain a global feature vector for each frame of the image, determines key features based on the features of each frame of the continuous multiple frames of the image, fuses the global feature vector and the key features, and uses the fused features to perform prediction to obtain a prediction result; based on the prediction result, it is determined whether the target behavior is detected in the collected image.

[0177] The electronic device disclosed in this embodiment is implemented based on the behavior detection method disclosed in the above embodiments, which will not be described in detail here.

[0178] The electronic device disclosed in this embodiment, when it is necessary to detect whether the target behavior exists, inputs the collected image into a target detection model, uses the target detection model to extract features from the image, divides the extracted features into multiple paths and processes them in parallel, fuses the features after parallel processing to obtain a global feature vector for each frame of the image, and determines key features based on the features of each frame in multiple consecutive frames of images. The global feature vector and the key features are then fused, and prediction is performed using the fused features to obtain a prediction result, thereby determining whether the target behavior is detected in the collected image. This solution can more accurately capture diverse features by dividing the image features into multiple paths and processing them in parallel, and then fusing the results, thereby improving the accuracy of behavior detection in the image and facilitating accurate adjustment of the screen content.

[0179] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any behavior detection method provided in the embodiment of the present application.

[0180] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any behavior detection method provided in the embodiment of the present application.

[0181] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0182] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0183] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0184] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A behavior detection method, comprising: obtaining collected images; Inputting the image into a target detection model, the target detection model performs feature extraction on the image, divides the features extracted from each frame of the image into multiple paths, fuses the features obtained after processing the multiple paths in parallel to obtain a global feature vector for each frame of the image, determines key features based on the features of each frame of the continuous multiple frames of the image, fuses the global feature vector and the key features, and performs prediction using the fused features to obtain a prediction result; Based on the prediction result, it is determined whether the target behavior is detected in the collected image.

2. The method according to claim 1, wherein the target detection model extracts features from the image, divides the features extracted from each frame of the image into multiple paths, fuses the features obtained after processing the multiple paths in parallel to obtain a global feature vector for each frame of the image, determines key features based on the features of each frame of the continuous multiple frames of the image, fuses the global feature vector and the key features, and performs prediction using the fused features to obtain a prediction result, comprising: Performing two-dimensional feature extraction on the image using the target detection model, the two-dimensional feature extraction at least comprising: performing feature extraction on each frame of the image, dividing the extracted features into multiple paths, fusing the features obtained after parallel processing of the multiple paths, and obtaining a global feature vector for each frame of the image; Performing three-dimensional feature extraction on the image using the target detection model, the three-dimensional feature extraction at least comprising: determining weight data of features at the same spatial position in different frames of image based on features of each frame of image in a plurality of consecutive frames, and determining key features based on the weight data of features at the same spatial position in the different frames of image; The target detection model is used to perform feature fusion on the global feature vector and the key features, and the feature fusion at least includes: fusing the global feature vector and the key features at the channel level, and analyzing the spatial information of the global feature vector and the key features.

3. The method according to claim 2, wherein the extracting two-dimensional features from the image using the target detection model comprises: Extracting features from each frame of the image, dividing the extracted features into multiple paths, and fusing the obtained features after processing the multiple paths in parallel; The fused features are pooled at different scales to obtain the global feature vector of each frame image.

4. The method according to claim 3, wherein the step of extracting two-dimensional features from the image using the target detection model further comprises: The initial global feature vector of each frame image obtained after the pooling operation is used to calculate channel attention and spatial attention to obtain all feature vectors in each frame image after attention weighting.

5. The method according to claim 2, wherein the extracting three-dimensional features of the image using the target detection model comprises: Performing a group convolution operation on the features of each frame of the continuous multiple frames in the image to obtain each group of convolution output results; For each group of the convolution output results, weight data of features at the same spatial position in different frame images are determined to determine the key features in each group of the convolution output results.

6. The method according to claim 5, wherein determining weight data of features at the same spatial position in different frame images for each set of the convolution output results to determine the key features in each set of the convolution output results comprises: Performing global average pooling on each group of convolution output results to obtain global features of each group; Obtaining the importance coefficient of each feature in each group by dot product calculation based on the global features of each group; Normalizing the importance coefficient of each feature in each group to determine the normalized weight coefficient of each feature in each group; An enhanced feature vector corresponding to each feature is determined based on the activation function and the normalized weight coefficient of each feature in each group.

7. The method according to claim 2, wherein the utilizing the target detection model to perform feature fusion on the global feature vector and the key feature comprises: Fusing the global feature vector and the key feature at the channel level to obtain a fused feature; The spatial information of the fusion feature is analyzed to obtain the fusion feature obtained after weight distribution using the spatial information.

8. The method according to claim 7, wherein analyzing the spatial information of the fused features to obtain the fused features obtained by weighting the spatial information comprises: Determining initial weights of features at different spatial positions in the fused features; Feature processing is performed using the fusion features set with the initial weights to obtain fusion features after adjusting the initial weights of the different spatial positions.

9. A behavior detection system comprising: an acquisition unit, configured to acquire an acquired image; an input unit, configured to input the image into a target detection model, wherein the target detection model performs feature extraction on the image, divides the features extracted from each frame of the image into multiple paths, fuses the features obtained after processing the multiple paths in parallel to obtain a global feature vector for each frame of the image, determines key features based on the features of each frame of the continuous multiple frames of the image, fuses the global feature vector and the key features, and performs prediction using the fused features to obtain a prediction result; A determination unit is configured to determine whether a target behavior is detected in the collected image based on the prediction result.

10. An electronic device comprising: An image acquisition device, used for acquiring images; A processor is used to obtain a captured image, input the image into a target detection model, the target detection model performs feature extraction on the image, divides the features extracted from each frame of the image into multiple paths, fuses the obtained features after parallel processing of the multiple paths to obtain a global feature vector for each frame of the image, determines key features based on the features of each frame of the continuous multiple frames of the image, fuses the global feature vector and the key features, and performs prediction using the fused features to obtain a prediction result; and determines whether target behavior is detected in the captured image based on the prediction result.

Images