Video behavior recognition method, system, device and equipment

Abstract

The application provides a video behavior recognition method, system, device and equipment. The method of the application extracts motion features of different spatial scales in multiple layers by inputting multiple video frames into a behavior recognition model for sampling, respectively analyzes the motion features in the multiple layers to obtain correlation features in the multiple layers, and the spatial scales of the correlation features in different layers are different. The correlation features in the multiple layers are fused to obtain multi-scale fused correlation features, and a dense correlation feature field is obtained, so that the behavior recognition model can capture more fine motion features, such as small object movement and fast movement feature information. The multi-scale fused correlation features and the extracted motion features are fused to obtain fused features for behavior classification and recognition, and a behavior recognition result is obtained, thereby improving the effect and performance of video behavior recognition and improving the robustness of the behavior recognition model in actual application scenarios.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a video behavior recognition method, system, apparatus, and device. Background Technology

[0002] Video behavior recognition technology has a wide range of applications in many fields such as intelligent monitoring, human-computer interaction, video sequence understanding, healthcare, and smart education.

[0003] Traditional video action recognition models model motion features based on the pose key points of target objects in the video, and these features are usually manually designed. Deep learning-based video action recognition models model motion features using convolutional neural networks. However, these action recognition methods do not model spatiotemporal movement information and lack the ability to model fine motion features such as small objects and fast movements, resulting in poor action recognition performance in real-world applications. Summary of the Invention

[0004] This application provides a video behavior recognition method, system, apparatus, and device to solve the problem of poor behavior recognition performance in existing behavior recognition methods.

[0005] On the one hand, this application provides a video behavior recognition method, including:

[0006] Acquire video data to be identified, wherein the video data contains multiple video frames;

[0007] The multiple video frames are input into the behavior recognition model, and the behavior recognition model extracts multi-layer action features of the multiple video frames. The action features of different layers have different spatial scales.

[0008] Correlation analysis is performed on the multi-layered action features to obtain multi-layered correlation features. The spatial scale of the correlation features of different layers is different. The multi-layered correlation features are fused to obtain multi-scale fused correlation features. The multi-scale fused correlation features are then incorporated into the extracted action features to obtain fused features.

[0009] Based on the fusion features, behavior classification and recognition are performed to obtain the behavior recognition results of the target in the video data.

[0010] On the other hand, this application provides a video behavior recognition system, including:

[0011] An end-side device is used to acquire the video to be identified and to sample the video to obtain multiple video frames;

[0012] A cloud-based device is used to input the multiple video frames into a behavior recognition model, extract multi-layered action features from the multiple video frames through the behavior recognition model, with different layers of action features having different spatial scales; perform correlation analysis on the multi-layered action features to obtain multi-layered correlation features, with different layers of correlation features having different spatial scales; fuse the multi-layered correlation features to obtain multi-scale fused correlation features, and integrate the multi-scale fused correlation features into the extracted action features to obtain fused features; perform behavior classification and recognition based on the fused features to obtain the behavior recognition result of the target in the video data;

[0013] The cloud-side device is also used to send the behavior recognition result to the end-side device;

[0014] The end-side device is also used to perform post-processing based on the behavior recognition result and output the post-processing result.

[0015] On the other hand, this application provides a video behavior recognition device, comprising:

[0016] The video data acquisition module is used to acquire video data to be identified, wherein the video data contains multiple video frames;

[0017] The feature extraction module is used to input the multiple video frames into the behavior recognition model, and extract multi-layer action features of the multiple video frames through the behavior recognition model. The action features of different layers have different spatial scales.

[0018] The correlation module is used to perform correlation analysis on the multi-layer action features to obtain multi-layer correlation features. The spatial scale of the correlation features of different layers is different. The multi-layer correlation features are fused to obtain multi-scale fused correlation features. The multi-scale fused correlation features are then incorporated into the extracted action features to obtain fused features.

[0019] The classification and recognition module is used to perform behavior classification and recognition based on the fused features to obtain the behavior recognition result of the target in the video data.

[0020] On the other hand, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0021] The memory stores computer-executed instructions;

[0022] The processor executes computer execution instructions stored in the memory to implement the video behavior recognition method described in any of the above aspects.

[0023] On the other hand, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the video behavior recognition method described in any of the above aspects.

[0024] On the other hand, this application provides a computer program product, including a computer program that, when executed by a processor, implements the video behavior recognition method described in any of the above aspects.

[0025] The video behavior recognition method, system, apparatus, and device provided in this application extract action features by inputting multiple video frames into the feature extraction module of the behavior recognition model. The feature extraction module includes multiple feature extraction layers for extracting action features at different spatial scales. Correlation analysis is performed on the action features extracted from multiple feature extraction layers to obtain multi-layered correlation features. The spatial scales of the correlation features at different layers are different, and the correlation features at each layer capture the movement information of the target object. Furthermore, by fusing the multi-layered correlation features to obtain multi-scale fused correlation features, a dense correlation feature field can be obtained, enabling the behavior recognition model to capture more refined action features, such as the movement of small objects and rapid movement information. Furthermore, by integrating the multi-scale fused correlation features into the action features extracted by the feature extraction module, the action features include more refined features such as the movement information of small objects and rapid movement information. Based on the fused features, behavior classification and recognition are performed to obtain the behavior recognition result of the target in the video data, improving the effect and performance of video behavior recognition and enhancing the robustness of the behavior recognition model in practical application scenarios. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] Figure 1 A schematic diagram of an example network architecture to which this application applies;

[0028] Figure 2 This is a schematic diagram of another example network architecture to which this application applies;

[0029] Figure 3 A flowchart of a video behavior recognition method provided as an exemplary embodiment of this application;

[0030] Figure 4 An example diagram illustrating an overall framework for video behavior recognition provided as an exemplary embodiment of this application;

[0031] Figure 5Example diagram of an overall framework for video behavior recognition provided as another exemplary embodiment of this application;

[0032] Figure 6 Example diagram of an overall framework for video behavior recognition provided as yet another exemplary embodiment of this application;

[0033] Figure 7 A flowchart of a video behavior recognition method provided as another exemplary embodiment of this application;

[0034] Figure 8 Flowchart of a behavior recognition model training method provided for an exemplary embodiment of this application;

[0035] Figure 9 A schematic diagram of a video behavior recognition system provided as an exemplary embodiment of this application;

[0036] Figure 10 A flowchart of a video behavior recognition method provided as an exemplary embodiment of this application;

[0037] Figure 11 This is a schematic diagram of the structure of a video behavior recognition device provided in an example embodiment of this application;

[0038] Figure 12 This is a schematic diagram of the structure of a video behavior recognition device provided in another example embodiment of this application;

[0039] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an example embodiment of this application.

[0040] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0042] First, let me explain the terms used in this application:

[0043] Video behavior recognition: refers to identifying one or more actions of a target object from video data.

[0044] Motion modeling: refers to the analysis and learning of the movement information of objects in a video.

[0045] Visual relevance: refers to the similarity between image textures, usually calculated in pixels.

[0046] Spatial scale: refers to the spatial resolution of a feature.

[0047] Traditional video behavior recognition models model action features based on the pose key points of target objects in the video, and these features are usually designed manually; deep learning-based video behavior recognition models model action features through convolutional neural networks.

[0048] However, these behavior recognition methods do not model spatiotemporal movement information and lack the ability to model fine motion features such as small objects and rapid movements, resulting in low accuracy and poor performance of behavior recognition in real-world applications.

[0049] A deep learning-based video behavior recognition method uses visual correlation analysis to learn the movement information of objects in the video by performing correlation analysis at the image level. However, in the subsequent feature extraction process, the resolution of the features (movement information) is continuously reduced by multiple downsampling operations, causing the positional information of small objects to disappear in the deep layers. The spatial features and semantic features do not match, and the method still lacks the ability to model the actions of small objects and fast-moving objects. As a result, the performance of behavior recognition is poor in practical application scenarios.

[0050] To address the aforementioned technical problems, this application provides a video behavior recognition method. Based on a trained video behavior recognition model, this method achieves video behavior recognition by including multiple feature extraction layers for extracting action features at different spatial scales. Different feature extraction layers have different spatial scales, resulting in different spatial scales for the extracted action features. This video behavior recognition method obtains multi-layered correlation features by performing correlation analysis on the action features extracted from multiple feature extraction layers. The correlation features of any layer have the same spatial scale as the action features of that layer, while the spatial scales of the correlation features from different layers differ. Furthermore, by fusing the multi-layered correlation features, a multi-scale fused correlation feature field is obtained, which enables the video behavior recognition model to capture more refined action features, such as the movement of small objects and rapid movement. Moreover, by integrating the multi-scale fused correlation features into the action features extracted by the feature extraction module, behavior classification and recognition are performed based on the fused features to obtain the behavior recognition results of the target in the video data. This improves the robustness of the model in practical application scenarios and enhances the effectiveness and performance of video behavior recognition.

[0051] The video behavior recognition method provided in this embodiment can be applied to many fields such as security, human-computer interaction, video understanding, medical health, smart education, and intelligent transportation, and has a very wide range of applications.

[0052] For example, a typical application scenario in the security field is intelligent monitoring: based on the monitored video, multiple video frames are sampled, and these frames are input into a behavior recognition model. The behavior recognition model identifies the actions of the target object appearing in the video, obtaining the behavior recognition result. Furthermore, based on the behavior recognition result, it is possible to analyze and determine whether the target object in the video has committed or been committed dangerous acts (such as theft, fighting, peeping, etc.), thereby automatically identifying dangerous behaviors and providing early warning processing.

[0053] For example, an example application scenario in the field of human-computer interaction is as follows: Based on an input video containing interactive objects, multiple video frames are sampled. These multiple video frames are input into a behavior recognition model, which identifies the behavioral actions (such as gestures, body postures, etc.) performed by the interactive objects in the video, obtaining the behavior recognition result. Furthermore, based on the behavior recognition result, the action intention of the interactive object can be determined, and corresponding feedback information can be generated and provided to the interactive object.

[0054] For example, an exemplary application scenario in the field of intelligent education is as follows: Based on online or offline videos of teachers lecturing, multiple video frames are sampled. These frames are then input into a behavior recognition model, which identifies the teacher's actions in the videos, yielding behavior recognition results. Furthermore, based on these results, it can be analyzed to determine whether the teacher has performed any pre-defined behaviors (such as behaviors that do not conform to behavioral norms). This allows for online warnings or provides information to relevant personnel for offline processing, thereby regulating the teacher's behavior during lectures.

[0055] For example, an exemplary application scenario in the field of intelligent transportation is as follows: Videos of a driver operating a vehicle, captured by roadside units and / or vehicle-mounted cameras, are sampled to obtain multiple video frames. These frames are then input into a behavior recognition model, which identifies the driver's actions in the video to obtain behavior recognition results. Furthermore, based on these results, it is possible to analyze and determine whether the driver engaged in any pre-defined unsafe driving behaviors, such as fatigued driving, smoking, or making phone calls. This allows for online warnings to the driver or provides information to relevant personnel for offline intervention, thereby regulating safe driving behavior and improving driving safety.

[0056] Furthermore, the rules for sampling video data can be flexibly configured when applied to different application scenarios / domains, and the behavior recognition model is trained and determined using the dataset of the specific application scenario / domain.

[0057] Figure 1 This is a schematic diagram of an example network architecture to which this application applies. Figure 1 As shown, the network architecture includes a first electronic device responsible for video behavior recognition, and a second electronic device responsible for acquiring and providing video data to the first electronic device. The second electronic device is also responsible for performing post-processing of the behavior recognition results.

[0058] The first electronic device can be a server cluster deployed in the cloud, a local device with computing capabilities, or an Internet of Things (IoT) device. This first electronic device stores a behavior recognition model trained on a training set specific to a particular application scenario / domain. Through pre-set computational logic, the first electronic device can use the behavior recognition model to perform behavior recognition on multiple video frames sampled from video data, obtaining behavior recognition results. Based on the behavior recognition results, the first electronic device can provide feedback to the second electronic device.

[0059] The second electronic device can specifically be a hardware device with network communication, computing and information display functions, including but not limited to smartphones, tablets, desktop computers and other terminals, servers, Internet of Things devices, etc.

[0060] Through communication with the first electronic device, the user can submit collected video data to the first electronic device via the second electronic device. The first electronic device can sample multiple video frames from the video data and input these frames into the feature extraction module of the behavior recognition model for action feature extraction. The feature extraction module contains multiple feature extraction layers for extracting action features at different spatial scales. Correlation analysis is performed on the action features extracted by the multiple feature extraction layers to obtain multi-layer correlation features. The spatial scales of the correlation features at different layers are different. The multi-layer correlation features are fused to obtain multi-scale fused correlation features, which are then integrated into the action features extracted by the feature extraction module to obtain fused features. Based on the fused features, behavior classification and recognition are performed to obtain the behavior recognition result of the target in the video data. The first electronic device feeds back the behavior recognition result to the second electronic device, which can perform post-processing on the behavior recognition result based on post-processing rules and output the final post-processed result.

[0061] For example, in a typical application scenario in the field of intelligent education, the second electronic device can be a device that collects and stores videos of teachers lecturing. During online behavior recognition, the second electronic device sends the video data of the teacher's lecture to the first electronic device. The first electronic device acquires the video data of the teacher's lecture, samples multiple video frames, inputs these frames into a behavior recognition model, identifies the teacher's actions in the video through the behavior recognition model, obtains the behavior recognition result, and feeds the behavior recognition result back to the second electronic device. Further, based on the behavior recognition result, the second electronic device analyzes and determines whether the teacher has performed a preset behavior (such as behavior that does not conform to behavioral norms), thereby enabling online warnings or providing information to relevant personnel for offline processing to regulate the teacher's behavior during lectures.

[0062] Figure 2 This is a schematic diagram of another example network architecture to which this application applies. Figure 2 As shown, the network architecture includes a first electronic device responsible for video behavior recognition, and a second electronic device responsible for acquiring and providing video data to the first electronic device. The first electronic device is also responsible for performing post-processing of the behavior recognition results.

[0063] The first electronic device can be a server cluster deployed in the cloud, a local device with computing capabilities, or an Internet of Things (IoT) device. This first electronic device stores a behavior recognition model trained on a training set specific to a particular application scenario / domain. Through pre-set computational logic, the first electronic device can use the behavior recognition model to perform behavior recognition on multiple video frames sampled from video data, obtaining behavior recognition results. Further, the first electronic device performs post-processing on the behavior recognition results based on post-processing rules, and feeds back the final post-processed result to the second electronic device.

[0064] The second electronic device can specifically be a hardware device with network communication, computing, and information display functions, including but not limited to smartphones, tablets, desktop computers, servers, and IoT devices. Through communication and interaction with the first electronic device, the user can submit collected video data to the first electronic device via the second electronic device. The first electronic device can sample multiple video frames from the video data and input these frames into the feature extraction module of the behavior recognition model for action feature extraction. The feature extraction module contains multiple layers for extracting action features at different spatial scales. Correlation analysis is performed on the action features extracted from multiple feature extraction layers to obtain multi-layered correlation features. The spatial scales of the correlation features from different layers are different. Multi-layered correlation features are fused to obtain multi-scale fused correlation features, which are then integrated into the action features extracted by the feature extraction module to obtain fused features. Based on the fused features, behavior classification and recognition are performed to obtain the behavior recognition result of the target in the video data. Further, the first electronic device performs post-processing on the behavior recognition result based on post-processing rules and feeds back the final post-processed result to the second electronic device.

[0065] For example, taking a typical application scenario in the field of human-computer interaction, the second electronic device can be a terminal device used by the interactive object to collect video data of the interactive object and send it to the first electronic device. The first electronic device receives the video data of the interactive object, samples multiple video frames, inputs the multiple video frames into a behavior recognition model, and identifies the behavioral actions (such as gestures, body movements, etc.) performed by the interactive object in the video through the behavior recognition model to obtain the behavior recognition result. Further, based on the behavior recognition result, the following post-processing is performed: determining the action intention of the interactive object, and generating corresponding feedback information for the action intention of the interactive object. Then, the first electronic device sends the feedback information to the interactive object, so that the first electronic device can provide feedback to the interactive object based on the feedback information.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0067] Figure 3 This is a flowchart of a video behavior recognition method provided as an exemplary embodiment of this application. The execution subject of this embodiment is the aforementioned method. Figure 1 The first electronic device shown. (As shown) Figure 3 As shown, the specific steps of this method are as follows:

[0068] Step S301: Obtain the video data to be identified, which contains multiple video frames.

[0069] The video data to be identified contains multiple video frames, which are sampled from the video to be identified.

[0070] In practical applications, a video containing the target object for behavior recognition can be acquired. The video is sampled using predefined frame sampling rules to obtain multiple video frames, which are then used for behavior recognition.

[0071] The frame sampling rules can be implemented using the rules for sampling video frames from any existing video behavior recognition method, and can be configured and adjusted according to the needs of actual application scenarios, which will not be elaborated here.

[0072] Optionally, in this step, the first electronic device may receive video data to be identified, which contains multiple video frames, sent by the second electronic device. This video data is obtained by the second electronic device from frame sampling of the original video.

[0073] Optionally, in this step, the first electronic device may also receive the original video sent by the second electronic device. The first electronic device performs frame sampling on the original video to obtain multiple video frames, and the multiple sampled video frames constitute the video data to be identified.

[0074] Step S302: Input multiple video frames into the behavior recognition model, and extract multi-layer action features of multiple video frames through the behavior recognition model. The action features of different layers have different spatial scales.

[0075] In this embodiment, a trained end-to-end behavior recognition model is used to perform behavior recognition based on multiple input video frames to obtain the behavior recognition result.

[0076] Action recognition models typically include a feature extraction module for extracting action features. This module usually consists of multiple feature extraction layers stacked from top to bottom. The output of the upper layer serves as the input to the lower layer, and the output of the bottom layer represents the action features extracted by the feature extraction module.

[0077] Different feature extraction layers have different spatial scales (i.e., spatial resolutions), resulting in different spatial scales of the extracted action features.

[0078] In this step, multiple video frames are input into the top feature extraction layer of the behavior recognition model. This top feature extraction layer extracts features from each video frame to obtain the corresponding action features. Then, these multi-frame action features are processed by multiple feature extraction layers to obtain the action features extracted by each layer, also known as the action features of each layer. In any given layer, features are extracted from each input frame to obtain the corresponding action features for that layer, thus obtaining the action features of the current layer. This includes multi-frame action features, where the number of action feature frames is the same as the number of video frames, and each multi-frame action feature corresponds one-to-one with multiple video frames.

[0079] In addition, the number of feature extraction layers in the behavior recognition model and the size of the spatial scale of different layers can be set based on empirical values, and no specific limit is made here.

[0080] For example, taking a feature extraction module containing three feature extraction layers as an example, the three feature extraction layers extract three motion features at different spatial scales. The spatial scale can be spatial resolution, and the three different spatial resolutions are 56, 28, and 14 in order from top to bottom. The number of frames for the motion features in different layers is the same, which is the same as the number of frames in the input video. The length and width of the motion features in different layers are also consistent.

[0081] Step S303: Perform correlation analysis on the multi-layer motion features to obtain multi-layer correlation features. The spatial scale of the correlation features of different layers is different. Fuse the multi-layer correlation features to obtain multi-scale fused correlation features. Then, integrate the multi-scale fused correlation features into the extracted motion features to obtain fused features.

[0082] In this embodiment, a correlation module is added to the behavior recognition model to perform correlation analysis on the action features extracted by multiple feature extraction layers to obtain multi-layer correlation features. The spatial scale of the correlation features of different layers is different. Multi-layer correlation features are fused to obtain multi-scale fused correlation features. The multi-scale fused correlation features are then incorporated into the action features extracted by the feature extraction module, so that the final output features of the feature extraction module incorporate multi-scale fused correlation features and contain more refined action features, such as the movement information of small objects and the movement information of fast actions.

[0083] By adding a correlation module to multiple feature extraction layers, correlation analysis is performed on the multi-frame motion features extracted by the current layer in the time dimension to obtain the correlation features of the current layer, which can better capture the movement information of the target object.

[0084] The spatial scale of correlation features at any given layer is the same as that of action features, while the spatial scale of correlation features at different layers is different. Furthermore, by fusing correlation features from multiple layers, correlation features at different spatial scales can be integrated, enabling the capture of more refined action features, such as the movement information of small objects and the movement information of rapid actions. Furthermore, by incorporating the multi-scale fused correlation features obtained from fusing multiple layers into the extracted action features, the output features of the feature extraction module include more refined action features such as the movement information of small objects and the movement information of rapid actions. Based on this fused feature, behavior classification and recognition can improve the robustness and effectiveness of the behavior recognition model.

[0085] Optionally, when performing correlation analysis, multiple layers can be extracted from the feature extraction layers in the behavior recognition model to obtain the correlation features of the corresponding layers. Furthermore, to improve the ability to capture more refined motion features, such as the movement information of small objects and rapid movements, the spatial scale span of the extracted feature extraction layers can be appropriately expanded, thereby resulting in a larger spatial scale span for the obtained multi-layer correlation features.

[0086] Optionally, when performing correlation analysis, correlation analysis can be performed on each feature extraction layer in the behavior recognition model to obtain the time-related features of each layer.

[0087] Optionally, when incorporating the multi-scale fused correlation features into the action features extracted by the feature extraction module, the multi-scale fused correlation features can be incorporated into the action features extracted by one or more feature extraction layers, and the multi-scale fused correlation features incorporated into different layers can be the same or different.

[0088] Optionally, when integrating the multi-scale fused correlation features into the action features extracted by the feature extraction module to obtain fused features, the multi-scale fused correlation features can be added to the action features extracted by the feature extraction module to achieve the fusion of the multi-scale fused correlation features and action features to obtain fused features.

[0089] Step S304: Based on the fusion features, perform behavior classification and recognition to obtain the behavior recognition results of the target in the video data.

[0090] In this embodiment, by performing correlation analysis on the multi-frame action features extracted by the current layer in the time dimension across multiple feature extraction layers, the correlation features of the current layer are obtained, which can effectively capture the movement information of the target object. The correlation features of any layer have the same spatial scale as the action features, while the spatial scales of the correlation features of different layers differ. By fusing multi-layer correlation features, multiple correlation features at different spatial scales can be integrated, effectively capturing the movement information of small objects and fast-moving actions. Furthermore, by incorporating the multi-scale fused correlation features obtained from fusing multi-layer correlation features into the extracted action features, the fused features output by the feature extraction module contain more refined action features, such as the movement information of small objects and fast-moving actions. Based on these fused features, behavior classification and recognition can improve the robustness and effectiveness of the behavior recognition model.

[0091] In an optional embodiment, the multi-scale fused correlation features obtained by fusing multi-layer correlation features in step S303 above can be implemented in the following way:

[0092] The correlation feature with the smallest spatial scale among the multi-layer correlation features to be fused is taken as the target feature; the correlation features to be fused other than the target feature are downsampled, and the downsampled features have the same spatial scale as the target features; the downsampled features are fused with the target features to obtain the multi-scale fused correlation features.

[0093] In this embodiment, since the spatial scales of the correlation features of different layers are different, in order to fuse the correlation features of different layers, the correlation features of larger spatial scales can be downsampled according to the minimum spatial scale of the multi-layer correlation features, so that the spatial scale of the downsampled features is equal to the minimum spatial scale. Then, the downsampled features and the correlation features of the minimum spatial scale are fused to obtain the multi-scale fused correlation features.

[0094] Optionally, the downsampled features can be fused with the target features by concatenating the downsampled features and the target features sequentially according to the order of the corresponding video frames; or by summing the downsampled features and the target features.

[0095] In another optional embodiment, the multi-scale fused correlation features obtained by fusing multi-layer correlation features in step S303 above can also be achieved in the following way:

[0096] According to the configured preset spatial scale, which is less than or equal to the minimum spatial scale of the multi-layer correlation features to be fused; the multi-layer correlation features to be fused are downsampled to the preset spatial scale, and the downsampled features are fused to obtain multi-scale fused correlation features.

[0097] Optionally, the downsampled features can be fused by concatenating the downsampled features sequentially according to the order of the corresponding video frames; or by summing the downsampled features.

[0098] Furthermore, in any of the above embodiments, the correlation features of each layer include multi-frame features that correspond one-to-one with multiple video frames. Fusion of multi-layer correlation features refers to fusing the features corresponding to the same video frame from different layers; the resulting multi-scale fused correlation features still include multi-frame features that correspond one-to-one with video frames.

[0099] In an optional embodiment, in step S303 above, when integrating the multi-scale fused correlation features into the extracted action features, the multi-scale fused correlation features can be fused with the lowest-level output features to obtain fused features. These fused features are the output features of the feature extraction module and are used for subsequent behavior classification and recognition.

[0100] For example, Figure 4 An example diagram illustrating an overall framework for video behavior recognition provided in this application embodiment, such as... Figure 4 As shown, taking a behavior recognition model with a feature extraction module containing three feature extraction layers of different spatial scales as an example, action features at three different spatial scales (represented by graphics of different sizes) are extracted. Correlation analysis is performed on the action features extracted by each feature extraction layer to obtain the correlation features at the three different spatial scales. Based on the spatial scale of the correlation features of the lowest layer, the correlation features of the two upper layers are downsampled so that the downsampled features have the same spatial scale as the correlation features of the lowest layer. The sampled features are then fused with the correlation features of the lowest layer to obtain multi-scale fused correlation features. This is a dense correlation feature field, which can capture more refined action features, such as the movement of small objects and rapid movement. Furthermore, the multi-scale fused correlation features are fused with the output features of the lowest layer to obtain fused features. This fused feature is input into the behavior classification and recognition module for behavior classification and recognition, outputting behavior recognition results. This improves the robustness of the model in practical application scenarios and enhances the effect and performance of video behavior recognition.

[0101] It should be noted that, Figure 4 Taking a video with 5 frames as an example, each feature extraction layer extracts 5 frames of motion features. Correlation analysis is then performed on the 5 frames of motion features and other motion features to obtain the 5 frames of correlation features corresponding to the 5 frames of motion features. These 5 frames of correlation features correspond one-to-one with the 5 video frames.

[0102] in addition, Figure 4The text uses graphics of different sizes to represent the relationships between features (action features, correlation features) at different spatial scales. However, the size ratio of the graphics representing features does not represent the size ratio of the spatial scale, and the size of the graphics does not reflect the specific value of the spatial scale.

[0103] Based on this embodiment, in other optional embodiments, as an alternative to fusing the correlation features fused by multi-scale fusion with the output features of the lowest layer, the correlation features fused by multi-scale fusion can also be fused with the action features output by the lowest layer in the multi-layer that has undergone correlation analysis processing. In this way, the correlation features fused by multi-scale fusion can also be incorporated into the action features extracted by the feature extraction module.

[0104] In this embodiment, a correlation module is added to the basic network of the behavior recognition model. This correlation module can perform correlation analysis based on the action features extracted by the multi-layer feature extraction layer to obtain correlation features and fuse the multi-layer correlation features, thereby fusing multiple correlation features at different spatial scales and fusing them with the action features output by the feature advance module. This allows the deep action features to also contain better spatial and temporal information, enabling the behavior recognition model to capture more refined action features, such as the movement information of small objects and information on rapid movement, thereby improving the robustness of the behavior recognition model in practical application scenarios.

[0105] In another optional embodiment, in step S303 above, in the layer that has undergone correlation analysis, the correlation features of the current layer can be fused with the action features to obtain the output features of the current layer, which are then used as the input to the next layer. Furthermore, multi-layer correlation features are fused to obtain multi-scale fused correlation features, which are then fused with the output features of the lowest layer to obtain fused features. These fused features are the output features of the feature extraction module and are used for subsequent behavior classification and recognition.

[0106] Optionally, in each layer that has undergone correlation analysis, the correlation features of the current layer can be fused with the action features to obtain the output features of the current layer, and the output features of the current layer can be used as the input of the next layer.

[0107] Optionally, one or more layers that have undergone correlation analysis can be selected, and the correlation features and action features of each layer can be fused in the selected layer to obtain the output features of each layer. The output features of each layer are then used as the input of the next layer.

[0108] For example, if the lowest layer in the feature extraction module has undergone correlation analysis, the correlation features of each layer and the action features of each layer that has undergone correlation analysis can be fused to obtain the output features of each layer, and the output features of each layer can be used as the input of the next layer.

[0109] For example, Figure 5 An example diagram illustrating an overall framework for video behavior recognition provided in this application embodiment, such as... Figure 5 As shown, taking a behavior recognition model with a feature extraction module containing three feature extraction layers of different spatial scales as an example, action features at three different spatial scales (represented by graphics of different sizes) are extracted. Correlation analysis is performed on the action features extracted by each feature extraction layer to obtain the correlation features at the three different spatial scales. In the top two layers, the correlation features of the current layer are fused with the action features extracted by the current layer, and the fused features are used as the input to the next layer. This allows correlation features to be fused from the action features extracted by each layer, resulting in the fused features of the top two layers containing better motion information. Furthermore, based on the spatial scale of the correlation features of the bottom layer, the correlation features of the top two layers are downsampled so that the downsampled features have the same spatial scale as the correlation features of the bottom layer. The sampled features are then fused with the correlation features of the bottom layer to obtain multi-scale fused correlation features. This is a dense correlation feature field, which can capture more refined action features, such as the movement of small objects and the feature information of rapid movement. Finally, the multi-scale fused correlation features are fused with the output features of the bottom layer to obtain the fused features. This fusion feature input behavior classification and recognition module performs behavior classification and recognition, and outputs behavior recognition results, which can improve the robustness of the model in real application scenarios and improve the effect and performance of video behavior recognition.

[0110] It should be noted that, Figure 5 Taking a video with 5 frames as an example, each feature extraction layer extracts 5 frames of motion features. Correlation analysis is then performed on the 5 frames of motion features and other motion features to obtain the 5 frames of correlation features corresponding to the 5 frames of motion features. These 5 frames of correlation features correspond one-to-one with the 5 video frames.

[0111] in addition, Figure 5 The text uses graphics of different sizes to represent the relationships between features (action features, correlation features) at different spatial scales. However, the size ratio of the graphics representing features does not represent the size ratio of the spatial scale, and the size of the graphics does not reflect the specific value of the spatial scale.

[0112] In this embodiment, a correlation module is added to the basic network of the behavior recognition model. This correlation module performs correlation analysis on the action features extracted by multiple feature extraction layers to obtain correlation features. Within one or more layers that have undergone correlation analysis, the correlation features of the current layer are fused with the action features to obtain the output features of the current layer. The output features of the current layer serve as the input to the next layer, ensuring that the action features of each layer contain rich movement information. Furthermore, by fusing multi-layer correlation features, correlation features at multiple spatial scales are integrated and fused with the action features output by the feature advance module. This allows deeper action features to also contain good spatial and temporal information, enabling the behavior recognition model to capture more refined action features, such as the movement of small objects and the movement of express delivery packages, thereby improving the robustness of the behavior recognition model in practical application scenarios.

[0113] In another optional embodiment, in the layer that has undergone correlation analysis processing, the correlation features of the current layer can be fused with the correlation features of at least one upper layer to obtain the multi-scale fused correlation features of the current layer. The multi-scale fused correlation features of the current layer are then fused with the action features to obtain the output features of the current layer. By adding correlation analysis processing in each feature extraction layer to obtain the correlation features of the action features, the motion information of the target in different video frames can be captured. Furthermore, by fusing the correlation features of different spatial scales of the upper layer in each layer, a pyramid-shaped multi-scale fused correlation feature can be constructed and fused into the extracted action features to obtain a dense correlation feature field. This field is then fused into the extracted action features for behavior recognition, enabling the deep features to also contain good spatial and temporal information. This allows the behavior recognition model to capture more refined action features, such as the movement of small objects and the movement of express delivery, thereby improving the robustness of the behavior recognition model in practical application scenarios.

[0114] If a layer that has undergone correlation analysis does not have correlation features from the upper layer, then the correlation features of the current layer are fused with the action features of the current layer to obtain the output features of the current layer.

[0115] For example, for the topmost feature extraction layer in the feature extraction module, since there is no upper-level feature extraction layer, the correlation features of the current layer are fused with the action features of the current layer to obtain the output features of the current layer.

[0116] For example, a layer that has undergone correlation analysis may have an upper feature extraction layer, but the upper feature extraction layers have not undergone correlation analysis. In this case, there are no correlation features of the upper layer. The correlation features of the upper layer are fused with the action features of the upper layer to obtain the output features of the upper layer.

[0117] Optionally, in each layer that has undergone correlation analysis, the correlation features of the current layer can be fused with the correlation features of at least one upper layer to obtain the multi-scale fused correlation features of the current layer, and the multi-scale fused correlation features of the current layer can be fused with the action features to obtain the output features of the current layer, and the output features of the current layer can be used as the input of the next layer.

[0118] Optionally, one or more layers can be selected from the layers that have undergone correlation analysis. In the selected layer, the correlation features of the current layer are fused with the correlation features of at least one upper layer to obtain the multi-scale fused correlation features of the current layer. The multi-scale fused correlation features of the current layer are then fused with the action features to obtain the output features of the current layer. The output features of the current layer are used as the input of the next layer.

[0119] In a preferred embodiment, in each layer that has undergone correlation analysis, the correlation features of the current layer are fused with the correlation features of all upper layers to obtain the multi-scale fused correlation features of the current layer. The multi-scale fused correlation features of the current layer are then fused with action features to obtain the output features of the current layer. This allows for the capture of more refined action features, such as the movement information of small objects and rapid movement information. Furthermore, a pyramid-shaped multi-scale fused correlation feature can be constructed and fused into the extracted action features to obtain a dense correlation feature field. This enables the deep features to also contain good spatial and temporal information, allowing the behavior recognition model to capture more refined action features, such as the movement of small objects and the movement of express delivery vehicles, thereby improving the robustness of the behavior recognition model in practical application scenarios.

[0120] For example, Figure 6 An example diagram illustrating an overall framework for video behavior recognition provided in this application embodiment, such as... Figure 6As shown, taking a behavior recognition model with a feature extraction module containing three feature extraction layers at different spatial scales as an example, action features at three different spatial scales (represented by graphics of different sizes) are extracted. Correlation analysis is performed on the action features extracted by each feature extraction layer to obtain the correlation features at the three different spatial scales. In each layer where correlation analysis is performed, in the first layer, since there is no upper-level feature extraction layer, the correlation features of the current layer are fused with the action features to obtain the output features of the current layer, which are then used as the input to the second layer. In the second layer, the correlation features of the first layer are fused with the correlation features of the current layer (the second layer) to obtain the multi-scale fused correlation features of the current layer (the second layer). These multi-scale fused correlation features are then fused with the action features to obtain the output features of the current layer (the second layer), which are then used as the input to the third layer. In the third layer, the correlation features of the first layer, the second layer, and the current layer (third layer) are fused to obtain the multi-scale fused correlation features of the current layer (third layer). The multi-scale fused correlation features of the current layer (third layer) are then fused with the action features to obtain the output features of the current layer (third layer), which are used as the fused features output by the feature extraction module.

[0121] It should be noted that, Figure 6 Taking a video with 5 frames as an example, each feature extraction layer extracts 5 frames of motion features. Correlation analysis is then performed on the 5 frames of motion features and other motion features to obtain the 5 frames of correlation features corresponding to the 5 frames of motion features. These 5 frames of correlation features correspond one-to-one with the 5 video frames.

[0122] in addition, Figure 6 The text uses graphics of different sizes to represent the relationships between features (action features, correlation features) at different spatial scales. However, the size ratio of the graphics representing features does not represent the size ratio of the spatial scale, and the size of the graphics does not reflect the specific value of the spatial scale.

[0123] In this embodiment, by adding correlation analysis processing to each feature extraction layer, the correlation features of action features are obtained, which can capture the movement information of target objects in video data. Furthermore, by fusing the correlation features of different spatial scales of the upper layer at each layer, a pyramid-shaped multi-scale fused correlation feature can be constructed and fused into the extracted action features to obtain a dense correlation feature field. This allows the deep features to also contain good spatial and temporal information, enabling the behavior recognition model to capture more refined action features, such as the movement of small objects and the movement of express delivery, thereby improving the robustness of the behavior recognition model in practical application scenarios.

[0124] Based on any of the above embodiments, correlation analysis is performed on the action features of any layer to obtain the correlation features of the current layer. This can be achieved in the following way:

[0125] Each layer of motion features contains one frame of motion features corresponding to each video frame. Multiple frames of motion features are extracted from each layer, with the number of frames equal to the number of frames in the input video. During correlation analysis, each frame of motion features is treated as a target frame, and the correlation between this target frame and at least one adjacent frame is analyzed.

[0126] Specifically, for any target frame in the motion features of any layer, at least one neighboring frame of the target frame is determined in the motion features of the current layer; the correlation between the target frame and each neighboring frame is calculated to obtain the correlation features between the target frame and each neighboring frame; the correlation features between the target frame and at least one neighboring frame are fused to obtain the correlation features corresponding to the target frame of the current layer.

[0127] By performing correlation analysis on the motion features of each frame with the motion features of at least one other adjacent frame, the movement information of the target object in the time dimension can be captured. Furthermore, by performing correlation analysis on the motion features extracted by the multi-layer feature extraction layer, correlation features at different spatial scales can be extracted. This allows the movement information of small objects to be captured at a larger spatial scale and integrated into the extracted motion features. This enables the behavior recognition model to capture more refined motion features, such as the movement of small objects and the movement of express delivery vehicles, thereby improving the robustness of the behavior recognition model in practical application scenarios.

[0128] Optionally, when determining the neighboring frame for correlation analysis with the target frame in the motion features of any layer, the motion features of the frame following the target frame are taken as the neighboring frame according to the temporal information of the corresponding video frame, and the correlation analysis is performed between the target frame and the neighboring frame to obtain the correlation features of the target frame.

[0129] Optionally, when determining at least one adjacent frame of the target frame in the motion features of any layer, a preset number of adjacent video frames corresponding to the target frame can be determined in the motion features of the current layer according to a preset number configured, wherein the preset number is greater than 1; and the motion features corresponding to the adjacent video frames are taken as adjacent frames of the target frame.

[0130] By performing correlation analysis between the target frame and a preset number of adjacent frames, long-term correlation features can be obtained, which can capture movement information over a longer period of time. This allows for the capture of rapid movement information and better characterizes the behavior of the target object. As a result, the behavior recognition model can capture more refined movement features, such as the movement of small objects and the movement of express delivery vehicles, thereby improving the robustness of the behavior recognition model in practical application scenarios.

[0131] Furthermore, the correlation between the target frame and each adjacent frame is calculated to obtain the correlation features between the target frame and each adjacent frame. This can be achieved in the following way:

[0132] Based on the configured feature block size, multiple first feature blocks are determined in the target frame, and second feature blocks that match the first feature blocks are determined in adjacent frames; the similarity between each first feature block and the matching second feature block is calculated, and the correlation features between the target frame and each adjacent frame are determined based on the similarity between each first feature block and the matching second feature block.

[0133] The size of the feature block can be set according to the actual application scenario and empirical values, and no specific limit is set here.

[0134] Specifically, when determining multiple first feature blocks in the target frame, for each pixel, the region centered on each pixel can be obtained as a feature block according to the feature block size, and each pixel corresponds to a feature block.

[0135] When determining the second feature block that matches the first feature block in adjacent frames, first determine the matching pixel point corresponding to the pixel point at the center of the first feature block in the adjacent frames, and determine the region centered on the matching pixel point in the adjacent frames according to the feature block size, as the second feature block that matches the first feature block.

[0136] In other embodiments, the sizes of the first feature block and the matching second feature block may be different. When determining the second feature block that matches the first feature block, a region of a specified size centered on the matching pixel in an adjacent frame is determined according to a configured specified size (different from the aforementioned feature block size) as the second feature block that matches the first feature block.

[0137] Optionally, when calculating the similarity between each first feature block and the matching second feature block, the inner product of each first feature block and the matching second feature block can be calculated as the similarity.

[0138] Optionally, when calculating the similarity between each first feature block and the matching second feature block, the cosine similarity between each first feature block and the matching second feature block is calculated.

[0139] Furthermore, based on the similarity between each first feature block and its matching second feature block, the correlation features between the target frame and each adjacent frame can be determined. The value of each pixel in the correlation features between the target frame and each adjacent frame is the similarity between the first feature block corresponding to that pixel and its matching second feature block; alternatively, the similarity between each first feature block and its matching second feature block is normalized, and the value of each pixel in the correlation features between the target frame and each adjacent frame is the normalized value of the similarity between the first feature block corresponding to that pixel and its matching second feature block.

[0140] Furthermore, when fusing the correlation features between the target frame and multiple adjacent frames, the correlation features between the target frame and multiple adjacent frames can be concatenated to obtain the correlation features corresponding to the target frame; or the correlation features between the target frame and multiple adjacent frames can be summed to obtain the correlation features corresponding to the target frame.

[0141] Based on any of the above embodiments, before performing correlation analysis on the action features of any layer to obtain the correlation features of the current layer, a first convolution operation can be performed on the action features extracted from the current layer to reduce the number of channels of the action features, thereby reducing the computational load of correlation analysis and improving the efficiency of video behavior recognition.

[0142] Based on any of the above embodiments, before fusing the correlation features and action features, the correlation features can be converted into action information by performing a second convolution operation on the correlation features, and then the action information can be fused with the action features, which can improve the fusion effect.

[0143] Figure 7 This is a flowchart of a video behavior recognition method provided as another exemplary embodiment of this application. The execution subject of this embodiment is the aforementioned method. Figure 2 The first electronic device shown. (As shown) Figure 7 As shown, the specific steps of this method are as follows:

[0144] Step S701: Obtain the video to be identified and sample the video to obtain multiple video frames.

[0145] This step is similar to the implementation of step S301 above. For details, please refer to the relevant description of step S301, which will not be repeated here.

[0146] Step S702: Input multiple video frames into an end-to-end behavior recognition model, perform behavior recognition through the behavior recognition model, and obtain behavior recognition results. The behavior recognition model is used to extract multi-layer action features from multiple video frames. The action features of different layers have different spatial scales. Perform correlation analysis on the multi-layer action features to obtain multi-layer correlation features. The spatial scales of the correlation features of different layers are different. Fuse the multi-layer correlation features to obtain multi-scale fused correlation features, and integrate the multi-scale fused correlation features into the extracted action features to obtain fused features. Based on the fused features, perform behavior classification and recognition to obtain behavior recognition results.

[0147] This step is similar to the implementation of steps S302-S304 above. Please refer to the relevant content in the above embodiments for details, which will not be repeated here.

[0148] Step S703: Perform post-processing based on the behavior recognition results and output the post-processing results.

[0149] In practical applications, different post-processing rules can be configured according to different application scenarios. After obtaining the behavior recognition results, post-processing is performed according to the configured post-processing rules to obtain the final output results.

[0150] The output method of the post-processing results can be configured according to the actual application scenario. Different application scenarios may output the final processing results in different ways.

[0151] In this embodiment, based on Figure 2 The network architecture shown has a first electronic device that is equipped with a trained behavior recognition model and provides video behavior recognition services based on the behavior recognition model. The final result of the service is determined by post-processing the behavior recognition results.

[0152] The video behavior recognition method provided in this embodiment can be applied to many fields such as intelligent monitoring, human-computer interaction, video sequence understanding, medical health, and intelligent education, and has a very wide range of applications.

[0153] The behavior recognition model used in any of the above embodiments is an end-to-end model trained on a training set based on a specific application scenario / domain. Figure 8 A flowchart of a behavior recognition model training method provided for an exemplary embodiment of this application is shown below. Figure 8 As shown, the behavior recognition model used in any of the above embodiments can be trained in the following way:

[0154] Step S801: Obtain the training set for the current application scenario. The training set includes audio sample data and the behavior categories corresponding to the audio sample data. The audio sample data contains multiple video sample frames.

[0155] Step S802: Input multiple video sample frames into the behavior recognition model, and extract multi-layer action features of multiple video frames through the behavior recognition model. The action features of different layers have different spatial scales.

[0156] Step S803: Perform correlation analysis on the multi-layer action features to obtain multi-layer correlation features. The spatial scale of the correlation features of different layers is different. Fuse the multi-layer correlation features to obtain multi-scale fused correlation features. Then, integrate the multi-scale fused correlation features into the action features extracted by the feature extraction module to obtain fused features.

[0157] Step S804: Based on the fusion features, perform behavior classification and recognition to obtain the behavior recognition results of the target in the video sample data.

[0158] The specific implementation of steps S802-S804 is similar to that of steps S302-S304. Please refer to the relevant descriptions in the above embodiments for details. This embodiment will not repeat them here.

[0159] Step S805: Update the parameters of the behavior recognition model based on the behavior recognition results and corresponding behavior categories of the target in the video sample data.

[0160] In this embodiment, the specific implementation method of updating model parameters based on behavior recognition results and labeled behavior categories is similar to the implementation method of updating model parameters when training video behavior recognition models based on machine learning in the prior art, and will not be described in detail here.

[0161] The trained behavior recognition model can be based on Figure 1 or Figure 2 Deploy the network structure shown.

[0162] The method described in this application can be used to train a behavior recognition model that can capture more refined motion features, such as the movement information of small objects and rapid movement information, thereby improving the robustness and effectiveness of the behavior recognition model in practical application scenarios.

[0163] Figure 9 This is a schematic diagram of a video behavior recognition system provided as an exemplary embodiment of this application. Figure 9 As shown, the video behavior recognition system 900 provided in this embodiment includes: an end-side device 901 and a cloud-side device 902 that is communicatively connected to the end-side device 901.

[0164] The edge device 901 acquires the video data to be identified, which contains multiple video frames. This process is similar to the implementation of step S301 above, and will not be repeated here.

[0165] The cloud-side device 902 inputs multiple video frames into a behavior recognition model, extracting multi-layered action features from these frames. Different layers of action features have different spatial scales. Correlation analysis is performed on these multi-layered action features to obtain multi-layered correlation features. These correlation features also have different spatial scales. Multi-layered correlation features are fused to obtain multi-scale fused correlation features, which are then integrated into the extracted action features to obtain fused features. Based on these fused features, behavior classification and recognition are performed to obtain the behavior recognition result of the target in the video data. The behavior recognition result is then sent to the edge device 901. The specific implementation of this process is similar to steps S302-S304 described above; please refer to the relevant descriptions in the above embodiments for details, which will not be repeated here.

[0166] The edge device 901 performs post-processing based on the behavior recognition results and outputs the post-processing results.

[0167] In this embodiment, the end-side device 901 can be an edge cloud device deployed at the network edge by various network platforms, responsible for collecting various types of data generated by terminal devices within the coverage area of ​​the end-side device. The end-side device 901 can be a server-side device such as a conventional server, cloud server, or server array. Terminal devices include, but are not limited to, desktop computers, laptops, or smartphones. The various types of data generated by the terminal devices include, but are not limited to, collected videos, such as user-side videos collected during human-computer interaction, or videos of teachers lecturing in smart education. Network platforms include, but are not limited to, e-commerce platforms, short video platforms, news and information platforms, and education and training platforms.

[0168] For example, Figure 10 This is a flowchart illustrating a video behavior recognition method provided as an exemplary embodiment of this application. In this embodiment, the end-side device can be a roadside or vehicle-mounted camera used to capture video of the driver during vehicle movement. Figure 10 As shown, the specific steps of this method are as follows:

[0169] Step S1: The end-side device collects video data of the driver during the vehicle's driving process.

[0170] Optionally, the collected driver video data can be sampled to obtain multiple video frames, which can then be used as video data to be identified for subsequent processing steps.

[0171] Step S2: The terminal device sends the driver's video data to the cloud device.

[0172] Step S3: The cloud-side device inputs multiple video frames into the behavior recognition model, extracts multi-layered action features from the multiple video frames through the behavior recognition model. The action features of different layers have different spatial scales. Correlation analysis is performed on the action features of each layer to obtain multi-layered correlation features. The spatial scales of the correlation features of different layers are different. The multi-layered correlation features are fused to obtain multi-scale fused correlation features. The multi-scale fused correlation features are then incorporated into the extracted action features to obtain fused features. Based on the fused features, behavior classification and recognition are performed to obtain the behavior recognition results of the target in the video data.

[0173] The specific implementation method of this step is similar to that of steps S302-S304 above. Please refer to the relevant descriptions in the above embodiments for details. This embodiment will not repeat them here.

[0174] Step S4: The cloud-side device sends the behavior recognition results to the end-side device.

[0175] Step S5: Based on the behavior recognition results, when the terminal device determines that the driver has a preset unsafe driving behavior, it outputs a warning message to the driver through the output device.

[0176] In this embodiment, taking the driver behavior detection scenario applied in the field of intelligent transportation as an example, the process of linking the terminal device and the cloud device to realize video behavior recognition is explained.

[0177] Figure 11 This is a schematic diagram of a video behavior recognition device provided in an example embodiment of this application. The device provided in this embodiment is used to perform... Figure 3 The video behavior recognition method shown is as follows. Figure 11 As shown, the video behavior recognition device 110 includes: a video data acquisition module 111, a feature extraction module 112, a correlation module 113, and a classification and recognition module 114.

[0178] The video data acquisition module 111 is used to acquire the video data to be identified, which contains multiple video frames.

[0179] The feature extraction module 112 is used to input multiple video frames into the behavior recognition model and extract multi-layer action features of multiple video frames through the behavior recognition model. The action features of different layers have different spatial scales.

[0180] The correlation module 113 is used to perform correlation analysis on the multi-layer motion features to obtain multi-layer correlation features. The spatial scale of the correlation features of different layers is different. The multi-layer correlation features are fused to obtain multi-scale fused correlation features, and the multi-scale fused correlation features are incorporated into the extracted motion features to obtain fused features.

[0181] The classification and recognition module 114 is used to perform behavior classification and recognition based on the fused features to obtain the behavior recognition results of the target in the video data.

[0182] In an optional embodiment, the behavior recognition model includes a feature extraction module, which comprises multiple feature extraction layers stacked from top to bottom. The output of the upper layer serves as the input of the lower layer, and multiple video frames are input into the uppermost feature extraction layer. When performing correlation analysis on the action features of each layer to obtain multi-layer correlation features, where the spatial scales of the correlation features of different layers are different, and fusing the multi-layer correlation features to obtain multi-scale fused correlation features, and then incorporating the multi-scale fused correlation features into the extracted action features to obtain fused features, the correlation module 113 is further used for:

[0183] In the multi-layer feature extraction layer, correlation analysis is performed on the action features extracted in the current layer to obtain the correlation features of the current layer; the correlation features of the current layer are fused with the correlation features of at least one upper layer to obtain the multi-scale fused correlation features of the current layer; the multi-scale fused correlation features of the current layer are fused with the action features to obtain the output features of the current layer; the output features of the bottom layer feature extraction layer are the fused features output by the feature extraction module.

[0184] In an optional embodiment, the behavior recognition model includes a feature extraction module, which comprises multiple feature extraction layers stacked from top to bottom. The output of the upper layer serves as the input of the lower layer, and multiple video frames are input into the uppermost feature extraction layer. When performing correlation analysis on the action features of each layer to obtain multi-layer correlation features, where the spatial scales of the correlation features of different layers are different, and fusing the multi-layer correlation features to obtain multi-scale fused correlation features, and then incorporating the multi-scale fused correlation features into the extracted action features to obtain fused features, the correlation module 113 is further used for:

[0185] In a multi-layer feature extraction layer, correlation analysis is performed on the action features extracted in the current layer to obtain the correlation features of the current layer. The correlation features of the current layer are then fused with the action features to obtain the output features of the current layer. The correlation features of multiple layers are fused to obtain the multi-scale fused correlation features. The multi-scale fused correlation features are then fused with the output features of the lowest layer to obtain the fused features.

[0186] In an optional embodiment, when performing correlation analysis on the action features of any layer to obtain the correlation features of the current layer, the correlation module 113 is further configured to:

[0187] The motion features of any layer contain the motion features of one frame corresponding to each video frame. Each frame of motion features is taken as the target frame. At least one neighboring frame of the target frame is determined in the motion features of the current layer. The correlation between the target frame and each neighboring frame is calculated to obtain the correlation features between the target frame and each neighboring frame. The correlation features between the target frame and at least one neighboring frame are fused to obtain the correlation features corresponding to the target frame of the current layer.

[0188] In an optional embodiment, when determining at least one adjacent frame of the target frame from the motion features of the current layer, the correlation module 113 is further configured to:

[0189] Based on the preset number configured, determine a preset number of adjacent video frames corresponding to the target frame from the motion features of the current layer. The preset number is greater than 1. Use the motion features corresponding to the adjacent video frames as the adjacent frames of the target frame.

[0190] In an optional embodiment, when performing correlation calculation between the target frame and each adjacent frame to obtain the correlation features between the target frame and each adjacent frame, the correlation module 113 is further configured to:

[0191] Based on the configured feature block size, multiple first feature blocks are determined in the target frame, and second feature blocks that match the first feature blocks are determined in adjacent frames; the similarity between each first feature block and the matching second feature block is calculated, and the correlation features between the target frame and each adjacent frame are determined based on the similarity between each first feature block and the matching second feature block.

[0192] In an optional embodiment, when calculating the similarity between each first feature block and a matching second feature block, the correlation module 113 is further configured to:

[0193] Calculate the inner product of each first feature block and its matching second feature block;

[0194] or,

[0195] Calculate the cosine similarity between each first feature block and its matching second feature block.

[0196] In an optional embodiment, when fusing multi-layer correlation features to obtain multi-scale fused correlation features, the correlation module 113 is further configured to:

[0197] The correlation feature with the smallest spatial scale among the multi-layer correlation features to be fused is taken as the target feature; the correlation features to be fused other than the target feature are downsampled, and the downsampled features have the same spatial scale as the target features; the downsampled features are fused with the target features to obtain the multi-scale fused correlation features.

[0198] In an optional embodiment, when acquiring the video data to be identified, the correlation module 113 is further configured to:

[0199] The video to be identified is acquired, and multiple video frames are sampled from the video to be identified. These multiple video frames constitute the video data to be identified.

[0200] The device provided in this embodiment can be specifically used to perform the above-described... Figure 1 The specific functions and technical effects of any embodiment of the video behavior recognition method shown will not be elaborated here.

[0201] Figure 12 This is a schematic diagram of a video behavior recognition device provided in another example embodiment of this application. The device provided in this embodiment is used to perform... Figure 7 The video behavior recognition method shown is as follows. Figure 12 As shown, the video behavior recognition device 120 includes: a video frame acquisition module 1201, a behavior recognition module 1202, and a post-processing module 1203.

[0202] The video frame acquisition module 1201 is used to acquire the video to be identified and to sample the video to obtain multiple video frames.

[0203] The behavior recognition module 1202 is used to input multiple video frames into an end-to-end behavior recognition model, perform behavior recognition through the behavior recognition model, and obtain behavior recognition results. The behavior recognition model includes a feature extraction module and a correlation module. The feature extraction module contains multiple feature extraction layers with different spatial scales, used to extract multi-layer action features based on multiple input video frames. The correlation module is used to perform correlation analysis on the multi-layer action features to obtain multi-layer correlation features. The spatial scale of the correlation features of different layers is different. The multi-layer correlation features are fused to obtain multi-scale fused correlation features, and the multi-scale fused correlation features are integrated into the action features extracted by the feature extraction module to obtain fused features. The behavior recognition model also includes a classification recognition module, which is used to perform behavior classification recognition based on the fused features to obtain the behavior recognition results of the target in the video data.

[0204] The post-processing module 1203 is used to perform post-processing based on the behavior recognition results and output the post-processing results.

[0205] The device provided in this embodiment can be specifically used to perform the above-described... Figure 7 The specific functions and technical effects of the video behavior recognition method shown will not be elaborated here.

[0206] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an example embodiment of this application. Figure 13As shown, the electronic device 130 includes a processor 1301 and a memory 1302 communicatively connected to the processor 1301, the memory 1302 storing computer execution instructions.

[0207] The processor executes computer execution instructions stored in the memory to implement the solution provided in any of the above method embodiments. The specific functions and technical effects that can be achieved will not be elaborated here.

[0208] This application also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, the computer-executable instructions are used to implement the solution provided in any of the above method embodiments. The specific functions and technical effects to be achieved are not described here.

[0209] This application also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium. The at least one processor executes the computer program to cause the electronic device to perform the solution provided in any of the above method embodiments. The specific functions and technical effects that can be achieved are not described here.

[0210] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence number itself does not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0211] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0212] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A video behavior recognition method, characterized in that, include: Acquire video data to be identified, wherein the video data contains multiple video frames; The multiple video frames are input into the behavior recognition model, and the behavior recognition model extracts multi-layer action features of the multiple video frames. The action features of different layers have different spatial scales. The behavior recognition model includes a feature extraction module, which includes multiple feature extraction layers stacked from top to bottom. The output of the upper layer is used as the input of the lower layer, and the multiple video frames are input into the uppermost feature extraction layer. Correlation analysis is performed on the motion features of each layer to obtain the corresponding layer correlation features. The correlation features of each layer are then fused with at least one corresponding upper layer correlation feature to obtain the multi-scale fused correlation features of each layer. The multi-scale fusion correlation features of each layer are fused with the corresponding layer action features to obtain the output features of each layer, and the output features of the bottom layer feature extraction layer are used as the fusion features. or, In a multi-layer feature extraction layer, correlation analysis is performed on the action features extracted in the current layer to obtain the correlation features of the current layer, and the correlation features of the current layer are fused with the action features to obtain the output features of the current layer; Multi-scale fused correlation features are obtained by fusing multi-layer correlation features, and the multi-scale fused correlation features are fused with the output features of the lowest layer to obtain fused features. Based on the fusion features, behavior classification and recognition are performed to obtain the behavior recognition results of the target in the video data.

2. The method according to claim 1, characterized in that, The process involves performing correlation analysis on the action features of each layer to obtain the corresponding layer correlation features, fusing each layer correlation feature with at least one corresponding upper-layer correlation feature to obtain the multi-scale fused correlation features of each layer; fusing each layer's multi-scale fused correlation features with the corresponding layer action features to obtain the output features of each layer, using the output features of the lowest-level feature extraction layer as the fused features, including: In a multi-layer feature extraction layer, correlation analysis is performed on the action features extracted in the current layer to obtain the correlation features of the current layer; The correlation features of the current layer are fused with the correlation features of at least one upper layer to obtain the multi-scale fused correlation features of the current layer. The correlation features of the multi-scale fusion of the current layer are fused with the action features to obtain the output features of the current layer; The output features of the bottommost feature extraction layer are the fused features output by the feature extraction module.

3. The method according to any one of claims 1-2, characterized in that, In the multi-layer feature extraction process, correlation analysis is performed on the action features extracted in the current layer to obtain the correlation features of the current layer, including: The motion features of any layer include a frame motion feature corresponding to each video frame. Each frame motion feature is taken as the target frame, and at least one adjacent frame of the target frame is determined in the motion features of the current layer. The correlation between the target frame and each of the adjacent frames is calculated to obtain the correlation features between the target frame and each of the adjacent frames; The correlation features between the target frame and at least one adjacent frame are fused to obtain the correlation features corresponding to the target frame in the current layer.

4. The method according to claim 3, characterized in that, Determining at least one neighboring frame of the target frame from the motion features of the current layer includes: Based on a preset number, a preset number of adjacent video frames corresponding to the target frame are determined from the action features of the current layer, wherein the preset number is greater than 1; The motion features corresponding to the adjacent video frames are used as the adjacent frames of the target frame.

5. The method according to claim 3, characterized in that, The step of calculating the correlation between the target frame and each of the adjacent frames to obtain the correlation features between the target frame and each of the adjacent frames includes: Based on the configured feature block size, a plurality of first feature blocks are determined in the target frame, and a second feature block matching the first feature block is determined in the adjacent frame; Calculate the similarity between each first feature block and a matching second feature block, and determine the correlation features between the target frame and each of the adjacent frames based on the similarity between each first feature block and a matching second feature block.

6. The method according to claim 1, characterized in that, The fusion of multi-layer correlation features yields multi-scale fused correlation features, including: The correlation feature with the smallest spatial scale among the multi-layer correlation features to be fused is taken as the target feature. The relevant features to be fused, excluding the target features, are downsampled, and the downsampled features have the same spatial scale as the target features. The downsampled features are fused with the target features to obtain multi-scale fused correlation features.

7. The method according to claim 1, characterized in that, The acquisition of the video data to be identified includes: The video to be identified is acquired, and multiple video frames are sampled from the video to be identified. The multiple video frames constitute the video data to be identified.

8. A video behavior recognition system, characterized in that, include: An end-side device is used to acquire video data to be identified, the video data containing multiple video frames; A cloud-based device is used to input the multiple video frames into a behavior recognition model, and to extract multi-layered action features from the multiple video frames through the behavior recognition model. The action features of different layers have different spatial scales. The behavior recognition model includes a feature extraction module, which comprises multiple feature extraction layers stacked from top to bottom. The output of the upper layer serves as the input of the lower layer, and the multiple video frames are input into the uppermost feature extraction layer. Correlation analysis is performed on the action features of each layer to obtain the corresponding layer correlation features. The correlation features of each layer are fused with at least one corresponding upper-layer correlation feature to obtain multi-scale fused correlation features for each layer. The scale-fused correlation features are fused with the corresponding layer action features to obtain the output features of each layer, and the output features of the bottom layer feature extraction layer are used as the fused features; or, in a multi-layer feature extraction layer, correlation analysis is performed on the action features extracted in the current layer to obtain the correlation features of the current layer, and the correlation features of the current layer are fused with the action features to obtain the output features of the current layer; the correlation features of multiple layers are fused to obtain multi-scale fused correlation features, and the multi-scale fused correlation features are fused with the output features of the bottom layer to obtain the fused features; based on the fused features, behavior classification and recognition are performed to obtain the behavior recognition results of the target in the video data; The cloud-side device is also used to send the behavior recognition result to the end-side device; The end-side device is also used to perform post-processing based on the behavior recognition result and output the post-processing result.

9. The system according to claim 8, characterized in that, The end-side device acquires the video to be identified, including: Collect video data of the driver during vehicle operation; The edge device performs post-processing based on the behavior recognition result and outputs the post-processing result, including: Based on the behavior recognition results, when it is determined that the driver has engaged in a preset unsafe driving behavior, a warning message is output to the driver through the output device.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-7.

Images