Video behavior recognition method and device, electronic equipment and storage medium
Feature extraction and fusion of videos through preprocessing networks is solved, and the problems of high computing cost and limited timing processing capabilities in the prior art when processing long-term videos or high-resolution videos are achieved, achieving higher video behavior recognition accuracy and lower computing complexity.
Patent Information
- Application Number
- CN202411808234.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-05-02
AI Technical Summary
When processing long-term video or high-resolution video, existing convolutional neural networks have high computing costs and limited timing processing capabilities, resulting in low accuracy.
The video to be identified is processed by the pre-processing network, and the first video frame sequence and the second video frame sequence of different lengths are determined, and feature extraction and aggregation are performed respectively to obtain the fusion feature. Then, global feature extraction is performed on the fusion feature to capture the timing information in the video.
It improves the accuracy of video behavior recognition, reduces computational complexity and information loss, adapts to videos of different lengths, and enhances the capture of timing information.
Smart Images
Figure CN119919847A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a video behavior recognition method, device, electronic device, and computer-readable storage medium. Background Art
[0002] Video behavior analysis refers to the technology of analyzing video data to identify and classify the actions or behaviors of people or objects contained in the video. It is widely used in security monitoring, traffic management, retail management and other fields.
[0003] Currently, convolutional neural networks are used to learn hierarchical features of videos to achieve automatic recognition of video behaviors. However, convolutional neural networks have high computational costs and limited timing processing capabilities, which means that they require a lot of computing resources and have low accuracy when processing long or high-resolution videos. Summary of the invention
[0004] The embodiments of the present application provide a video behavior recognition method, device, electronic device, and storage medium, which can improve the accuracy of video behavior recognition.
[0005] In a first aspect, an embodiment of the present application provides a video behavior recognition method, comprising:
[0006] The video to be identified is used as an input of a preprocessing network to obtain a fusion feature output by the preprocessing network, wherein the preprocessing network is used to determine a first video frame sequence and a second video frame sequence of different lengths according to the video to be identified, perform feature extraction on the first video frame sequence and the second video frame sequence respectively, and aggregate the extracted features to obtain the fusion feature;
[0007] Performing global feature extraction on the fused features to obtain features to be identified;
[0008] A behavior recognition result is determined according to the feature to be recognized, and the behavior recognition result is used to indicate the category of the behavior in the video to be recognized.
[0009] In a possible implementation manner of the first aspect, the preprocessing network includes a fast branch, a slow branch, and a fusion network, and taking the video to be recognized as an input of the preprocessing network to obtain a fusion feature output by the preprocessing network includes:
[0010] Performing a first sampling process on the to-be-identified video through the fast branch, and extracting features from the first video frame sequence obtained by sampling to obtain dynamic features;
[0011] Performing a second sampling process on the video to be identified through the slow branch, and performing feature extraction on the second video frame sequence obtained by sampling to obtain a static feature, wherein the length of the first video frame sequence is greater than the length of the second video frame sequence;
[0012] The dynamic feature and the static feature are fused through the fusion network to obtain the fusion feature.
[0013] In a possible implementation manner of the first aspect, performing a first sampling process on the to-be-identified video by using the fast branch includes:
[0014] The fast branch is used to perform resolution reduction processing on each video frame of the video to be identified, and the first video frame sequence is determined according to each video frame after the resolution reduction. The resolution reduction processing is used to reduce the resolution of the video frame while maintaining the size of the video frame.
[0015] In a possible implementation manner of the first aspect, before fusing the dynamic feature and the static feature through the fusion network to obtain the fused feature, the method further includes:
[0016] Merging the features of different channels in the static features along the channel direction through the slow branch to obtain the merged static features;
[0017] Correspondingly, the fusing the dynamic features and the static features through the fusion network to obtain the fused features includes:
[0018] The dynamic features and the combined static features are fused through the fusion network to obtain the fused features.
[0019] In a possible implementation manner of the first aspect, before performing global feature extraction on the fused features to obtain features to be identified, the method further includes:
[0020] Performing embedding processing on the fused feature to obtain the embedded fused feature, wherein the embedding processing is used to embed at least one of CLS Token, location information and time information in the fused feature;
[0021] Correspondingly, performing global feature extraction on the fused features to obtain features to be identified includes:
[0022] Perform global feature extraction on the embedded fusion features to obtain the features to be identified.
[0023] In a possible implementation manner of the first aspect, performing global feature extraction on the fused features to obtain features to be identified includes:
[0024] The fused features are subjected to global feature extraction through a Mamba-based encoder to obtain the features to be identified.
[0025] In a second aspect, an embodiment of the present application provides a video behavior recognition device, including:
[0026] A preprocessing module, used to use the video to be identified as an input of a preprocessing network to obtain a fusion feature output by the preprocessing network, wherein the preprocessing network is used to determine a first video frame sequence and a second video frame sequence of different lengths according to the video to be identified, perform feature extraction on the first video frame sequence and the second video frame sequence respectively, and aggregate the extracted features to obtain the fusion feature;
[0027] A global feature extraction module is used to perform global feature extraction on the fused features to obtain features to be identified;
[0028] The recognition module is used to determine a behavior recognition result according to the feature to be recognized, and the behavior recognition result is used to indicate the category of the behavior in the video to be recognized.
[0029] In a possible implementation manner of the second aspect, the preprocessing network includes a fast branch, a slow branch, and a fusion network, and the preprocessing module includes:
[0030] A fast branch processing unit, configured to perform a first sampling process on the to-be-identified video through the fast branch, and extract features from the first video frame sequence obtained by sampling to obtain dynamic features;
[0031] a slow branch processing unit, configured to perform a second sampling process on the video to be identified through the slow branch, and perform feature extraction on the second video frame sequence obtained by sampling to obtain a static feature, wherein the length of the first video frame sequence is greater than the length of the second video frame sequence;
[0032] A fusion unit is used to fuse the dynamic feature and the static feature through the fusion network to obtain the fusion feature.
[0033] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the video behavior recognition method described in the first aspect when executing the computer program.
[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video behavior recognition method described in the first aspect are implemented.
[0035] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the video behavior recognition method described in the first aspect above.
[0036] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0037] In an embodiment of the present application, the video to be identified is first processed through a preprocessing network. Since the preprocessing network will first process the video to be identified into two video frame sequences of different lengths, and then extract features from the two video frame sequences and aggregate the extracted features, it can better adapt to videos to be identified of different lengths, reduce redundant information in the fused features, and avoid problems such as high computational complexity and information loss that may be caused by directly processing the entire video. Furthermore, global features are extracted from the fused features with less redundant information, the timing information in the video is captured, and the accuracy of the features to be identified is improved, thereby improving the accuracy of the final behavior recognition results. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0039] Figure 1 It is a flowchart of a video behavior recognition method provided by an embodiment of the present application;
[0040] Figure 2 This is a schematic diagram of a process of processing a video to be identified by a preprocessing network provided in an embodiment of the present application;
[0041] Figure 3 This is a flow chart of another preprocessing network provided in an embodiment of the present application for processing a video to be identified;
[0042] Figure 4 It is a structural diagram of a video behavior recognition model provided in an embodiment of the present application;
[0043] Figure 5 is a structural diagram of a video behavior recognition device provided in an embodiment of the present application;
[0044] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0046] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0047] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0048] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0049] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.
[0050] Embodiment 1:
[0051] Figure 1 A schematic diagram of a video behavior recognition method provided by an embodiment of the present application is shown, and is described in detail as follows:
[0052] S101. Using the video to be identified as the input of a preprocessing network to obtain fusion features output by the preprocessing network, wherein the preprocessing network is used to determine a first video frame sequence and a second video frame sequence of different lengths according to the video to be identified, respectively extract features from the first video frame sequence and the second video frame sequence, and aggregate the extracted features to obtain the fusion features.
[0053] It should be understood that the method for obtaining the video to be identified can be set according to the actual application scenario.
[0054] For example, in a retail management scenario, the videos to be identified may include videos shot by cameras installed on unmanned vending machines, as well as videos shot by surveillance cameras inside stores. Video behavior recognition can be used to analyze customer behavior, prevent theft and damage to videos, and provide personalized shopping experiences and analyze customer flow peaks.
[0055] For example, in a smart security scenario, the video to be identified can be a video taken by a smart access control camera (such as a cat-eye camera or an infrared camera), which can monitor and record the behavior of people entering and leaving in real time through video behavior recognition, thereby optimizing management efficiency, effectively preventing illegal intrusions, and improving security management levels.
[0056] It should be understood that the preprocessing network can be a network built and trained based on the actual application scenario requirements, or it can be a general pre-trained model (such as a large language model), and the embodiments of the present application do not impose specific restrictions on this.
[0057] As an example, a preprocessing network can be constructed based on the deep learning architecture Slow-Fast including fast branches and slow branches, and training samples corresponding to actual application scenarios can be obtained by manually collecting annotations or generating images, and the preprocessing network constructed by this training can be obtained to obtain a pre-trained preprocessing network with high adaptability to actual application scenarios. Furthermore, when constructing a pretraining network, a lightweight preprocessing network can be constructed by reducing the channel capacity of feature extraction, so that the preprocessing network can also better process the video to be recognized when computing resources are effective, and achieve efficient and accurate video sampling and feature extraction processing.
[0058] Optionally, when constructing a preprocessing network based on a two-dimensional image-based network structure such as Slow-Fast, the original two-dimensional convolutional neural network in the network can be replaced with a three-dimensional convolutional neural network, so that the constructed preprocessing network can better process the entire video to be identified. Exemplarily, the RepVit (Representative Vision Transformer) component composed of a three-dimensional convolutional neural network can be sampled to replace the two-dimensional convolutional neural network in the network structure, and the RepVit component is a lightweight convolutional neural network that draws on the lightweight ViT (Vision Transformer). The preprocessing network constructed based on the RepVit component has good performance on mobile devices such as mobile phones, and can better capture the spatial and temporal features in the video, effectively improving the processing efficiency and accuracy of the preprocessing network.
[0059] As another example, a pre-trained general large model such as Florence-2 can be used as a pre-processing network. Since the general large model far exceeds the conventional machine learning model in terms of training data, model parameter quantity and computing resources, it can better handle a variety of tasks and applications. Therefore, the use of a pre-trained general large model as a pre-processing network does not require additional construction and training of the network, effectively reducing development costs, and at the same time can better adapt to different video sampling requirements and feature extraction requirements, which is conducive to practical applications. It should be understood that when a general large model is used as a pre-processing network, the video to be identified and its corresponding video sampling requirements and feature extraction requirements can be input into the general large model together, so that the general large model can flexibly process the video to be identified in combination with actual needs.
[0060] It should be understood that the preprocessing network can use different sampling frequencies to sample the video to be recognized, so as to obtain the first video frame sequence and the second video frame sequence of different lengths (the length is the number of video frames). It can also perform key frame extraction or random extraction on the video to be recognized based on different quantity requirements (such as 150 frames and 100 frames) to obtain the first video frame sequence and the second video frame sequence of different lengths. The embodiment of the present application does not impose any specific limitation on the specific implementation method of sampling to obtain the first video frame sequence and the second video frame sequence.
[0061] Optionally, the length of the first video frame sequence (assuming it is called the first length) and the length of the second video frame sequence (assuming it is called the second length) may be a multiple relationship, such as the first length may be 4 times the second length.
[0062] In an embodiment of the present application, two video frame sequences of different lengths are sampled from the video to be identified through a preprocessing network, so that different processing methods can be used to extract information from the two video frame sequences of different lengths. This can better adapt to videos to be identified of different lengths. At the same time, there is no need to directly perform feature extraction processing on the entire video to be identified, which effectively reduces the probability of high computational complexity and information loss during the processing process.
[0063] S102: Perform global feature extraction on the above fused features to obtain features to be identified.
[0064] Global feature extraction refers to extracting feature information from data such as videos or images that can fully represent their content.
[0065] It should be understood that when performing global feature extraction on the fused features, the global feature extraction of the fused features can be achieved through convolutional neural networks, recurrent neural networks or global average pooling, and the embodiments of the present application do not impose specific restrictions on this.
[0066] Optionally, before performing global feature extraction on the fused features, the fused features can be first subjected to dimensionality reduction processing. The dimensionality of the fused features can be reduced by dimensionality reduction processing, thereby reducing the amount of computation for global feature extraction and subsequent recognition. At the same time, key information useful for subsequent processing can be retained as much as possible, thereby ensuring the quality of the fused features after dimensionality reduction.
[0067] As an example, the fused features can be converted into one-dimensional features through dimensionality reduction processing, which can significantly simplify the data structure of the fused features and reduce the time and resources required for calculation, thereby effectively improving the efficiency of global feature extraction and subsequent analysis.
[0068] In the embodiment of the present application, global features are extracted from the fused features extracted by the preprocessing network, which can better integrate the temporal and spatial information in the video to be identified, capture the overall content and context of the video, and obtain features to be identified with higher accuracy. Subsequently, subsequent identification based on the features to be identified can achieve more accurate and comprehensive video behavior recognition.
[0069] S103: determining a behavior recognition result according to the features to be recognized, wherein the behavior recognition result is used to indicate a category of the behavior in the video to be recognized.
[0070] It should be understood that when determining the behavior recognition result based on the features to be identified, the features to be identified can be identified through multimodal (such as vision, sound and text) fusion recognition, posture estimation analysis, probability statistics or classification network to obtain the required behavior recognition result.
[0071] It should be understood that when determining the behavior recognition result of the video to be recognized based on the features to be recognized, the corresponding behavior can be recognized for the features to be recognized in combination with the actual application scenario to obtain the desired behavior recognition result.
[0072] For example, in a retail management scenario, when determining the behavior recognition result based on the features to be recognized, it can be to recognize the category of human actions (such as stopping to view or adding to a purchase, etc.). For another example, in a public security management scenario, when determining the behavior recognition result of a video to be recognized based on the features to be recognized, it can be to determine whether the human behavior in the video to be recognized belongs to a specific behavior (such as fighting or stealing, etc.). At this time, the category of the behavior can include specific behavior and non-specific behavior, that is, the behavior recognition result can indicate whether the behavior in the video to be recognized belongs to a specific behavior.
[0073] In the embodiment of the present application, since the preprocessing network will first process the video to be identified into two video frame sequences of different lengths when processing the video to be identified, and then perform feature extraction on the two video frame sequences respectively and aggregate the extracted features, it can better adapt to videos to be identified of different lengths, reduce redundant information in the fused features, and avoid problems such as high computational complexity and information loss that may be caused by directly processing the entire video. Furthermore, global features are extracted from the fused features with less redundant information, the timing information in the video is captured, and the accuracy of the features to be identified is improved, thereby improving the accuracy of the final behavior recognition results.
[0074] In some embodiments, the preprocessing network includes a fast branch, a slow branch and a fusion network, and the step S101 includes:
[0075] A1. Performing a first sampling process on the video to be identified through the fast branch, and extracting features from the first video frame sequence obtained by sampling to obtain dynamic features.
[0076] A2. Perform a second sampling process on the video to be identified through the slow branch, and perform feature extraction on the second video frame sequence obtained by sampling to obtain static features, wherein the length of the first video frame sequence is greater than the length of the second video frame sequence.
[0077] A3. The dynamic features and the static features are fused through the fusion network to obtain the fusion features.
[0078] Since the length of the first video frame sequence obtained by the fast branch through the first sampling processing of the video to be identified is relatively long, that is, the fast branch adopts a high sampling rate with a short time interval, therefore, when the fast branch performs feature extraction based on the first video frame sequence obtained at the high sampling rate, it can focus on the details of the time dimension, better capture the rapidly changing features in the video to be identified, such as human body movements or object motion trajectories, and obtain dynamic features with higher accuracy.
[0079] The slow branch uses a low sampling rate with a relatively long time interval to perform a second sampling process on the video to be identified, and obtains a shorter second video frame sequence. When performing feature extraction based on the second video frame sequence obtained by sampling at the low sampling rate, it can focus on the global information in the spatial and temporal dimensions, better capture the global static information in the video to be identified, and obtain static features with higher accuracy.
[0080] It should be understood that the processing of fast branches and slow branches is performed in parallel. Figure 2As shown in the figure, after the dynamic features and static features focusing on different dimensions are extracted by the fast branch and the slow branch, the complementary dynamic features and static features are fused through the fusion network. The information of the time dimension and the space dimension can be integrated through the fusion process to obtain the fusion features that can more accurately and comprehensively reflect the behavior in the video to be identified, which is helpful for the subsequent accurate identification of the behavior in the video to be identified.
[0081] It should be understood that the fast branch and the slow branch can be constructed based on neural networks such as convolutional neural networks and recurrent neural networks, or can be implemented through a general large model. The embodiments of the present application do not impose specific restrictions on this.
[0082] Optionally, the slow branch can be constructed based on a convolutional neural network such as VGG (Visual Geometry Group Network) or DenseNet that can apply multi-size convolution kernels, so that when the slow branch performs feature extraction on the second video frame sequence, it can extract features of different scales in the video frames through convolution kernels of different sizes, which helps to capture richer spatial features, thereby improving the accuracy of the final static features.
[0083] Optionally, the fast branch can be constructed based on a lightweight convolutional neural network such as ShuffleNet or EfficientNet, and the computational complexity of the fast branch when performing feature extraction on a longer first video frame sequence is reduced through a lightweight network design.
[0084] Optionally, the fusion network can be a network constructed based on sequence-related network structures such as graph neural networks, recurrent neural networks, or long short-term memory networks, so that the fusion network can better fuse features corresponding to video frame sequences of different lengths to ensure the accuracy of the obtained fusion features.
[0085] For example, the fusion network can be a network built based on a graph neural network. When fusing dynamic features and static features, the fusion network can determine the graph nodes according to the features corresponding to different video frames in the dynamic features and static features, and determine the edges of the graph according to the relationship between video frames (i.e., the time sequence relationship). The graph neural network can be used to capture the spatiotemporal dependency between video frames to effectively fuse dynamic features and static features, which is conducive to improving the accuracy of subsequent video behavior recognition.
[0086] In the embodiment of the present application, features are first extracted in parallel from the corresponding video frame sequences through the fast branch and the slow branch, which can extract feature information of different dimensions with high processing efficiency. Then, the feature information of different dimensions extracted by the fast branch and the slow branch are fused through the fusion network, which can more comprehensively capture the information in the video to be identified, which is beneficial to improving the recognition accuracy of the video to be identified.
[0087] In some embodiments, before extracting features from the first video frame sequence sampled by the fast branch to obtain dynamic features, the method further includes:
[0088] The fast branch is used to perform resolution reduction processing on each video frame in the first video frame sequence to obtain the first video frame sequence after resolution reduction. The resolution reduction processing is used to reduce the resolution of the video frame while maintaining the size of the video frame.
[0089] Correspondingly, the first video frame sequence sampled by the fast branch is subjected to feature extraction to obtain dynamic features, including:
[0090] The fast branch is used to extract features from the first video frame sequence after resolution reduction to obtain the dynamic features.
[0091] It should be understood that the above-mentioned resolution reduction processing can be implemented through methods such as interpolation algorithm, image generation, image mapping or sparse coding, and the embodiment of the present application does not impose any specific restrictions on the specific implementation method of the resolution reduction processing.
[0092] For example, the fast branch may include a resolution reduction network constructed based on a convolutional neural network, and the training sample of the resolution reduction network is an image pair including a high-resolution image and its corresponding low-resolution image, so that the mapping relationship between the high-resolution image and the low-resolution image can be better learned during the training process of the resolution reduction network. When the fast branch performs resolution reduction processing on each video frame in the first video frame sequence, the video frame is directly used as the input of the resolution reduction network. The resolution reduction network maps the higher-resolution video frame to the low-resolution video frame according to the mapping relationship learned during the training process to obtain the video frame after the resolution reduction. In order to improve the accuracy of the resolution reduction, the corresponding images can be obtained based on the actual application scenario of the video behavior analysis to construct an image pair, which is used as the training sample of the resolution reduction network to improve the processing effect of the resolution reduction network on the video frames in the first video frame sequence.
[0093] In other embodiments, the fast branch may perform resolution reduction processing on each video frame in the first video frame sequence if the first video frame sequence meets the resolution reduction condition, otherwise, the fast branch may directly perform feature extraction on the first video frame sequence. The resolution reduction condition includes: at least one of the first length being greater than or equal to a length threshold (such as 200), the first data volume (i.e., the data volume of the first video frame sequence) being greater than or equal to a data volume threshold (such as 500 megabytes), and the video frame resolution being greater than or equal to a resolution threshold (such as 1920*1080).
[0094] In an embodiment of the present application, the fast branch will first perform resolution reduction processing on each video frame of the first video frame sequence before performing feature extraction on a first video frame sequence with a longer length. The resolution reduction processing can reduce the resolution of the video frame while maintaining the size of the video frame, that is, the video frame can be kept consistent with the size of the video frame in the second video frame sequence, while reducing the amount of data of each video frame, thereby effectively reducing the amount of data of the first video frame sequence finally obtained, and further effectively reducing the amount of data processing and computational complexity when the fast branch performs feature extraction.
[0095] In some embodiments, before extracting features from a shorter second video frame sequence through a slow branch, each video frame in the second video frame sequence may be first processed by upgrading the resolution through a slow branch, and then the slow branch may extract features from the upgraded second video frame sequence to obtain dynamic features.
[0096] The resolution upscaling process is used to improve the resolution of the video frame while keeping the size of the video frame unchanged. The resolution upscaling process can be implemented by, for example, interpolation algorithms, sparse coding, dictionary learning, and image generation. For example, the video frame can be used as the input of a pre-trained adversarial generative network, and the adversarial generative network generates a corresponding high-resolution video frame based on the input video frame to improve the resolution of the video frame. Through the above processing, the video frame after the resolution upscaling has richer detail information, so that the slow branch can capture more useful information from it, thereby improving the accuracy of the static features obtained.
[0097] In other embodiments, the first video frame sequence acquired by the fast branch may include all video frames of the video to be identified, so that when feature extraction is subsequently performed based on the first video frame sequence, dynamic features in the video to be identified can be more comprehensively captured. Figure 3 As shown, during the processing, the fast branch performs resolution reduction processing on the first video frame sequence, and then extracts features from the first video frame sequence after the resolution reduction, thereby reducing the data processing amount and computational complexity of the fast branch. At the same time, the second video frame sequence is processed by the slow branch with increased resolution, so that the slow branch can extract richer and more accurate static features based on the high-resolution video frames. Through the above processing, while ensuring the processing efficiency and computational complexity, the accuracy and comprehensiveness of the extracted features are improved as much as possible, which is conducive to improving the accuracy of subsequent video behavior recognition.
[0098] In some embodiments, before the above step A3, the method further includes:
[0099] The features of different channels in the static features are merged along the channel direction through the slow branch to obtain the merged static features.
[0100] Correspondingly, the above step A3 includes:
[0101] The dynamic features and the combined static features are fused through the fusion network to obtain the fusion features.
[0102] In an embodiment of the present application, in order to reduce the computational complexity of subsequent processing, the slow branch extracts features from the second video frame sequence and obtains static features. It then merges features of different channels in the static features along the channel direction. The number of feature channels is reduced through the merging process, and a merged static feature with a smaller number of channels is obtained.
[0103] At the same time, taking into account the impact of dynamic information on video behavior recognition, in an embodiment of the present application, the features of different channels in the static features are merged through a slow branch, and there is no need to merge the features of different channels in the dynamic features through a fast branch. This reduces the amount of data for subsequent processing while ensuring the accuracy of video behavior recognition.
[0104] It should be understood that when merging the features of different channels in the static features along the channel direction through the slow branch, the features of every two adjacent channels can be merged into the features of one channel, or the features of multiple (such as 5) continuous channels can be merged into the features of one channel. The embodiment of the present application does not impose any specific limitation on the way of merging the features of different channels.
[0105] It should be understood that when the slow branch performs merging processing on features of different channels, it can implement its merging processing by means of weighted summation, maximum value merging, bilinear merging or grouped convolution, etc. For example, the slow branch can implement merging processing on features of different channels by grouped three-dimensional convolution.
[0106] In some embodiments, before the above step S102, the method further includes:
[0107] The above-mentioned fusion feature is embedded to obtain the above-mentioned fusion feature after embedding, and the above-mentioned embedding is used to embed at least one of CLS Token, location information and time information in the above-mentioned fusion feature.
[0108] Correspondingly, the above-mentioned fusion features are subjected to global feature extraction to obtain features to be identified, including:
[0109] Perform global feature extraction on the embedded fusion features to obtain the features to be identified.
[0110] CLS Token is a special token used for classification tasks. Its main function is to capture and represent the global information of the entire input sequence.
[0111] In order to process and understand the fusion features more accurately, before performing global feature extraction on the fusion features, learnable CLS Tokens can be embedded in the fusion features to obtain the embedded fusion features. When performing global feature extraction on the embedded fusion features, CLS Tokens will interact with other features in the embedded fusion features, thereby aggregating the global dependencies in the embedded fusion features, which helps to improve the accuracy of the extracted global features.
[0112] Optionally, when embedding the CLS Token in the fusion feature, the fusion feature can be first converted into a one-dimensional fusion feature through dimensionality reduction processing, and then the CLS Token is embedded in the head of the one-dimensional fusion feature.
[0113] Alternatively, considering the importance of the order of the input sequence, position information can be embedded in the fused feature. Since the position information can reflect the position of the feature in the video frame sequence, embedding the position information in the fused feature can help the model (the model here generally refers to the network used for global feature extraction) understand the order and context of the corresponding video frame sequence. When performing global feature extraction on the embedded fused feature, the model can more accurately capture local dependencies and global dependencies, thereby obtaining more accurate features to be identified.
[0114] Optionally, the position information can be represented by a position encoding such as sinusoidal position encoding or learnable position embedding. When embedding the position information in the fusion feature, the position encoding can be embedded into the fusion feature by addition or concatenation to obtain the embedded fusion feature.
[0115] Alternatively, considering that the video frame sequence obtained based on the video to be identified is a time series, time information is more important for understanding the dynamic changes and trends in the video frame sequence. Therefore, time information can also be embedded in the fusion feature. When the model performs global feature extraction on the fusion feature after the embedding of time information, it can better understand the dynamic changes and timing relationships in the corresponding video frame sequence, thereby enhancing the feature expression ability of the features to be identified, which helps to improve the accuracy of subsequent video behavior recognition.
[0116] Optionally, when embedding time information in the fusion feature, the timestamp or time feature (such as frame number or time difference between adjacent frames, etc.) can be used as an additional input feature, and then embedded into the fusion feature by addition or concatenation. Alternatively, the fusion feature can be convolved in the time dimension through a temporal convolution layer to fuse the temporal relationship in the feature and generate a fusion feature containing time information (i.e., the fusion feature after embedding the time information).
[0117] It should be understood that when embedding the fused feature, it is also possible to embed multiple CLS Token, location information and time information in the fused feature to obtain a fused feature embedded with multiple information, and further improve the feature representation capability of the embedded fused feature by embedding multiple information.
[0118] In an embodiment of the present application, one or more of CLS Token, location information, and time information is embedded in the fusion feature. When global feature extraction is performed on the embedded fusion feature, the embedded information can help the model better capture the global dependencies and temporal relationships in the fusion feature, thereby improving the accuracy of the extracted features to be identified, and helping to improve the accuracy of subsequent video behavior recognition.
[0119] In some embodiments, the above step S102 includes:
[0120] The above fusion features are subjected to global feature extraction through a Mamba-based encoder to obtain the above features to be identified.
[0121] Mamba is a deep learning architecture based on structured state-space sequence models (SSMs) that can capture relevant information in long sequences more efficiently and effectively through selective state spaces.
[0122] A Mamba-based encoder is an encoder composed of at least one Mamba Block module. The MambaBlock module is the basic component of the Mamba model, encapsulating multiple network layers and operation functions for processing input data. The combination of these different network layers and operation functions enables the Mamba Block module to effectively handle complex sequence modeling tasks.
[0123] Exemplarily, the Mamba-based encoder may be a network consisting of a plurality of (eg, 8) stacked Mamba Block modules.
[0124] It should be understood that when the Mamba-based encoder is a network composed of multiple stacked Mamba Block modules, when global feature extraction is performed on the fused features based on the encoder, the fused features are used as the input of the first Mamba Block module, and the first Mamba Block module performs global feature extraction processing on the fused features and outputs the extracted features; the features output by the first Mamba Block module are used as the input of the second Mamba Block module, that is, the features output by the previous MambaBlock module are used as the input of the next Mamba Block module, and so on, and the features output by the last Mamba Block module of the stack are used as the final extracted features to be identified.
[0125] In some embodiments, when the duration of the video to be identified is greater than or equal to a duration threshold (such as 60 minutes), the corresponding fused features may be subjected to global feature extraction through a Mamba-based encoder. When the duration of the video to be identified is less than the duration threshold (such as 60 minutes), the fused features may be subjected to global feature extraction through methods such as global average pooling.
[0126] In the embodiment of the present application, since the Mamba Block module can process long video sequences with linear computational complexity and effectively handle complex sequence modeling tasks, an encoder composed of the Mamba Block module is used to perform global feature extraction on the fused features, which can significantly reduce the computational complexity and processing efficiency of global feature extraction while improving the accuracy of global feature extraction, which is beneficial to practical applications.
[0127] It should be understood that in some embodiments, after the network structures in the above-mentioned embodiments are trained, the trained network structures can be integrated into a model to obtain a video behavior recognition model that can be used for video behavior recognition. As shown in Figure 4, the video behavior recognition model may include a preprocessing network, a global feature extraction network, and a behavior recognition network. Among them, the preprocessing network may include a fast branch, a slow branch, and a fusion network, the global feature extraction network includes 6 stacked MambaBlock modules, and the behavior recognition network includes a classification head composed of a fully connected layer. After obtaining the video to be recognized, the video to be recognized can be directly used as the input of the video behavior recognition model to obtain the behavior recognition result output by the video behavior recognition model.
[0128] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0129] Embodiment 2:
[0130] Corresponding to the video behavior recognition method described in the above embodiment, Figure 5 A structural block diagram of a video behavior recognition device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0131] Reference Figure 5 The device comprises: a preprocessing module 51, a global feature extraction module 52 and a recognition module 53.
[0132] The preprocessing module 51 is used to use the video to be identified as the input of the preprocessing network to obtain the fusion features output by the preprocessing network. The preprocessing network is used to determine the first video frame sequence and the second video frame sequence of different lengths according to the video to be identified, perform feature extraction on the first video frame sequence and the second video frame sequence respectively, and aggregate the extracted features to obtain the fusion features.
[0133] The global feature extraction module 52 is used to perform global feature extraction on the above fused features to obtain features to be identified.
[0134] The recognition module 53 is used to determine a behavior recognition result according to the features to be recognized, and the behavior recognition result is used to indicate the category of the behavior in the video to be recognized.
[0135] In the embodiment of the present application, since the preprocessing network will first process the video to be identified into two video frame sequences of different lengths when processing the video to be identified, and then perform feature extraction on the two video frame sequences respectively and aggregate the extracted features, it can better adapt to videos to be identified of different lengths, reduce redundant information in the fused features, and avoid problems such as high computational complexity and information loss that may be caused by directly processing the entire video. Furthermore, global features are extracted from the fused features with less redundant information, the timing information in the video is captured, and the accuracy of the features to be identified is improved, thereby improving the accuracy of the final behavior recognition results.
[0136] In some embodiments, the preprocessing network includes a fast branch, a slow branch and a fusion network, and the preprocessing module 51 includes:
[0137] The fast branch processing unit is used to perform a first sampling process on the video to be identified through the fast branch, and to extract features from the first video frame sequence obtained by sampling to obtain dynamic features.
[0138] A slow branch processing unit is used to perform a second sampling process on the video to be identified through the slow branch, and to extract features from the second video frame sequence obtained by sampling to obtain static features, wherein the length of the first video frame sequence is greater than the length of the second video frame sequence.
[0139] The fusion unit is used to fuse the dynamic features and the static features through the fusion network to obtain the fusion features.
[0140] In some embodiments, the preprocessing module 51 further includes:
[0141] A resolution reduction processing unit is used to perform resolution reduction processing on each video frame in the above-mentioned first video frame sequence through the above-mentioned fast branch to obtain the above-mentioned first video frame sequence after resolution reduction. The above-mentioned resolution reduction processing is used to reduce the resolution of the above-mentioned video frame while maintaining the size of the video frame.
[0142] The dynamic feature extraction unit is used to extract features from the first video frame sequence after resolution reduction through the fast branch to obtain the dynamic features.
[0143] In some embodiments, the preprocessing module 51 further includes:
[0144] The merging processing unit is used to merge the features of different channels in the static features along the channel direction through the slow branch to obtain the merged static features.
[0145] Correspondingly, the above fusion unit is specifically used for:
[0146] The dynamic features and the combined static features are fused through the fusion network to obtain the fusion features.
[0147] In some embodiments, the global feature extraction module 52 includes:
[0148] The global feature extraction unit is used to perform global feature extraction on the above fusion features through a Mamba-based encoder to obtain the above features to be identified.
[0149] In some embodiments, the video behavior recognition device further includes:
[0150] The embedding module is used to perform embedding processing on the above-mentioned fusion feature to obtain the above-mentioned fusion feature after embedding, and the above-mentioned embedding processing is used to embed at least one of CLS Token, location information and time information in the above-mentioned fusion feature.
[0151] Correspondingly, the above-mentioned global feature extraction includes 52 and also includes:
[0152] The extraction unit is used to perform global feature extraction on the embedded fusion features to obtain the features to be identified.
[0153] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0154] Embodiment three:
[0155] Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 6 As shown, the electronic device 6 of this embodiment includes: at least one processor 60 ( Figure 6 Only one processor is shown in the figure), a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, wherein the processor 60 implements the steps of any of the above-mentioned method embodiments when executing the computer program 62.
[0156] The electronic device 6 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 6 It is only an example of the electronic device 6 and does not constitute a limitation on the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.
[0157] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0158] In some embodiments, the memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. In other embodiments, the memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 6. Further, the memory 61 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 61 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 61 may also be used to temporarily store data that has been output or is to be output.
[0159] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0160] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.
[0161] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0162] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0164] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0165] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0166] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0168] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A video behavior recognition method, characterized in that: include: The video to be identified is used as an input of a preprocessing network to obtain a fusion feature output by the preprocessing network, wherein the preprocessing network is used to determine a first video frame sequence and a second video frame sequence of different lengths according to the video to be identified, perform feature extraction on the first video frame sequence and the second video frame sequence respectively, and aggregate the extracted features to obtain the fusion feature; Performing global feature extraction on the fused features to obtain features to be identified; A behavior recognition result is determined according to the feature to be recognized, and the behavior recognition result is used to indicate the category of the behavior in the video to be recognized.
2. The video behavior recognition method according to claim 1, characterized in that: The preprocessing network includes a fast branch, a slow branch and a fusion network. The video to be identified is used as the input of the preprocessing network to obtain the fusion features output by the preprocessing network, including: Performing a first sampling process on the to-be-identified video through the fast branch, and extracting features from the first video frame sequence obtained by sampling to obtain dynamic features; Performing a second sampling process on the to-be-recognized video through the slow branch, and extracting features from the second video frame sequence obtained by sampling to obtain static features, wherein the length of the first video frame sequence is greater than the length of the second video frame sequence; The dynamic feature and the static feature are fused through the fusion network to obtain the fusion feature.
3. The video behavior recognition method according to claim 2, characterized in that: Before extracting features from the first video frame sequence sampled by the fast branch to obtain dynamic features, the method further includes: Performing resolution reduction processing on each video frame in the first video frame sequence through the fast branch to obtain the first video frame sequence after resolution reduction, wherein the resolution reduction processing is used to reduce the resolution of the video frame while maintaining the size of the video frame; Correspondingly, extracting features from the first video frame sequence sampled by the fast branch to obtain dynamic features includes: The fast branch is used to extract features from the first video frame sequence after resolution reduction to obtain the dynamic features.
4. The video behavior recognition method according to claim 2, characterized in that: Before fusing the dynamic features and the static features through the fusion network to obtain the fusion features, the method further includes: Merging the features of different channels in the static features along the channel direction through the slow branch to obtain the merged static features; Correspondingly, the fusing the dynamic features and the static features through the fusion network to obtain the fused features includes: The dynamic features and the combined static features are fused through the fusion network to obtain the fused features.
5. The video behavior recognition method according to claim 1, characterized in that: The step of performing global feature extraction on the fused features to obtain features to be identified includes: The fused features are subjected to global feature extraction through a Mamba-based encoder to obtain the features to be identified.
6. The video behavior recognition method according to any one of claims 1 to 5, characterized in that: Before extracting global features from the fused features to obtain features to be identified, the method further includes: Performing embedding processing on the fused feature to obtain the embedded fused feature, wherein the embedding processing is used to embed at least one of CLS Token, location information and time information in the fused feature; Correspondingly, performing global feature extraction on the fused features to obtain features to be identified includes: Perform global feature extraction on the embedded fusion features to obtain the features to be identified.
7. A video behavior recognition device, characterized in that: include: A preprocessing module, used to use the video to be identified as an input of a preprocessing network to obtain a fusion feature output by the preprocessing network, wherein the preprocessing network is used to determine a first video frame sequence and a second video frame sequence of different lengths according to the video to be identified, perform feature extraction on the first video frame sequence and the second video frame sequence respectively, and aggregate the extracted features to obtain the fusion feature; A global feature extraction module is used to perform global feature extraction on the fused features to obtain features to be identified; The recognition module is used to determine a behavior recognition result according to the feature to be recognized, and the behavior recognition result is used to indicate the category of the behavior in the video to be recognized.
8. The video behavior recognition device according to claim 7, characterized in that: The preprocessing network includes a fast branch, a slow branch and a fusion network, and the preprocessing module includes: A fast branch processing unit, configured to perform a first sampling process on the to-be-identified video through the fast branch, and extract features from the first video frame sequence obtained by sampling to obtain dynamic features; a slow branch processing unit, configured to perform a second sampling process on the video to be identified through the slow branch, and perform feature extraction on the second video frame sequence obtained by sampling to obtain a static feature, wherein the length of the first video frame sequence is greater than the length of the second video frame sequence; A fusion unit is used to fuse the dynamic feature and the static feature through the fusion network to obtain the fusion feature.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.