Dual-stream recognition method, system, device and medium for video behavior

Abstract

The application discloses a kind of video behavior dual-flow identification method, system, equipment and medium, comprising: the one-dimensional image block vector of the video to be identified is obtained by processing the video to be identified, and is divided, and visual flow data set and action flow data set are obtained;Position coding is carried out to two-dimensional image frame and is embedded into the one-dimensional image block sequence in visual flow data set, to obtain the visual flow data set containing position coding information;The visual flow data set containing position coding information is extracted using the first space-time separation Transformer network model unit, to obtain visual flow feature extraction result;Action flow data set is extracted using the second space-time separation Transformer network model unit, to obtain action flow feature extraction result;Visual flow feature extraction result and action flow feature extraction result are integrated, to obtain the dual-flow identification result of video;The application has higher identification efficiency, meets the identification requirement of high-resolution video.

Description

Dual-stream recognition methods, systems, devices, and media for video behavior Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a dual-stream recognition method, system, device, and medium for video behavior. Background Technology

[0002] Video action recognition is an important branch of computer vision, used to identify human actions in videos. Currently, most existing video action recognition methods employ convolutional neural network (CNN) or Transformer network models. CNN-based methods, however, suffer from limitations in processing quantity, convolution scale, and overall feature set, making it difficult to consider global features and unable to model dependencies outside the receptive field, resulting in longer computation times. While Transformer-based methods offer faster training and derivation compared to CNN-based methods, most existing Transformer-based methods use a single-channel RGB recognition structure, considering only visual information from the video, leading to significant deviations in action recognition results. Summary of the Invention

[0003] To address the technical problems existing in the prior art, this invention provides a dual-stream recognition method, system, device, and medium for video behavior, in order to solve the problem that most existing recognition methods based on the Transformer network model adopt a single-channel RGB recognition structure, which only considers the visual information of the video, resulting in a large deviation in the behavior recognition results.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] This invention provides a two-stream recognition method for video behavior, comprising:

[0006] Frame-level image preprocessing is performed on the video to be identified to obtain a set of two-dimensional frame images of the video to be identified; wherein, the set of two-dimensional frame images of the video to be identified includes several two-dimensional image frames;

[0007] The two-dimensional image frames are divided into blocks to obtain a one-dimensional image block vector of the video to be identified; wherein the one-dimensional image block vector of the video to be identified includes a sequence of one-dimensional image blocks.

[0008] Several one-dimensional image block sequences are divided to obtain a visual flow dataset and an action flow dataset;

[0009] The two-dimensional image frame is position-encoded to obtain an image frame position-encoded information sequence; the image frame position-encoded information sequence is embedded into a one-dimensional image patch sequence in a visual stream dataset to obtain a visual stream dataset containing position-encoded information.

[0010] The first spatiotemporal separation Transformer network model unit is used to extract features from the visual flow dataset containing positional encoding information to obtain visual flow feature extraction results; the second spatiotemporal separation Transformer network model unit is used to extract features from the action flow dataset to obtain action flow feature extraction results.

[0011] The visual stream feature extraction results and the motion stream feature extraction results are integrated to obtain the dual-stream recognition results of the video to be recognized.

[0012] Furthermore, the process of dividing the two-dimensional image frame into blocks to obtain a one-dimensional image block vector of the video to be identified is as follows:

[0013] Using the image segmentation formula, each two-dimensional image frame is segmented to obtain the corresponding one-dimensional image block sequence.

[0014] Based on the sequence of one-dimensional image blocks corresponding to all two-dimensional image frames, the one-dimensional image block vector of the video to be identified is obtained;

[0015] The image segmentation formula is as follows:

[0016] N = HW / P 2

[0017] Where N is the number of blocks in the one-dimensional image block sequence after the two-dimensional image frame is processed; H is the spatial height of the two-dimensional image frame; W is the spatial width of the two-dimensional image frame; and P is the resolution of the one-dimensional image block sequence.

[0018] Furthermore, the process of performing position encoding on the two-dimensional image frame to obtain the image frame position encoding information sequence is as follows:

[0019] Position learning is performed using the convolution of the two-dimensional image frame. Then, conditional position coding results are adaptively generated through deconvolution. The conditional position coding results are reshaped to obtain a one-dimensional position coding information sequence, which is the image frame position information sequence.

[0020] Furthermore, the process of extracting features from the visual flow dataset containing location encoding information using the first spatiotemporal separation Transformer network model unit to obtain the visual flow feature extraction results is as follows:

[0021] The visual stream dataset containing location encoding information is used as the input to the first spatiotemporal separation Transformer network model unit;

[0022] Temporal self-attention computation is performed on the visual stream dataset containing location encoding information to obtain the temporal attention computation result of the visual stream;

[0023] Spatial self-attention calculation is performed on the temporal attention calculation results of the visual stream to obtain the video stream feature extraction results.

[0024] Furthermore, the process of using the second spatiotemporal separation Transformer network model unit to extract features from the action flow dataset and obtain the action flow feature extraction results is as follows:

[0025] The action flow dataset is used as the input to the second spatiotemporal separation Transformer network model unit;

[0026] Perform temporal self-attention calculation on the action flow dataset to obtain the temporal attention calculation result of the action flow;

[0027] Spatial self-attention calculation is performed on the temporal attention calculation results of the action flow to obtain the action flow feature extraction results.

[0028] Furthermore, the process of integrating the visual flow feature extraction results and the action flow feature extraction results to obtain the dual-stream recognition results of the video to be recognized is as follows:

[0029] Using a convolutional neural network, pooling is performed on the visual flow feature extraction results and the action flow feature extraction results to obtain the dual-stream recognition results of the video to be recognized.

[0030] Furthermore, the pooling operation on the visual flow feature extraction results and the action flow feature extraction results is performed according to the following formula:

[0031] M fin =Merge(Wx,x)=ReLU(Dropout(Pooling(Conv(Wx,x))))

[0032] Among them, M fin The result is the dual-stream recognition result of the video to be recognized; Merge(*) is a function consisting of four cascaded operations; Wx is the result of visual stream feature extraction; x is the result of action stream feature extraction; ReLU(*) is the linear rectified function; Dropout(*) is the dropout function; Pooling(*) is the max time pooling operation; Conv(*) is the convolution operation.

[0033] The present invention also provides a dual-stream video behavior recognition system, comprising:

[0034] The data processing layer module is used to perform frame-level image preprocessing on the video to be recognized, obtaining a set of two-dimensional frame images of the video to be recognized; wherein the set of two-dimensional frame images of the video to be recognized includes several two-dimensional image frames; each of the two-dimensional image frames is divided into blocks to obtain a one-dimensional image block vector of the video to be recognized; wherein the one-dimensional image block vector of the video to be recognized includes several one-dimensional image block sequences; the several one-dimensional image block sequences are divided to obtain a visual stream dataset and an action stream dataset; the two-dimensional image frames are position-encoded to obtain an image frame position-encoded information sequence; the image frame position-encoded information sequence is embedded into the one-dimensional image block sequence in the visual stream dataset to obtain a visual stream dataset containing position-encoded information;

[0035] The dual-stream recognition network module is used to extract features from the visual stream dataset containing positional encoding information using a first spatiotemporal separation Transformer network model unit to obtain visual stream feature extraction results; and to extract features from the action stream dataset using a second spatiotemporal separation Transformer network model unit to obtain action stream feature extraction results.

[0036] The data integration layer module is used to integrate the visual stream feature extraction results and the motion stream feature extraction results to obtain the dual-stream recognition results of the video to be recognized.

[0037] The present invention also provides a dual-stream video behavior recognition device, comprising:

[0038] Memory, used to store computer programs;

[0039] A processor is configured to implement the steps of the dual-stream recognition method for video behavior when executing the computer program.

[0040] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the dual-stream video behavior recognition method.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] This invention provides a dual-stream video behavior recognition method and system, which divides the video to be recognized into a visual stream dataset containing location-encoded information and an action stream dataset. The visual stream dataset containing location-encoded information is a dataset containing RGB color information and embedded with location-encoded information; the action stream dataset is a dataset containing action information. Two parallel spatiotemporally separated Transformer network model units are used to extract features from the visual stream dataset and the action stream dataset containing location-encoded information, respectively. This approach considers both the visual information and the rich action information within the video to be recognized, effectively improving the accuracy of behavior recognition. The recognition model has a short training time and high recognition efficiency, meeting the requirements for recognizing high-resolution videos. Attached Figure Description

[0043] Figure 1 is a flowchart of the dual-stream video behavior recognition method of the present invention. Detailed Implementation

[0044] To make the technical problems solved by the present invention, the technical solutions, and the beneficial effects clearer, the following specific embodiments provide a further detailed description of the present invention. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of the invention.

[0045] As shown in Figure 1, the present invention provides a dual-stream recognition method for video behavior, comprising the following steps:

[0046] Step 1: Input the video to be recognized into the data processing layer, preprocess the input video to obtain a visual stream dataset and an action stream dataset containing location encoding information.

[0047] The preprocessing process of the data processing layer is as follows:

[0048] Step 11: Perform frame-level image preprocessing on the video to be identified to obtain a set of two-dimensional frame images of the video to be identified; wherein the set of two-dimensional frame images to be identified includes several two-dimensional image frames.

[0049] Step 12: Divide all two-dimensional image frames into blocks to obtain a one-dimensional image block vector of the video to be identified; wherein the one-dimensional image block vector of the video to be identified includes a sequence of one-dimensional image blocks.

[0050] The block processing procedure is as follows:

[0051] Using the image segmentation formula, each two-dimensional image frame is segmented to obtain the corresponding one-dimensional image block sequence.

[0052] Based on the sequence of one-dimensional image blocks corresponding to all two-dimensional image frames, the one-dimensional image block vector of the video to be identified is obtained;

[0053] The image segmentation formula is as follows:

[0054] N = HW / P 2

[0055] Where N is the number of blocks in the one-dimensional image block sequence after the two-dimensional image frame is processed; H is the spatial height of the two-dimensional image frame; W is the spatial width of the two-dimensional image frame; and P is the resolution of the one-dimensional image block sequence.

[0056] In this invention, combining existing frame-level image preprocessing methods in the field of image processing, a three-dimensional video is divided into a set of two-dimensional frame images. Then, the two-dimensional image frames are broken down to obtain a one-dimensional image block sequence. Specifically, the process of breaking down the two-dimensional image frames to obtain the one-dimensional image block sequence involves dividing each two-dimensional image frame into N blocks of size P one-dimensional image blocks. After block partitioning and flattening, the entire video is represented by a vector i. (p,t) ∈3P 2 , where p represents the image block position, between 1 and N; t represents the frame position of the image in the video, between 1 and F.

[0057] Step 13: Divide several one-dimensional image block sequences to obtain a visual flow dataset and an action flow dataset.

[0058] Step 14: Perform position encoding on the two-dimensional image frame to obtain an image frame position encoding information sequence, reflecting the specific position of the video image block in the original video; embed the image frame position encoding information sequence into the one-dimensional image block sequence in the visual stream dataset to obtain a visual stream dataset containing position encoding information; specifically, by adding the image frame position encoding information sequence to the one-dimensional image block sequence in the visual stream dataset, a visual stream dataset containing position encoding information is obtained; wherein, the visual stream dataset containing position encoding information is used as the input of the encoder in the first spatiotemporal separation Transformer network model unit.

[0059] The location encoding process is as follows:

[0060] Position learning is performed using the convolution of the two-dimensional image frame. The height and width of the input image are read from the inside. Then, conditional position coding results are adaptively generated through deconvolution. The conditional position coding results are reshaped to obtain a one-dimensional position coding information sequence, which is the image frame position information sequence.

[0061] Step 2: Input the visual stream dataset containing location encoding information and the action stream dataset into the dual-stream recognition network model; wherein, the first spatiotemporal separation Transformer network model unit is used to extract features from the visual stream dataset containing location encoding information to obtain visual stream feature extraction results; the second spatiotemporal separation Transformer network model unit is used to extract features from the action stream dataset to obtain action stream feature extraction results.

[0062] The process of obtaining the visual flow feature extraction results is as follows:

[0063] The visual stream dataset containing location encoding information is used as the input to the first spatiotemporal separation Transformer network model unit;

[0064] Temporal self-attention computation is performed on the visual stream dataset containing location encoding information to obtain the temporal attention computation result of the visual stream;

[0065] Spatial self-attention calculation is performed on the temporal attention calculation results of the visual stream to obtain the video stream feature extraction results.

[0066] The process of obtaining the action flow feature extraction results is as follows:

[0067] The action flow dataset is used as the input to the second spatiotemporal separation Transformer network model unit;

[0068] Perform temporal self-attention calculation on the action flow dataset to obtain the temporal attention calculation result of the action flow;

[0069] Spatial self-attention calculation is performed on the temporal attention calculation results of the action flow to obtain the action flow feature extraction results.

[0070] In this invention, the first spatiotemporal separation Transformer network model unit and the second spatiotemporal separation Transformer network model unit are spatiotemporal separation Transformer network models with the same structure and in parallel.

[0071] The feature extraction process of the spatiotemporal separation Transformer network model is as follows:

[0072] According to the spatiotemporal self-attention calculation formula, self-attention calculation is performed on the visual stream dataset containing positional encoding information and the action stream dataset respectively to obtain the corresponding feature extraction results;

[0073] The spatiotemporal self-attention calculation formula is as follows:

[0074] A s-t =Attention s (Attention t (I input ))

[0075] Among them, A s-t For visual flow or action flow feature extraction results; Attention s (*) represents spatial self-attention computation operations; Attention t (*) represents a temporal self-attention computation operation; I input The input to the spatiotemporal separation Transformer network model is either the visual stream dataset containing location encoding information or the action stream dataset.

[0076] The following details the process of performing self-attention calculation on the visual stream dataset containing location encoding information and the action stream dataset:

[0077] First, temporal self-attention calculation is performed on the input data to obtain the temporal self-attention result of the input data; that is, temporal attention calculation is performed on the same image block sequence in the video to be recognized, which is in the same position but in different time dimensions; wherein, the input data is the visual stream dataset containing position encoding information or the action stream dataset.

[0078] The temporal self-attention calculation process is as follows:

[0079]

[0080] Among them, A t The result of the temporal attention calculation for the input data; Attention(*) represents the attention calculation operation; q t The query value vector is the result of processing the input data using a linearization function. The input data is processed by a linearization function to form a key-value vector; v t This is the value vector of the input data after it has been processed by a linearization function.

[0081] Secondly, spatial self-attention calculation is performed on the temporal self-attention results of the input data to obtain video stream feature extraction results or action stream feature extraction results; that is, self-attention calculation is performed on image block sequences at different locations but in the same temporal dimension.

[0082] The spatial self-attention calculation process is as follows:

[0083]

[0084] Where, q s The query value vector is the result of the self-attention results over time on the input data, processed by a linearization function; k s The key-value vector is the result of the temporal self-attention of the input data, processed by a linearization function; v s The input data is a vector of self-attention results over time, processed by a linearization function.

[0085] Step 3: Input the visual flow feature extraction results and the action flow feature extraction results into the data integration layer, integrate the visual flow feature extraction results and the action flow feature extraction results, and output the dual-stream recognition result of the video to be recognized; wherein, the integration process specifically involves: using a convolutional neural network to perform pooling operations on the visual flow feature extraction results and the action flow feature extraction results to obtain the dual-stream recognition result of the video to be recognized; wherein, the two features are kept at a uniform scale to adapt to subsequent classification processing, and the two sets of feature extraction results are integrated into a fixed-length vector to capture the optimal features in the feature extraction process.

[0086] In this invention, a convolutional neural network is used for pooling operations, and then the two feature extraction results are concatenated. Specifically, a CNN neural network model with max pooling is used to aggregate the two feature extraction results. The pooling operation on the visual flow feature extraction results and the action flow feature extraction results is performed according to the following formula:

[0087] M fin =Merge(Wx,x)=ReLU(Dropout(Pooling(Conv(Wx,x))))

[0088] Among them, M fin The result is the dual-stream recognition result of the video to be recognized; Merge(*) is a function consisting of four cascaded operations; Wx is the result of visual stream feature extraction; x is the result of action stream feature extraction; ReLU(*) is the linear rectified function; Dropout(*) is the dropout function; Pooling(*) is the max time pooling operation; Conv(*) is the convolution operation.

[0089] In this invention, the input video frame is divided into a visual stream containing RGB color information and a motion stream containing motion information; specifically, the video to be identified I inputThe spatial height H, spatial width W, and temporal dimension F in H×W×3×F are extracted, transforming F×N+1 one-dimensional vectors into a two-dimensional sequence (F+1)×(N+1). Classification representations are then set for the spatial and temporal dimensions to optimize the algorithm's time complexity and address the large-dimensionality issue when expanding from images to videos. Subsequently, two parallel spatiotemporal separation Transformer network model units are used to extract features from visual stream and action stream datasets containing positional encoding information, respectively. First, the temporal self-attention result is calculated, and then this result is used as input for the spatial self-attention calculation, thus completing the entire spatiotemporal separation self-attention calculation. This reduces the algorithm's time complexity from O((FWH)). 2 ) decreased to O(F 2 +(WH) 2 Meanwhile, by introducing an adaptive positional encoding process, positional encoding embedding can be performed covertly, resulting in better optimization efficiency.

[0090] The dual-stream video behavior recognition method of this invention first inputs the video to be recognized into the data processing layer. In the data preprocessing layer, the input video to be processed undergoes frame-level image preprocessing. Specifically, the 3D video is first divided into a set of 2D frame images, and then the 2D image frames are broken down into a sequence of 1D image blocks. Then, the processed 1D image block sequence is divided into two subsets: a visual stream dataset and an action stream dataset, forming a dual-stream input. Furthermore, after converting the video into a 1D image block sequence, position encoding needs to be added to the visual stream dataset to reflect the specific position of the video image blocks in the original video. Specifically, convolution is first used to learn the position of the 2D image frames, and then conditional position encoding is adaptively generated through deconvolution and reshaped into 1D to obtain the image frame position encoding information sequence. The image frame position encoding information sequence is then embedded into the visual stream dataset. The visual stream dataset, consisting of a sequence of one-dimensional image patches, is used to obtain a visual stream dataset containing positional encoding information. Positional encoding is performed so that the sequence of one-dimensional image patches can be recombined into a two-dimensional spatial image. Next, the visual stream dataset containing positional encoding information and the action stream dataset are input into a parallel spatiotemporal separation Transformer network model unit for discriminative feature extraction. Finally, the visual stream feature extraction results and the action stream feature extraction results are input into a data integration layer for integration, resulting in a dual-stream recognition result for the video to be recognized. Specifically, the feature elements of the video stream and action stream are summarized and aggregated into a fixed-length vector using a convolutional neural network. The visual vector and action vector are then concatenated. Using the integrated feature vector, the probability of different action categories is calculated, ultimately yielding the classification result for the behavior recognition task.

[0091] In this invention, video information is split into visual and action streams. Then, spatiotemporal separation self-attention computation is used as the main processing method to extract video features, and finally, the features of the two streams are aggregated. This invention adopts an adaptive coding method, which not only achieves better recognition accuracy, but also broadens the recognition scale of video. It provides methodological support for practical video recognition tasks, high-resolution video processing, and other work, and broadens the direction for future research.

[0092] This invention also provides a dual-stream video behavior recognition system, including a data processing layer module, a dual-stream recognition network module, and a data integration layer module. The data processing layer module performs frame-level image preprocessing on the video to be recognized, obtaining a set of two-dimensional frame images of the video to be recognized; wherein the set of two-dimensional frame images of the video to be recognized includes several two-dimensional image frames; each of the two-dimensional image frames is divided into blocks to obtain a one-dimensional image block vector of the video to be recognized; wherein the one-dimensional image block vector of the video to be recognized includes several one-dimensional image block sequences; the several one-dimensional image block sequences are further divided to obtain a visual stream dataset and an action stream dataset; and the two-dimensional image frames are position-encoded to obtain image frame position codes. The image frame position coding information sequence is embedded into a one-dimensional image patch sequence in the visual stream dataset to obtain a visual stream dataset containing position coding information; a dual-stream recognition network module is used to extract features from the visual stream dataset containing position coding information using a first spatiotemporal separation Transformer network model unit to obtain visual stream feature extraction results; a second spatiotemporal separation Transformer network model unit is used to extract features from the motion stream dataset to obtain motion stream feature extraction results; a data integration layer module is used to integrate the visual stream feature extraction results and the motion stream feature extraction results to obtain the dual-stream recognition results of the video to be recognized.

[0093] The present invention also provides a dual-stream video behavior recognition device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the dual-stream video behavior recognition method.

[0094] When the processor executes the computer program, it implements the steps of the dual-stream recognition method for video behavior described above, for example: performing frame-level image preprocessing on the video to be recognized to obtain a set of two-dimensional frame images of the video to be recognized; wherein the set of two-dimensional frame images of the video to be recognized includes several two-dimensional image frames; performing block processing on all two-dimensional image frames to obtain a one-dimensional image block vector of the video to be recognized; wherein the one-dimensional image block vector of the video to be recognized includes several one-dimensional image block sequences; dividing the several one-dimensional image block sequences to obtain a visual stream dataset and an action stream dataset; and performing position encoding on the two-dimensional image frames to obtain image frame position encoding. The image frame position coding information sequence is embedded into a one-dimensional image patch sequence in the visual stream dataset to obtain a visual stream dataset containing position coding information. A first spatiotemporal separation Transformer network model unit is used to extract features from the visual stream dataset containing position coding information to obtain visual stream feature extraction results. A second spatiotemporal separation Transformer network model unit is used to extract features from the motion stream dataset to obtain motion stream feature extraction results. The visual stream feature extraction results and the motion stream feature extraction results are integrated to obtain the dual-stream recognition result of the video to be recognized.

[0095] Alternatively, when the processor executes the computer program, it implements the functions of each module in the above system. For example, a data processing layer module is used to perform frame-level image preprocessing on the video to be identified to obtain a set of two-dimensional frame images of the video to be identified; wherein the set of two-dimensional frame images of the video to be identified includes several two-dimensional image frames; all two-dimensional image frames are divided into blocks to obtain a one-dimensional image block vector of the video to be identified; wherein the one-dimensional image block vector of the video to be identified includes several one-dimensional image block sequences; the several one-dimensional image block sequences are divided to obtain a visual flow dataset and an action flow dataset; and the two-dimensional image frames are position-encoded to obtain an image frame position-encoded information sequence. The image frame position encoding information sequence is embedded into a one-dimensional image patch sequence in the visual stream dataset to obtain a visual stream dataset containing position encoding information. A dual-stream recognition network module is used to extract features from the visual stream dataset containing position encoding information using a first spatiotemporal separation Transformer network model unit to obtain visual stream feature extraction results. A second spatiotemporal separation Transformer network model unit is used to extract features from the motion stream dataset to obtain motion stream feature extraction results. A data integration layer module is used to integrate the visual stream feature extraction results and the motion stream feature extraction results to obtain the dual-stream recognition results of the video to be recognized.

[0096] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing preset functions, the instruction segments describing the execution process of the computer program in the dual-stream video behavior recognition device.

[0097] The dual-stream video behavior recognition device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The dual-stream video behavior recognition device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above are examples of dual-stream video behavior recognition devices and do not constitute a limitation on the device. It may include more components than described above, or combine certain components, or use different components. For example, the dual-stream video behavior recognition device may also include input / output devices, network access devices, buses, etc.

[0098] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or any conventional processor. The processor is the control center of the dual-stream video behavior recognition device, connecting various parts of the device via various interfaces and lines.

[0099] The memory can be used to store the computer program and / or modules. The processor implements various functions of the dual-stream video behavior recognition device by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory.

[0100] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0101] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the dual-stream video behavior recognition method described above.

[0102] If the modules / units integrated in the dual-stream video behavior recognition system are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0103] Based on this understanding, the present invention can implement all or part of the processes in the above-described dual-stream video behavior recognition method, or it can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above-described dual-stream video behavior recognition method. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or a preset intermediate form, etc.

[0104] The computer-readable storage medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0105] It should be noted that the content contained in the computer-readable storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0106] The dual-stream video behavior recognition method and system described in this invention divides the input video frame into a visual stream containing RGB color information and an action stream containing action information. Then, it processes the video stream using two parallel spatiotemporally separated Transformer network model units to extract spatial and action features. Adaptive positional encoding embeds positional encoding information into the visual stream, significantly optimizing time complexity. This not only enables faster processing of the same video but also allows for the handling of more video content and the exploration of longer video sequences, meeting the requirements for high-resolution video behavior recognition.

[0107] This invention extends the spatiotemporal separation Transformer network model to the field of behavior recognition, achieving the effects of reducing the training time of the recognition model and improving the accuracy of behavior recognition. Compared with traditional convolutional neural networks, it effectively avoids the limitations of processing quantity and convolutional kernel size, achieving a balance of global features and supporting the modeling of long sequences. In this invention, the spatiotemporal separation Transformer network model is used as the feature extraction skeleton for behavior recognition, a dual-stream recognition network module is designed, and adaptive position encoding suitable for video classification is adopted based on the position encoding characteristics of Transformer. The recognition model has a short training time, high recognition efficiency, and meets the recognition requirements for high-resolution videos.

[0108] The above embodiments are merely one of the implementation methods for achieving the technical solution of the present invention. The scope of protection claimed by the present invention is not limited to this embodiment, but also includes any variations, substitutions and other implementation methods that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention.

Claims

1. A dual-stream recognition method for video behavior, characterized in that, include: Frame-level image preprocessing is performed on the video to be recognized to obtain a set of two-dimensional frame images of the video to be recognized; wherein the set of two-dimensional frame images of the video to be recognized includes several two-dimensional image frames; each of the two-dimensional image frames is divided into blocks to obtain a one-dimensional image block vector of the video to be recognized; wherein the one-dimensional image block vector of the video to be recognized includes several one-dimensional image block sequences; the several one-dimensional image block sequences are divided to obtain a visual flow dataset and an action flow dataset; position encoding is performed on the two-dimensional image frames to obtain an image frame position encoding information sequence; the image frame position encoding information sequence is embedded into the one-dimensional image block sequence in the visual flow dataset to obtain a visual flow dataset containing position encoding information; the first spatiotemporal separation Transformer network model unit is used to extract features from the visual flow dataset containing position encoding information to obtain a visual... The motion stream feature extraction results are obtained by using the second spatiotemporal separation Transformer network model unit to extract features from the motion stream dataset. The visual stream feature extraction results and the motion stream feature extraction results are then integrated to obtain the dual-stream recognition results of the video to be recognized. The first and second spatiotemporal separation Transformer network model units are structurally identical and parallel spatiotemporal separation Transformer network models. Specifically, the feature extraction process of the spatiotemporal separation Transformer network model is as follows: Self-attention calculation is performed on the visual stream dataset containing positional encoding information and the motion stream dataset according to the spatiotemporal self-attention calculation formula to obtain the corresponding feature extraction results. The spatiotemporal self-attention calculation formula is: in, The result is the feature extraction result of visual flow or action flow; For spatial self-attention computational operations; For time-dependent self-attention computational operations; The input to the spatiotemporal separation Transformer network model is either the visual stream dataset containing location encoding information or the action stream dataset.

2. The dual-stream video behavior recognition method according to claim 1, characterized in that, The process of dividing the two-dimensional image frames into blocks to obtain a one-dimensional image block vector of the video to be identified is as follows: Each two-dimensional image frame is divided into blocks using an image block formula to obtain a corresponding one-dimensional image block sequence; based on the one-dimensional image block sequences corresponding to all two-dimensional image frames, the one-dimensional image block vector of the video to be identified is obtained; wherein, the image block formula is: in, The number of blocks in the one-dimensional image block sequence after the two-dimensional image frame is divided into blocks; The spatial height of a two-dimensional image frame; The spatial width of a two-dimensional image frame; The resolution of a one-dimensional image patch sequence.

3. The dual-stream video behavior recognition method according to claim 1, characterized in that, The process of performing position encoding on the two-dimensional image frame to obtain the image frame position encoding information sequence is as follows: position learning is performed by convolving the two-dimensional image frame, and then conditional position encoding results are adaptively generated by deconvolution. The conditional position encoding results are reshaped to obtain a one-dimensional position encoding information sequence, which is the image frame position information sequence.

4. The dual-stream video behavior recognition method according to claim 1, characterized in that, The process of extracting features from the visual flow dataset containing positional encoding information using the first spatiotemporal separation Transformer network model unit to obtain the visual flow feature extraction result is as follows: the visual flow dataset containing positional encoding information is used as the input of the first spatiotemporal separation Transformer network model unit; temporal self-attention calculation is performed on the visual flow dataset containing positional encoding information to obtain the temporal attention calculation result of the visual flow. Spatial self-attention calculation is performed on the temporal attention calculation results of the visual stream to obtain the video stream feature extraction results.

5. The dual-stream video behavior recognition method according to claim 1, characterized in that, The process of using the second spatiotemporal separation Transformer network model unit to extract features from the action flow dataset and obtain the action flow feature extraction results is as follows: the action flow dataset is used as the input of the second spatiotemporal separation Transformer network model unit; temporal self-attention calculation is performed on the action flow dataset to obtain the temporal attention calculation result of the action flow; spatial self-attention calculation is performed on the temporal attention calculation result of the action flow to obtain the action flow feature extraction result.

6. The dual-stream video behavior recognition method according to claim 1, characterized in that, The process of integrating the visual flow feature extraction results and the action flow feature extraction results to obtain the dual-stream recognition results of the video to be recognized is as follows: using a convolutional neural network, the visual flow feature extraction results and the action flow feature extraction results are pooled to obtain the dual-stream recognition results of the video to be recognized.

7. The dual-stream video behavior recognition method according to claim 6, characterized in that, The pooling operation on the visual flow feature extraction results and the action flow feature extraction results is performed according to the following formula: in, The results of dual-stream recognition of the video to be recognized; It is a function consisting of four cascaded operations; The results of visual flow feature extraction; The result is the action flow feature extraction result; It is a linear rectified function; For the dropout function; For maximum time pooling operation; This is a convolution operation.

8. A dual-stream video behavior recognition system, characterized in that, A dual-stream recognition method for video behavior as described in any one of claims 1-7, comprising: a data processing layer module, configured to perform frame-level image preprocessing on the video to be recognized to obtain a set of two-dimensional frame images of the video to be recognized; wherein the set of two-dimensional frame images of the video to be recognized includes a plurality of two-dimensional image frames; to perform block processing on all two-dimensional image frames respectively to obtain a one-dimensional image block vector of the video to be recognized; wherein the one-dimensional image block vector of the video to be recognized includes a plurality of one-dimensional image block sequences; to divide the plurality of one-dimensional image block sequences to obtain a visual stream dataset and an action stream dataset; to perform position encoding on the two-dimensional image frames to obtain an image frame position encoding information sequence; and to... The image frame position encoding information sequence is embedded into a one-dimensional image patch sequence in the visual stream dataset to obtain a visual stream dataset containing position encoding information; a dual-stream recognition network module is used to extract features from the visual stream dataset containing position encoding information using a first spatiotemporal separation Transformer network model unit to obtain visual stream feature extraction results; a second spatiotemporal separation Transformer network model unit is used to extract features from the motion stream dataset to obtain motion stream feature extraction results; a data integration layer module is used to integrate the visual stream feature extraction results and the motion stream feature extraction results to obtain the dual-stream recognition results of the video to be recognized.

9. A dual-stream video behavior recognition device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the dual-stream recognition method for video behavior as described in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the dual-stream recognition method for video behavior as described in any one of claims 1-7.

Images