A staged attention-temporal alignment approach for few-shot action recognition
By using a staged attention mechanism for timing alignment in few-sample action recognition, the problem of timing alignment in the prior art that violates timing before partial order relationships is solved, improving the accuracy of action recognition and resisting non-action interference.
Patent Information
- Application Number
- CN202210085490.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-25
- Publication Date
- 2025-04-08
- Estimated Expiration
- 2042-01-25
AI Technical Summary
The existing few-sample action recognition methods are likely to violate the timing of the video progress stage before the partial order relationship in the timing alignment process, resulting in stage-level timing mismatch, affecting the recognition accuracy.
The staged attention mechanism is used for timing alignment, and the action query video and the action sample video are divided into multiple stages. Each stage uses the attention module for timing alignment to ensure that the characteristics of each video clip are only aligned with the features of adjacent stages, thereby ensuring the partial order relationship of timing precedes the order.
Through the timing alignment of the phased attention mechanism, timing mismatch at the video stage level is avoided, the accuracy of small sample action recognition is improved, and the interference of non-action clip pairs is effectively resisted.
Smart Images
Figure CN114519893B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a few-sample action recognition method with phased attention to temporal alignment. Background Art
[0002] With the exponential growth of the number of video cameras in recent years, the interaction between people and smart devices such as mobile phones has become increasingly close and diverse. As a basic task in the field of video understanding, video action recognition has important applications in video surveillance, human-computer interaction, video retrieval, etc. Deep learning methods can automatically mine deep semantic information in videos through network fitting. For example, two-stream convolutional networks and 3D convolutional network methods have been used for action recognition and have made significant progress. These methods require a large amount of labeled data as support. However, in the real world, due to the difficulty of shooting, the cost of collecting annotations, and privacy ethics, some categories of data are often difficult to obtain in large quantities. On the other hand, humans can summarize and generalize knowledge from very few samples when learning. The ability to learn and generalize from very few samples is an obvious boundary that distinguishes artificial intelligence from human intelligence. Based on this, few-sample action modeling and recognition has more important application value.
[0003] Existing few-sample action recognition methods still face many difficulties and challenges. First, due to the small number of samples with supervised information, deep learning models are prone to overfitting during training; second, video actions contain rich temporal information. The same action category has different performance processes in different video scenes, which makes the duration of each stage of the same action different. Irrelevant information will appear at the beginning and end of the action, and blurred action boundaries will greatly affect the recognition accuracy; third, in fine-grained video action datasets, there will be a problem of small differences between action categories, that is, there are only slight differences between actions of different categories. It is extremely challenging for existing models to accurately distinguish these actions.
[0004] Temporal alignment is an effective means and an important influencing factor for achieving few-shot action recognition. Its purpose is to accurately match video clip pairs belonging to the same action stage. At present, there are methods to perform temporal alignment processing at the video clip or frame level. The similarity of the aligned video clip pairs is fused to determine the similarity between the video to be tested and the sample video, and the video action to be tested is identified. The accuracy of video recognition needs to be improved. There is a strict order of occurrence of each progress stage in the video action, that is, there is a partial order relationship of temporal precedence. More accurate video clip alignment should also ensure the partial order relationship of temporal precedence of the progress stage. However, the current video action temporal alignment method does not reasonably consider the temporal nature between clips, making the range of video clip temporal alignment too wide, resulting in the possibility that the temporal alignment of the video progress stage violates the temporal precedence relationship of the video progress stage, and fails to reasonably utilize the temporal characteristics of the video action clip, thus affecting the further improvement of the performance of few-shot action recognition. Summary of the invention
[0005] In view of the shortcomings in the prior art, the present invention provides a few-sample action recognition method with attention to timing alignment in stages, so as to solve the problem that in the timing alignment process, the timing alignment of each video progress stage may violate the timing precedence of the partial order relationship of the video progress stage, thereby causing timing mismatch at the stage level and affecting the accuracy of few-sample action recognition.
[0006] The present invention achieves the above technical objectives through the following technical means.
[0007] A staged attention-temporal alignment method for few-shot action recognition is proposed, specifically:
[0008] The action query video is evenly divided into m stages and the action sample video is evenly divided into m+1 stages. The attention module is used to perform temporal alignment for each stage, so that the features of each video clip in the i-th stage of the action query video are only aligned with the features of each video clip in the i-th and i+1-th stages of the action sample video.
[0009] The aligned action sample video clip features and action query video clip features are paired in order and the similarity is calculated. Only the top n similarity clip pairs are retained as the action video features for metric learning.
[0010] According to a further technical solution, the process of acquiring the video clip features is as follows: dividing the action sample video and the action query video into an unfixed number of video clips, each video clip containing 16 video frames, and using a video clip feature extractor to extract the video clip features.
[0011] In a further technical solution, the video clip feature extractor selects a pre-trained C3D model.
[0012] Further technical solution, the alignment attention weight matrix of the action query video features and the action sample video related segment features in the i-th stage is expressed as:
[0013] A i =softmax((S i W i +b)Q i T ), i∈(1,m)
[0014] in: Is The matrix composed of the action sample video clip features between Is The matrix composed of action query video clip features between them, aligning the attention weights W i is the attention module, b is the bias term, N is the number of action sample video clips, M is the number of action query video clips, d is the length of each video clip feature, and R is the real number field.
[0015] In a further technical solution, the attention weight matrix is multiplied by the action sample video feature to obtain the alignment result H of the action sample video feature in the i-th stage i : Splice the alignment results m times to get the action sample video feature H = (H1,...,H m ), where i∈(1,m).
[0016] A further technical solution is that the similarity top n segments are based on Determine, where M is the number of action query video clips, sim i is the similarity between the features of action query video segment i and the features of the corresponding action sample video segment, SIM refers to the set of similarities between the features of the action query video related segments and the features of the corresponding action sample video segments, and top(SIM,n) refers to the set of the first n largest values in the similarity set between the features of the action query video related segments and the features of the corresponding action sample video segments.
[0017] In a further technical solution, the similarity is a cosine distance between the corresponding action sample video segment features and the action query video segment features.
[0018] A further technical solution also includes inputting the similarity features of the action video clips into the unidirectional GRU unit and the fully connected layer in sequence, and fusing the learning to obtain the video-level correlation scores of the action query video and the action sample video. After the correlation scores of the action query video and different action sample videos are subjected to softmax, a probability distribution of the category to which the action query video belongs is formed. If the probability that the action query video belongs to a certain category of action sample videos is the largest, the action query video is determined to be an action video of that category.
[0019] The beneficial effects of the present invention are as follows: in the timing alignment stage, the present invention uses a staged attention mechanism to enable the attention modules of different stages to pay attention to the unique features of a specific stage of the video, thereby overcoming the problem that the scope of the existing single attention module timing alignment is too wide, ensuring that the aligned video clip pairs have a stage timing before the partial order relationship, avoiding the timing mismatch at the video stage level, thereby more reasonably and accurately obtaining and utilizing the timing information of the action in the video; at the same time, a Top n confidence selection method based on the similarity of the clips is used to eliminate the clips with too low similarity to perform metric learning action classification, thereby being able to resist the interference of non-action clips on the video action classification, and learn more effective action timing clip features to improve the accuracy of few-sample action recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is a flow chart of the method for few-sample action recognition with phased attention to temporal alignment according to the present invention;
[0021] Figure 2 A specific flow chart of the timing alignment of the staged attention mechanism of the present invention;
[0022] Figure 3 This is a specific flow chart of the metric learning action classification based on the fragment pair similarity Top n confidence selection described in the present invention. DETAILED DESCRIPTION
[0023] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.
[0024] The specific implementation of the present invention is divided into three stages: feature extraction, time alignment of staged attention mechanism, and Top n confidence selection metric learning. Figure 1 shown.
[0025] (1) Feature extraction: The video is divided into an unfixed number of video segments, each with 16 frames. The pre-trained C3D model is used as a video segment feature extractor to extract features for each video segment.
[0026] (2) Temporal alignment of staged attention mechanism: The temporal alignment of video clips is achieved through a staged multi-attention module. The action query video is divided into m stages on average, and the action sample video is divided into m+1 stages on average. Temporal alignment is performed on each stage separately, so that the features of each video clip in the i-th stage of the action query video are only aligned with the features of each video clip in the i-th stage and the i+1-th stage of the action sample video, so as to ensure that the final aligned video clip pairs have a temporal precedence relationship at the stage level.
[0027] (3) Top n confidence selection metric learning: The feature similarity between the corresponding video clip feature pairs is calculated respectively, and only the clip pairs with the top n similarity are retained as the action video features for metric learning, thereby eliminating the clip pairs with non-action information to resist the interference of non-action clips.
[0028] The present invention provides a method for identifying action with a small number of samples by paying attention to time sequence alignment in stages, which specifically comprises the following steps:
[0029] S1, for action sample videos and action query videos, divide the videos into an unfixed number of video segments, the number of segments is M and N respectively, and each video segment contains 16 video frames; use the pre-trained C3D model as the video segment feature extractor to extract features for each video segment, and use the video segment features for subsequent temporal alignment; the action query video segment feature is denoted as Q, and the action sample video segment feature is denoted as S.
[0030] S2, through the multi-attention module in stages, achieves the temporal alignment of video clips, so that the alignment of video clips in each stage is only performed within the relevant clips of the adjacent stage, so as to ensure that the final aligned video clip pairs have a temporal precedence relationship at the stage level; see Figure 2 , the specific steps are:
[0031] S21, dividing the action query video into m stages on average, and dividing the action sample video into m+1 stages on average;
[0032] S22, using the attention module to perform temporal alignment for each stage, restricting the features of each video clip in stage i of the action query video to be aligned only with the features of each video clip in stage i and stage i+1 of the action sample video, thereby ensuring that the temporal order of the video clip pairs precedes the partial order relationship at the stage level, so that the attention of each stage can focus on the action temporal features of the current stage;
[0033] The aligned attention weight matrix of the action query video features and the action sample video related segment features in the i-th stage is expressed as follows:
[0034] A i=softmax((S i W i +b)Q i T ), i∈(1,m)
[0035] in: Is The matrix composed of the action sample video clip features between Is The matrix composed of action query video clip features between them, aligning the attention weights W i is the attention module, b is the bias term, N is the number of action sample video clips, M is the number of action query video clips, d is the length of each video clip feature, and R is the real number field.
[0036] S23, multiply the attention weight matrix by the action sample video feature to obtain the alignment result H of the action sample video feature in the i-th stage i :
[0037]
[0038] S24, splice the alignment results m times to obtain the action sample video features
[0039] H=(H1,...,H m ).
[0040] S3, using the segment-pair similarity-based Top n confidence selection method to remove video segments with too low similarity, and perform metric learning action classification to resist the interference of non-action video segments; see Figure 3 , the specific steps are:
[0041] S31, for the aligned action sample video clip feature H∈R M×d and action query video clip features Q∈R M×d , pair them in order, and calculate the cosine distance between the corresponding action sample video clip features and the action query video clip features as the similarity sim between the features;
[0042] S32, consider the segment pairs with similarity in the top n as credible action-related segment pairs, retain them and discard the segment pairs with similarity lower than the top n for metric learning, so as to eliminate the segment pairs with non-action information. The credible top n segment pairs are selected as follows:
[0043]
[0044] Among them: sim iis the similarity between the features of action query video segment i and the features of the corresponding action sample video segment, SIM refers to the set of similarities between the features of the action query video related segments and the features of the corresponding action sample video segments, and top(SIM,n) refers to the set of the first n largest values in the similarity set between the features of the action query video related segments and the features of the corresponding action sample video segments.
[0045] S33, input the similarity feature b of the action video segment pair after the Top n segment pair similarity confidence selection into a unidirectional GRU unit, further fuse and learn the temporal features, input the output result of the last layer of the GRU unit into the fully connected layer, and obtain the video-level correlation score of the action query video and the action sample video through fusion learning; after the correlation score of the action query video and different action sample videos is subjected to softmax, a probability distribution of the category to which the action query video belongs is formed. If the probability that the action query video belongs to a certain category of action sample videos is the largest, then the action query video is determined to be an action video of that category.
[0046] The present invention proposes a few-shot action recognition method with staged attention and temporal alignment, so that the attention modules at different stages can pay attention to the unique features of a specific stage of the video, avoiding the temporal mismatch at the video stage level; at the same time, a Top n confidence selection method based on fragment similarity is used to eliminate fragments with low similarity for metric learning action classification, so as to resist the interference of non-action fragments on video action classification, and effectively improve the accuracy of few-shot action recognition.
[0047] The above-mentioned method for few-sample action recognition with phased attention and temporal alignment can be implemented in the form of a computer program, which can be run on a computer device, and the computer device can be a server or a terminal. The server can be an independent server or a server cluster; the terminal can be an electronic device such as a tablet computer, a laptop computer, and a desktop computer.
[0048] The computer device includes a processor and a memory connected via a system bus. The processor is used to provide computing and control capabilities to support the operation of the entire computer device. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium can store an operating system and a computer program; the internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute the staged attention timing alignment of the few-sample action recognition method of the present application.
[0049] It should be understood that the processor should preferably be a graphics processing unit (GPU) to cope with the large number of matrix operations required in the neural network. The processor can also be a central processing unit (CPU) or any other processor with considerable computing power.
[0050] In an embodiment of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement the staged attention timing alignment few-sample action recognition method of the present application.
[0051] The computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.
[0052] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.
Claims
1. A staged attention-temporal alignment method for few-shot action recognition, characterized by: The action query video is divided into m stages on average, and the action sample video is divided into m+1 stages on average. The attention module is used to perform temporal alignment for each stage, so that the features of each video clip in the i-th stage of the action query video are aligned with the features of each video clip in the i-th and i+1-th stages of the action sample video; The aligned action sample video clip features and action query video clip features are paired in order and the similarity is calculated. Only the clip pairs with the top n similarities are retained as the action video features for metric learning. The aligned attention weight matrix of the action query video features and the action sample video related segment features in the i-th stage is expressed as: A i =softmax((S i W i +b)Q i T ),i∈(1,m) in: Is The matrix composed of the action sample video clip features between Is The matrix composed of action query video clip features between the two, aligning the attention weight matrix W i is the network parameter of the attention module, b is the bias term, N is the number of action sample video clips, M is the number of action query video clips, d is the length of each video clip feature, and R is the real number domain; The transpose of the alignment attention weight matrix is multiplied by the action sample video feature to obtain the alignment result H of the action sample video feature in the i-th stage i : Splice the alignment results m times to get the action sample video feature H = (H1,...,H m ), where i∈(1,m).
2. The method for few-shot action recognition with staged attention and temporal alignment according to claim 1, characterized in that: The process of acquiring the video segment features is as follows: dividing the action sample video and the action query video into an unfixed number of video segments, each video segment containing 16 video frames, and extracting the video segment features using a video segment feature extractor.
3. The method for few-shot action recognition with staged attention and temporal alignment according to claim 2, characterized in that: The video clip feature extractor selects a pre-trained C3D model.
4. The method for few-shot action recognition with staged attention and temporal alignment according to claim 1, characterized in that: The top n similarity segments are based on Determine, where M is the number of action query video clips, sim i is the similarity between the features of action query video segment i and the features of the corresponding action sample video segment, SIM refers to the set of similarities between the features of the action query video related segments and the features of the corresponding action sample video segments, and top(SIM,n) refers to the set of the first n largest values in the similarity set between the features of the action query video related segments and the features of the corresponding action sample video segments.
5. The method for few-shot action recognition with staged attention and temporal alignment according to claim 4, characterized in that: The similarity is the cosine distance between the corresponding action sample video segment features and the action query video segment features.
6. The method for few-shot action recognition with staged attention and temporal alignment according to claim 1, characterized in that: Also includes: The similarity features of the action video clips are input into the unidirectional GRU unit and the fully connected layer in sequence, and the video-level correlation scores of the action query video and the action sample video are obtained by fusion learning. The correlation scores of the action query video and different action sample videos are subjected to softmax to form the probability distribution of the category to which the action query video belongs. If the probability that the action query video belongs to a certain category of action sample videos is the largest, the action query video is determined to be an action video of that category.