A visual transformation method for action recognition based on long-term and short-term temporal differences

Abstract

The present invention relates to the technical field of action recognition, and in particular to a visual conversion method for action recognition based on long-term and short-term temporal differences. An LSMD framework is established, and an image model is followed in the design process of the LSMD framework. Meanwhile, LMIM and SMIF are respectively designed in the image model, so that the image model can obtain long-term and short-term temporal information in two different but complementary ways, thereby enabling the image model to obtain spatiotemporal modeling capability. A time difference suppression method is designed for SMIF to reduce noise interference caused by omitting the time difference between clips, and short-term temporal elements can be captured more effectively. In addition, the use of LMIM can capture the long-term temporal dynamics of video activities without additional parameters or increased computing requirements, thereby solving the existing 3D video-based tasks. If the spatiotemporal transformed video data is directly applied, the number of patches and the quadratic complexity of self-attention calculation will increase significantly, thereby increasing the computing and memory burden.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition, and in particular to an action recognition visual conversion method based on long-term and short-term time differences. Background Art

[0002] In the past few years, many researchers have paid extensive attention to temporal modeling. Recently, Transformers-based methods have made significant progress in processing 2D image-based visual tasks. The current main efficient video understanding method is to express temporal information on a picture and then directly use a 2D convolutional neural network (2DCNN) to recognize it. However, for a 2D CNN on a single frame, temporal information cannot be modeled well, and in the task of video action recognition, the biggest challenge is how to capture dynamic information in a time series.

[0003] However, for 3D video-based tasks such as action recognition, directly applying spatiotemporal transformations to video data can lead to a substantial increase in the number of patches as well as the quadratic complexity of self-attention computation, thus resulting in heavy computational and memory burdens.

[0004] Therefore, building an efficient and accurate 3D self-attention model for video data has become a major challenge for Transformers. This study proposes an effective long-short term motion difference (LSMD) method, which introduces short-term motion information into the image and weights the difference of several consecutive frames into the image, so that the original image has the ability to model short-term motion. At the same time, a module that can capture long-term motion information is added, and the time difference across segments is directly integrated into the model through motion excitation to enhance the model's long-term motion modeling ability. A large number of experimental results show that the proposed LSMD maintains high recognition accuracy on multiple benchmarks (such as UCF101 and HMDB51). Summary of the invention

[0005] The purpose of the present invention is to provide a visual transformation method for action recognition based on long-term and short-term temporal differences, which solves the technical problem that in existing 3D video-based tasks, such as action recognition, if the spatiotemporal transformed video data is directly applied, the number of patches and the quadratic complexity of self-attention calculation will increase significantly, thereby increasing the computational and memory burden.

[0006] To achieve the above object, the present invention provides a method for visual conversion of action recognition based on long-term and short-term temporal differences, comprising the following steps:

[0007] Step 1: Input the video and use the video-level learning motion model of the entire video information to establish the long short-term motion difference (LSMD) framework;

[0008] Step 2: Based on the framework of the image model, the temporal difference operator is used to capture short-term and long-term motion information;

[0009] Step 3: placing the fusion of the short-term motion information before the image model input to form a short-term motion information image (SMIF);

[0010] Step 4: Add a long-term motion information module (LMIM) to the image model to obtain long-term motion information through a time difference factor;

[0011] Step 5: The input video is converted into a picture of the frame of the image model through the SMIF, and then sent to the image model with the LMIM added.

[0012] Among them, the LSMD in the step 1 is a strategy to promote effective spatiotemporal self-attention (SA) modeling in the 2D Transformers structure, which can provide efficient image representation.

[0013] The SMIF in step three acts on the network input and is used to extract the motion information of the image within the segment by fusing the time difference information, so that the single-frame RGB can perceive the local motion.

[0014] The specific operation mode of the SMIF in step 3 is as follows: for each sampling frame I i We are in a i The upper and lower frames are extracted from the local window centered as the short-term image set S i =[I i-2 ,I i-1 ,I i ,I i+1 ,I i+2 ], in the short-term image set, we make a difference between each other and perform short-term time difference:

[0015] [D i-2 ,D i-1 ,D i ,D i+1 ]=[I i-2 -I i-1 ,I i-1 -I i ,I i -I i+1 ,I i+1 -I i+2 ]

[0016] Due to image noise interference and illumination changes, although only the person itself changes, some small changes will still be calculated in places far away from the person. Such changes should not be considered to reflect real motion. In order to discard these small changes, we propose the time difference suppression (TDI) method based on the motion focus idea proposed by SSTSA. Specifically, a threshold is applied to the frame difference image D, and pixels below the threshold are turned into 0 to achieve the purpose of suppressing interference:

[0017]

[0018] Where α is the threshold 1≥α≥0, k=X max -X min is the scaling factor of the input pixel intensity value, that is, when the input D is not normalized, K = 255, β is the enhancement factor to enhance the motion information. In the image, the moved pixels are enhanced by β times to enhance the effect of suppressing interference. i Frame, we denote the adjacent frames of the same segment as I i-1 and I i+1 . Therefore, we can get the forward time difference and the backward time difference as:

[0019]

[0020]

[0021] SSTSA research shows that by analyzing adjacent frames through bidirectional calculations and capturing dynamic action trends in the front and back views, the model can well capture the motion details in the current frame without requiring a large number of parameters for easy calculation. Behind and Zero mapping was added earlier to ensure that the number of frames is restored to the initial state. In order to analyze the time trends in two directions, the collective time variance is processed as follows:

[0022]

[0023] Then along the time channel dimension SD(I i )=[D i-2 , D i-1 , D i , D i+1 , D i+2 ]Perform average frame difference operation.

[0024] D(I i )=AVG(D i-2 , D i-1 , D i , D i+l , D i+2 )

[0025] Where AVG represents the sum of the four frame difference images into a short-term suppressed frame difference image D(I i ), and finally D(I i ) is added to the original input fragment X to obtain the short-term information frame difference image.

[0026] The LMIM in step 4 is a module integrated with existing 2D Transformers, and its notable feature is that it can capture the long-term temporal dynamics of video activities without requiring additional parameters or increasing computational requirements.

[0027] Among them, when the LMIM in the step 4 is inserted into the block L, in addition to considering the computational cost, the spatial position transformation between the long-term motion information frames is also a problem. Therefore, a module for reducing the feature dimension is designed. First, the feature dimension is reduced to 1 / r of the original, and the time difference after alignment is calculated through adjacent segments:

[0028]

[0029] Where C(F i , F i+1 ) indicates segment F i The time difference after alignment, and It is the frame-level feature after reducing the number of features, thereby alleviating the problem of missing long-term motion alignment. Since the subtraction of adjacent frames only produces T-1 time differences, we need to f The start and C b The end parts of are added with zero mapping to compensate for the reduction of time difference caused by the reduction of adjacent frames. Then, the bidirectional inter-period time difference method is used to optimize the inter-segment features using the aligned time difference, as follows:

[0030]

[0031] Where ⊙ is an element-based multiplication. We also combine the original frame-level representation and enhance the representation through the residual connection shown in the above formula. The SMIF will complement the LMIM and provide information to each other to assist the model in extracting features.

[0032] The specific operation of the action recognition visual conversion method based on long-term and short-term temporal difference is as follows: since each video segment is of different lengths, the video V is divided into T segments without overlap. We select frames from each segment as x, with a shape of [N, T, C, H, W], where N is the number of videos processed simultaneously, T is the frame sequence size, C is the feature channel, H and W are the length and width of each frame respectively, and the interval between each frame is the same, so a total of T frames I = [I1, ..., I T], where the shape of I is [T, C, H, W]. SMIF aims to provide local motion information for images to improve their expressive power:

[0033]

[0034] in, Represents a short-term motion information image, which is from I i Extract local motion information from adjacent frames, D represents the frame difference image, and F represents the frame-level features extracted by sending these frames to 2D Transformer, where F = [F1, ..., F T ], the long-term motion module mainly uses the cross-segment temporal structure to enhance the frame-level feature representation:

[0035]

[0036] Among them, L represents the LMIM module. In the long-term motion information, only the information of adjacent frames is considered in each long-term temporal modeling. By superimposing multiple LMIMs, the model can capture the temporal structure in long-term motion.

[0037] The present invention discloses a visual conversion method for action recognition based on long-term and short-term temporal differences. The present invention establishes a long-term and short-term motion difference (LSMD) framework, and follows the image model in the design process of the LSMD framework. At the same time, a long-term motion information module (LMIM) and a short-term motion information image (SMIF) are respectively designed in the image model, so that the image model can obtain long-term and short-term temporal information in two different but complementary ways, thereby enabling the image model to obtain spatiotemporal modeling capabilities, thereby reducing noise interference caused by omitting the time difference between clips, and being able to more effectively capture short-term temporal elements. At the same time, it is possible to capture the long-term temporal dynamics of video activities without the need for additional parameters or increased computing requirements, thereby solving the existing 3D video-based tasks, such as action recognition, in which directly applying spatiotemporal transformed video data will lead to a substantial increase in the number of patches and the quadratic complexity of self-attention calculations, thereby increasing the computing and memory burdens. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0039] Figure 1 It is a flowchart of the action recognition visual conversion method based on long-term and short-term time differences according to the first embodiment of the present invention.

[0040] Figure 2 It is a schematic diagram of the overall framework of long-term and short-term motion differences according to the first embodiment of the present invention.

[0041] Figure 3 FIG. 4 is a schematic diagram of a short-term motion information image according to the first embodiment of the present invention.

[0042] Figure 4 FIG. 1 is a schematic diagram for summarizing the time difference suppression studied in the short-term motion information image according to the first embodiment of the present invention.

[0043] Figure 5 It is an explanatory diagram of the long-term motion information module according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0044] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0045] First embodiment:

[0046] See also Figures 1 to 5 ,in Figure 1 is a flowchart of a method for visual conversion of action recognition based on long-term and short-term temporal differences according to a first embodiment of the present invention. Figure 2 1 is a schematic diagram of the overall framework of long-term and short-term motion differences according to the first embodiment of the present invention. Figure 3 is a schematic diagram of a short-term motion information image according to a first embodiment of the present invention, Figure 4 is a schematic diagram of the overview of the time difference suppression studied in the short-term motion information image of the first embodiment of the present invention, Figure 5 is an explanatory diagram of a long-term motion information module of a first embodiment of the present invention. The present invention provides a method for visual conversion of motion recognition based on long-term and short-term time differences, comprising the following steps:

[0047] S101, step 1: input a video, and establish a long short-term motion difference (LSMD) framework using a video-level learning motion model of the entire video information;

[0048] S102, step 2: based on the framework of the image model, using a time difference operator to capture short-term and long-term motion information;

[0049] S103, step 3: placing the fusion of the short-term motion information before the image model input to form a short-term motion information image (SMIF);

[0050] S104, step 4: adding a long-term motion information module (LMIM) to the image model, and obtaining long-term motion information through a time difference factor;

[0051] S105, step five: converting the input video into a picture form of the frame of the image model through the SMIF, and then sending it to the image model with added LMIM.

[0052] In this embodiment, the present invention establishes a long-short-term motion difference (LSMD) framework, and follows the image model in the design process of the LSMD framework. At the same time, a long-term motion information module (LMIM) and a short-term motion information image (SMIF) are designed in the image model respectively, so that two different but complementary methods can enable the image model to obtain long-term and short-term time information, thereby enabling the image model to obtain spatiotemporal modeling capabilities, and a time difference suppression method is designed for SMIF to reduce noise interference caused by omitting the time difference between clips, so that short-term time elements can be captured more effectively. At the same time, the long-term temporal dynamics of video activities can be captured without additional parameters or increased computing requirements, solving the existing 3D video-based tasks, such as action recognition. If the spatiotemporal transformed video data is directly applied, it will lead to a substantial increase in the number of patches and the quadratic complexity of the self-attention calculation, thereby increasing the computing and memory burden. Technical problems.

[0053] Among them, the LSMD in the step 1 is a strategy to promote effective spatiotemporal self-attention (SA) modeling in a 2D Transformer structure, which can provide efficient image representation.

[0054] Secondly, the SMIF in step 3 acts on the network input and extracts the motion information of the image in the segment by fusing the time difference information, so that the single frame RGB can perceive the local motion. The specific operation mode of the SMIF in step 3 is: for each sampling frame I i We are in a i The upper and lower frames are extracted from the local window centered as the short-term image set S i =[I i-2 , I i-1 , I i , I i+1 , I i+2 ], subtract each other in the short-term image set, and make short-term time difference:

[0055] [D i-2 , D i-1 , D i , D i+1 ]=[I i-2 -I i-1 , I i-1 -I i , I i -I i+1 , I i+l -Ii+2 ]

[0056] Due to image noise interference and illumination changes, although only the person itself changes, some small changes will still be calculated in places far away from the person. Such changes should not be considered to reflect real motion. In order to discard these small changes, we propose the time difference suppression (TDI) method based on the motion focus idea proposed by SSTSA. Specifically, a threshold is applied to the frame difference image D, and pixels below the threshold are turned into 0 to achieve the purpose of suppressing interference:

[0057]

[0058] Where α is the threshold 1≥α≥0, k=X max -X min is the scaling factor of the input pixel intensity value, that is, when the input D is not normalized, K = 255, β is the enhancement factor to enhance the motion information. In the image, the moved pixels are enhanced by β times to enhance the effect of suppressing interference. i Frame, we denote the adjacent frames of the same segment as I i-1 and I i+1 . Therefore, we can get the forward time difference and the backward time difference as:

[0059]

[0060]

[0061] SSTSA research shows that by analyzing adjacent frames through bidirectional calculations and capturing dynamic action trends in the front and back views, the model can well capture the motion details in the current frame without requiring a large number of parameters for easy calculation. Behind and Zero mapping was added earlier to ensure that the number of frames is restored to the initial state. In order to analyze the time trends in two directions, the collective time variance is processed as follows:

[0062]

[0063] Then along the time channel dimension SD(I i )=[D i-2 , D i-1 , D i , D i+1 , D i+2 ]Perform average frame difference operation.

[0064] D(I i )=AVG(D i-2 , D i-1 , D i, D i+l , D i+2 )

[0065] Where AVG represents the sum of the four frame difference images into a short-term suppressed frame difference image D(I i ), and finally D(I i ) is added to the original input fragment X to obtain the short-term information frame difference image.

[0066] Again, the SMIF in step three can generate a single-frame RGB image for each frame of the video, which integrates visual appearance and motion details and can use images to describe the characteristics of video information.

[0067] At the same time, a time difference suppression (TDI) method is added to the SMIF in step three, which enhances the ability to integrate short-term motion details into the image. It captures short-term time elements more effectively by reducing noise interference caused by omitting the time difference between segments.

[0068] In addition, the LMIM in step 4 is a module integrated with existing 2D Transformers. Its notable feature is that it can capture the long-term temporal dynamics of video activities without additional parameters or increased computing requirements. When the LMIM in step 4 is inserted into block L, in addition to considering the computational cost, the spatial position transformation between long-term motion information frames is also a problem. Therefore, a module for reducing the feature dimension is designed. First, the feature dimension is reduced to 1 / r of the original, and the aligned time difference is calculated through adjacent segments:

[0069]

[0070] Where C(F i , F i+1 ) indicates segment F i The time difference after alignment, and It is the frame-level feature after reducing the number of features, thereby alleviating the problem of missing long-term motion alignment. Since the subtraction of adjacent frames only produces T-1 time differences, we need to f The start and C b The end parts of are added with zero mapping to compensate for the reduction of time difference caused by the reduction of adjacent frames. Then, the bidirectional inter-period time difference method is used to optimize the inter-segment features using the aligned time difference, as follows:

[0071]

[0072] Where ⊙ is an element-based multiplication. We also combine the original frame-level representation and enhance the representation through the residual connection shown in the above formula. The SMIF will complement the LMIM and provide information to each other to assist the model in extracting features.

[0073] Finally, the specific operation of the action recognition visual conversion method based on long-term and short-term temporal differences is as follows: since each video segment is of different lengths, the video V is divided into T segments without overlap. We select frames from each segment as X, with a shape of [N, T, C, H, W], where N is the number of videos processed simultaneously, T is the frame sequence size, C is the feature channel, H and W are the length and width of each frame respectively, and the interval between each frame is the same, so a total of T frames I = [I1, ..., I T ], where the shape of I is [T, C, H, W]. SMIF aims to provide local motion information for images to improve their expressive power:

[0074]

[0075] in, Represents a short-term motion information image, which is from I i Extract local motion information from adjacent frames, D represents the frame difference image, and F represents the frame-level features extracted by sending these frames to 2D Transformer, where F = [F1, ..., F T ], the long-term motion module mainly uses the cross-segment temporal structure to enhance the frame-level feature representation:

[0076]

[0077] Among them, L represents the LMIM module. In the long-term motion information, only the information of adjacent frames is considered in each long-term temporal modeling. By superimposing multiple LMIMs, the model can capture the temporal structure in long-term motion.

[0078] When using the action recognition visual conversion method based on long-term and short-term temporal differences of the present embodiment, a video is input, and a long-term and short-term motion difference (LSMD) framework is established using a video-level learning action model of the entire video information. Based on the framework of the image model, a time difference operator is used to capture short-term and long-term motion information. After that, the fusion of the short-term motion information is placed before the image model input to form a short-term motion information image (SMIF). A long-term motion information module (LMIM) is added to the image model to obtain long-term motion information through a time difference factor. Then, the input video is converted into a picture form of the framework of the image model through the SMIF, and then sent to the image model with the LMIM added to achieve the action recognition visual conversion of the video.

[0079] The technical solution of the action recognition visual conversion method based on long-term and short-term time difference adopted by the present invention is as follows:

[0080] Action recognition is a challenging and cornerstone problem in vision, and many deep learning based methods have been proposed recently. Based on the adopted network structure, they can be divided into CNN-based and Transformers-based.

[0081] 1. CNN-based methods: CNN-based methods usually use 3D convolution or 2D CNN with temporal modeling to build a backbone network for action recognition. For example, Tran et al. proposed C3D, a 3D CNN based on the VGG model, which achieved the goal of learning spatiotemporal features from video sequences by training the VGG model using 3D convolution. 3D expands all 2D convolution filters of the InceptionV1 model into 3D convolution. Slowfast uses two parallel 3D-CNN neural network branches, one for processing high frame rate video frames and the other for processing low frame rate video frames to achieve more accurate video analysis. Due to the heavy computational burden of 3D-CNN, many works have attempted to enhance 2D-CNN with temporal modules. For example, P3D decomposes 3D convolution into 1D temporal convolution and 2D spatial convolution, and then gradually adds 3D convolution kernels layer by layer, effectively capturing the temporal and spatial information in the video and improving the action recognition performance. TSM proposes an effective shift module, which uses left and right shifts of sub-channels to replace the grouped weighted fixed 1D temporal convolution. TEA adopts a temporal excitation mechanism and an information aggregation module to capture motion information and increase the temporal receptive field. TEINet combines temporal excitation and inter-frame interaction mechanisms, and performs effective temporal modeling through motion enhancement modules and deep 1D convolutions. However, CNN-based methods cannot effectively model long-range dependencies within or across frames, which limits their performance.

[0082] 2. Transformers-based methods: Recently, with the development of Transformers in 2D vision tasks, there have been many works applying Transformers to video action recognition. Unlike CNN's 3D convolution or its factorized version, Transformers mainly explore spatiotemporal video correlations through spatiotemporal SA modeling. People can calculate all SAs through blocks. However, this operation, like 3D CNN, has too large computational cost and parameters. Therefore, many works have proposed to reduce the computational burden of joint spatiotemporal SA modeling. For example, Timesformer decomposes the video into frame-level image blocks, splits time and space SA, and adds time SA after each spatial SA. ViT better captures the temporal relationship and dynamic changes in the video by superimposing several layers of temporal Transformer encoders on the output of the spatial encoder. VideoSwinTransformer reduces computational complexity and memory consumption by dividing local windows in space and time and performing self-attention within the window. TPS designs a block transfer and channel transfer module to improve the temporal modeling ability of the transformer.

[0083] The current mainstream method is to complete the modeling of time series through 2D CNN and 3D CNN. TSM first proposed temporal modeling between video frames to fully capture motion features. TRN enhanced temporal reasoning capabilities by fusing multi-scale features in the temporal dimension. STM proposed a new block to replace the conventional residual block to model spatiotemporal and motion features. TEA and TEINet use temporal difference operations to design network models. TDN considers multi-scale temporal difference modeling methods to fully capture motion information and can achieve end-to-end action recognition. TPS proposes block transfer based on TSM, which essentially complement each other. The success of the temporal module in 2D CNN prompted us to develop LSMD to enhance spatial Transformers with spatiotemporal feature learning capabilities. Our work improves the temporal modeling capabilities of Transformers without adding additional parameters. SMIF uses temporal differences to give images short-term motion information features, while LMIM reduces the number of channel features to alleviate the problem of excessive motion information differences between two frames.

[0084] Among them, the framework of long-short-term motion difference (LSMD) mainly transforms spatial Transformers into spatiotemporal Transformers through long-term motion information module (LMIM) and short-term motion information image (SMIF). Next, we describe the LMIM and SMIF methods in detail.

[0085] 1. Overview:

[0086] like Figure 2 As shown in the figure, the Long Short-Term Motion Difference (LSMD) framework we proposed is a video-level learning motion model framework that utilizes the entire video information. Based on the VIT framework, we use the temporal difference operator to capture short-term and long-term motion information. When designing the LSMD framework, considering the excessive calculation of VIT itself, we place the fusion of short-term information before the model input to form a short-term motion information image. At the same time, a pluggable LMIM module is added to the model to selectively reduce channel information and alleviate the motion misalignment problem caused by long-term motion.

[0087] Specifically, each video segment is of different lengths. The video V is divided into T segments without overlap. We select frames from each segment as X, with the shape of [N, T, C, H, W], where N is the number of videos processed simultaneously, T is the frame sequence size, C is the feature channel, H and W are the length and width of each frame respectively, and the interval between each frame is the same, so a total of T frames I = [I1, ..., I T ], where the shape of I is [T, C, H, W]. SMIF aims to provide local motion information for images to improve their expressive power:

[0088]

[0089] in Represents a short-term motion information image, which is from I i Extract local motion information from adjacent frames, D represents the frame difference image. F represents the frame-level features extracted by sending these frames to 2D Transformer, where F = [F1, ..., F T ], the long-term motion module mainly uses the cross-segment temporal structure to enhance the frame-level feature representation:

[0090]

[0091] Where L represents the LMIM module. In the long-term motion information, we only consider the information of adjacent frames in each long-term temporal modeling. By superimposing multiple LMIMs, the model can capture the temporal structure in long-term motion. The specific implementation will be described in the long-term information expression.

[0092] 2. Short-term information expression:

[0093] When analyzing action videos, we found that the data of adjacent frames are closely correlated, and simply stacking this information together for processing is very inefficient. In addition, selecting a single frame from each video clip can only capture the spatial aspects of the action without providing insight into the temporal information. Therefore, we introduced the SMIF method, which generates a single RGB image for each frame that integrates the visual appearance and motion details to describe the characteristics of the video information with an image.

[0094] By differentiating the original image and then performing time difference suppression on it, the obtained short-term frame difference image is combined with the original image to obtain a short-term motion information image.

[0095] Specifically, our SMIF acts on the network input and is used to extract the motion information of the image within the segment by fusing the temporal difference information, so that a single frame RGB can perceive local motion. Figure 3 As shown, for each sampling frame I j We are in a i The upper and lower frames are extracted from the local window centered as the short-term image set S i =[I i-2 , I i-1 , I i , I i+l , I i+2 ], subtract each other in the short-term image set, and make short-term time difference:

[0096] [D i-2 , D i-1 , D i , D i+1 ]=[I i-2 -I i-1 , I i-1 -I i , I i -I i+1 , I i+1 -I i+2 ]

[0097] Due to image noise interference and illumination changes, short-term temporal difference may cause some small changes in places far away from the person, even though only the person changes. Such changes should not be considered to reflect real motion. In order to discard these small changes, we propose the time difference suppression (TDI) method based on the motion focus idea proposed by SSTSA. Specifically, a threshold is applied to the frame difference image D, and pixels below the threshold are turned into 0 to achieve the purpose of suppressing interference:

[0098]

[0099] Where α is the threshold 1≥α≥0, k=X max -X min is the scaling factor of the input pixel intensity value, that is, when the input D is not normalized, K = 255, β is the enhancement factor to enhance the motion information. In the image, the moved pixels are enhanced by β times to enhance the effect of suppressing interference. i Frame, we denote the adjacent frames of the same segment as I i-1 and I i+1. Therefore, we can get the forward time difference and the backward time difference as:

[0100]

[0101]

[0102] SSTSA research shows that by analyzing adjacent frames through bidirectional calculations and capturing the dynamic action trends of the front view and the back view, the model can well capture the motion details in the current frame without requiring a large number of parameters. Behind and Zero mapping was added to ensure that the number of frames is restored to the initial state. In order to analyze the time trend in two directions, the collective time variance is processed as follows:

[0103]

[0104] Then along the time channel dimension SD(I i )=[D i-2 , D i-1 , D i , D i+1 , D i+2 ]Perform average frame difference operation.

[0105] D(I i )=AVG(D i-2 , D i-1 , D i , D i+l , D i+2 )

[0106] Where AVG represents the sum of the four frame difference images into a short-term suppressed frame difference image D(I i ),like Figure 4 As shown, finally D(I i ) is added to the original input fragment X to obtain the short-term information frame difference image.

[0107] 3. Long-term information expression:

[0108] While SMIF frame-based representations excel in capturing local spatiotemporal information, their applications are limited to intra-segment video analysis, thus limiting their scope in exploring the extended spatiotemporal dimension of motion models. Inspired by META and CLS-Net, our approach to LMIM focuses on leveraging inter-segment data to enhance the representation of long-term information. This is achieved through a bidirectional multi-scale temporal difference module tailored for long-term information analysis.

[0109] Among them, the long-term motion information module reduces the channel feature dimension, and then performs a differential process, and then performs a point multiplication on the channel with the original feature. This module is inserted into blockL. In addition to considering the computational cost, the spatial position transformation between long-term motion information frames is also a problem. Therefore, we designed a module to reduce the feature dimension, such as Figure 5 As shown, we first reduce the feature dimension to 1 / r of the original, and calculate the aligned time difference through adjacent segments:

[0110]

[0111] Where C(F i , F i+1 ) indicates segment F i The time difference after alignment, and is the frame-level feature after reducing the number of features, thus alleviating the problem of missing long-term motion alignment. Since the subtraction of adjacent frames only produces T-1 time differences, we need to f The start and C b The end parts of are added with zero mapping to compensate for the reduction of time difference caused by the reduction of adjacent frames. Then, the bidirectional inter-period time difference method is used to optimize the inter-segment features using the aligned time difference, as follows:

[0112]

[0113] Where ⊙ is element-based multiplication. We also combine the original frame-level representation and enhance the representation through residual connection. SMIF and LMIM complement each other and provide information to assist the model in extracting features.

[0114] 4. Experiment:

[0115] In this section, we present the experimental results of our LS-VIT. First, we describe the evaluation datasets and specific implementation details. Next, we conduct ablation experiments on the design of LS-VIT. Finally, we compare LS-VIT with state-of-the-art methods.

[0116] 4.1 Dataset

[0117] We evaluate our model on two datasets: UCF101 and HMDB51. UCF101 contains 101 human action categories and 13,320 labeled video clips collected from the Internet. HMDB51 is a collection of video clips extracted from various resources, including movies and online videos, and it contains 6,776 videos with 51 action categories.

[0118] 4.2 Experimental configuration:

[0119] The model in this paper is implemented based on VIT, and T=8 or T=16 frames are extracted from each video. According to convention, during the training process, the short side of each video frame is adjusted to 256, and then randomly cropped to 224X224 size. We use the ImageNet dataset provided by VIT to fine-tune LSMD on K400. The initial learning rate is 0.003, the learning decay rate is 0.1, and the total number of training rounds is 50.

[0120] 4.3 Ablation experiment

[0121] To study the design of SMIF and the use of LMIM, we conduct a series of experiments in this section. All experiments are performed on HMDB51 with VIT-B as the backbone network (pre-trained weights of ImageNet21k+K400). For clarity, the ablation experiments all use T=16, D=5.

[0122] Effect study of SMIF: Short-term motion information frames are generated by integrating short-term frame difference images with base images, and further scaling and weighting them after considering the critical properties of traditional image pixel values. As shown in Table 1, various base images produce different image representation patterns after being merged with frame differences of different proportions. We will evaluate how different proportions affect the accuracy of the final image. It is worth noting that in the short-term motion information image, when the proportion of the frame difference image exceeds that of the base image, the expressiveness of the image is enhanced. This improvement is attributed to the suppression of temporal differences in the frame difference image, thereby enriching the fusion of the frame difference image with the base image and conveying motion information more effectively.

[0123] Table 1 The impact of different scales on SMIF, where X represents the original image and D represents the average frame difference

[0124]

[0125]

[0126] Effect of threshold and enhancement factor on SMIF: By setting the pixel-level change threshold α, the static region is weakened, thus simplifying the task of highlighting the dynamic part. This study shows that although the pixels in consecutive motion frames vary in motion intensity, ignoring small pixel fluctuations can focus more clearly on the essence of motion. In this case, we adopted the LS-VIT model and tried four thresholds: α = 0.02, α = 0.05, α = 0.07 and α = 0.1. The results in the first part of Table 2 show that the threshold of α = 0.02 outperforms the other thresholds, and it can effectively treat up to 98% of the pixel activities as significant motion. Subsequently, we explore the effect of the amplification factor β, which amplifies the motion intensity of the active area by β times and assigns zero to the non-moving area. After trying values ​​such as β = 0.02, β = 0.05, β = 0.07 and β = 0.1, the second part of Table 2 shows that LS-VIT performs best when β = 0.07.

[0127] Table 2 Effects of different thresholds α and enhancement factors β on LSMD

[0128]

[0129]

[0130] Study on the effect of channel reduction ratio on LMIM: Previously, the study evaluated the reason for proposing LMIM, which aims to reduce the number of channel features to better adjust motion information. Specifically, for VIT in the attention stage, the number of channel features is C = 768. In order to simplify the calculation process and adjust the tensor shape more effectively, we tried various reduction amounts, including r = 2, r = 3, r = 4, r = 6, and r = 8. According to Table 3, reducing the image channels to 1 / 6 of the original number (r = 6) provides a more effective solution to the image alignment problem, and LMIM achieves the best performance in this case.

[0131] Table 3 Effect of different channel feature reduction ratios r on LMIM

[0132]

[0133] Effect of SMIF and LMIM on LS-VIT: SMIF and LMIM are integrated at different positions in the model, each making unique contributions to the processing of long-term and short-term motion information. This complementary effect can be evaluated by comparing the best performance of SMIF and LMIM in enhancing the recognition ability of LSMD, as shown in Table 4. Both SMIF and LMIM can enhance VIT recognition when used alone, with SMIF slightly outperforming LMIM. However, combining the two methods together does not significantly improve recognition. The results shown in Table 5 show that the combination of SMIF and LMIM is most effective when the reduction ratio is r = 1, indicating that LSMD can benefit from minimal channel reduction when extracting short-term motion information. This minimization avoids excessive compression of the channels, thereby fully capturing long-term motion information without affecting the effectiveness of the model.

[0134] Table 4 The impact of SMIF and LMIM on the model, √ means this module is used, - means it is not used.

[0135]

[0136]

[0137] Table 5 Effect of SMIF and different LMIM reduction ratios

[0138]

[0139] 4.4. Comparison with the state of the art (sota):

[0140] As shown in Table 6, we conducted experiments on the HMDB51 dataset with different numbers of input frames. To ensure a fair comparison, we followed the standard evaluation scheme for this dataset. From the experimental results, we can see that when the number of input frames is 16, our model significantly outperforms TSM and TDN in performance. The action categories in the HMDB51 dataset are highly correlated with motion information, so the importance of temporal modeling on this dataset is particularly prominent. Compared with other methods, our model can capture and utilize temporal information more accurately, resulting in a significant improvement in recognition performance.

[0141] Table 6 Comparison with the most advanced methods on HMDB51

[0142]

[0143] We also specifically compared with the current state-of-the-art methods on the UCF-101 dataset, and the detailed results are shown in Table 7. Similar to the HMDB51 dataset, when taking 16 frames as input, our LSMD model showed comparable performance to other models. It is worth noting that since the UCF-101 dataset is relatively small, by using "ImageNet+K400" as a pre-training model, most models have been able to achieve a fairly high recognition accuracy, close to the saturation level. For example, the recognition accuracy of TDN reached 97.4%. In this case, it becomes extremely challenging to further improve the accuracy of the model. However, from the comparison results, we can see that LSMD can still achieve performance close to the current SOTA, which fully demonstrates its strong competitiveness.

[0144] Table 7. Comparison with the most advanced methods on UCF101. “*” indicates implementation by public code.

[0145]

[0146]

[0147] 5. Summary:

[0148] To address the challenge of detecting actions in videos, our research introduces an innovative network that analyzes both long-term and short-term motion distributions, aptly named the LSMD network. The network is specifically designed to model temporal information in a bidirectional manner, capturing motion trends from both forward and backward perspectives. This approach aims to overcome the common obstacles that plague current methods, which are high computational requirements or inefficient use of temporal data. In addition, we integrate a motion suppression method in the LSMD network to effectively minimize the disturbance caused by illumination changes, thereby improving recognition accuracy.

[0149] We experimentally evaluate the LSMD network on two widely recognized benchmark datasets and conduct a detailed analysis of its individual components. The results obtained under consistent experimental conditions show that the LSMD network exhibits superior performance and can more effectively utilize temporal information than current state-of-the-art methods. This study not only verifies the effectiveness of LSMD, but also provides new ideas and methods for the development of video action recognition.

[0150] What is disclosed above is only one or more preferred embodiments of the present application, and cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for visual transformation of action recognition based on long-term and short-term temporal differences, characterized in that: The following steps are involved: Step 1: Input the video and use the video-level learning motion model of the entire video information to establish the long short-term motion difference (LSMD) framework; Step 2: Based on the framework of the image model, the temporal difference operator is used to capture short-term and long-term motion information; Step 3: placing the fusion of the short-term motion information before the image model input to form a short-term motion information image (SMIF); Step 4: Add a long-term motion information module (LMIM) to the image model to obtain long-term motion information through a time difference factor; Step 5: The input video is converted into a picture of the frame of the image model through the SMIF, and then sent to the image model with the LMIM added.

2. The method for visual conversion of action recognition based on long-term and short-term temporal differences according to claim 1, characterized in that: The LSMD in the step 1 is a strategy to promote effective spatiotemporal self-attention (SA) modeling in the 2D Transformers structure, which can provide efficient image representation.

3. The method for visual conversion of action recognition based on long-term and short-term temporal differences according to claim 1, characterized in that: The SMIF in step three acts on the network input and is used to extract the motion information of the image within the segment by fusing the time difference information, so that a single frame RGB can perceive local motion.

4. The method for visual conversion of action recognition based on long-term and short-term temporal differences according to claim 3, characterized in that: The specific operation mode of the SMIF in step 3 is: for each sampling frame I i We are in a i The upper and lower frames are extracted from the local window centered as the short-term image set S i =[I i-2 ,I i-1 ,I i ,I i+1 ,I i+2 ], in the short-term image set, we make a difference between each other and perform short-term time difference: [D i-2 ,D i-1 ,D i ,D i+1 ]=[I i-2 -I i-1 ,I i-1 -I i ,I i -I i+1 ,I i+1 -I i+2 ] Due to image noise interference and illumination changes, although only the person itself changes, some small changes will still be calculated in places far away from the person. Such changes should not be considered to reflect real motion. In order to discard these small changes, we propose the time difference suppression (TDI) method based on the motion focus idea proposed by SSTSA. Specifically, a threshold is applied to the frame difference image D, and pixels below the threshold are turned into 0 to achieve the purpose of suppressing interference: Where α is the threshold 1≥α≥0, k=X max -X min is the scaling factor of the input that reflects the input pixel intensity value, that is, when the input D is not normalized, K = 255, β is the enhancement factor to enhance the motion information. In the image, the moved pixels are enhanced by β times to enhance the effect of suppressing interference; for I from segment i i Frame, we denote the adjacent frames of the same segment as I i-1 and I i+1 ; Therefore, we can get the forward time difference and backward time difference as: SSTSA research shows that by analyzing adjacent frames through bidirectional calculations and capturing dynamic action trends in the front and back views, the model can well capture the motion details in the current frame without requiring a large number of parameters for easy calculation. Behind and Zero mapping was added earlier to ensure that the number of frames is restored to the initial state. In order to analyze the time trends in two directions, the collective time variance is processed as follows: Then along the time channel dimension SD(I i )=[D i-2 ,D i-1 ,D i ,D i+1 ,D i+2 ]Perform average frame difference operation. <h2 style=";text-align:left;direction:ltr">D(I<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (AVG(D))<h2 style=";text-align:left;direction:ltr"> i-2 <h2 style=";text-align:left;direction:ltr"> ,D<h2 style=";text-align:left;direction:ltr"> i-1 <h2 style=";text-align:left;direction:ltr"> ,D<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> ,D<h2 style=";text-align:left;direction:ltr"> i+1 <h2 style=";text-align:left;direction:ltr"> ,D<h2 style=";text-align:left;direction:ltr"> i+2 <h2 style=";text-align:left;direction:ltr"> ) Where AVG represents the sum of the four frame difference images into a short-term suppressed frame difference image D(I i ), and finally D(I i ) is added to the original input fragment X to obtain the short-term information frame difference image.

5. The method for visual conversion of action recognition based on long-term and short-term temporal differences according to claim 1, characterized in that: The LMIM in step 4 is a module fused with existing 2D Transformers, and its notable feature is that it can capture the long-term temporal dynamics of video activities without requiring additional parameters or increasing computational requirements.

6. The method for visual conversion of action recognition based on long-term and short-term temporal differences according to claim 5, characterized in that: When the LMIM in step 4 is inserted into block L, in addition to considering the computational cost, the spatial position transformation between long-term motion information frames is also a problem. Therefore, a module for reducing the feature dimension is designed. First, the feature dimension is reduced to 1 / r of the original, and the time difference after alignment is calculated through adjacent segments: Where C(F i ,F i+1 ) indicates segment F i The time difference after alignment, and It is the frame-level feature after reducing the number of features, thereby alleviating the problem of missing long-term motion alignment. Since the subtraction of adjacent frames only produces T-1 time differences, we need to f The start and C b The end parts of are added with zero mapping to compensate for the reduction of time difference caused by the reduction of adjacent frames. Then, the bidirectional inter-period time difference method is used to optimize the inter-segment features using the aligned time difference, as follows: Where ⊙ is an element-based multiplication. We also combine the original frame-level representation and enhance the representation through the residual connection shown in the above formula. The SMIF will complement the LMIM and provide information to each other to assist the model in extracting features.

7. The method for visual conversion of action recognition based on long-term and short-term temporal differences according to claim 1, characterized in that: The specific operation of the action recognition visual conversion method based on long-term and short-term temporal differences is as follows: since each video segment is of different lengths, the video V is divided into T segments without overlap. We select frames from each segment and represent them as X, with a shape of [N, T, C, H, W], where N is the number of videos processed simultaneously, T is the frame sequence size, C is the feature channel, H and W are the length and width of each frame respectively, and the interval between each frame is the same, so a total of T frames I = [I1, ..., I T ], where the shape of I is [T,C,H,W]. SMIF aims to provide local motion information for images to improve their expressiveness: SMIF: in, Represents a short-term motion information image, which is from I i Extract local motion information from adjacent frames, D represents the frame difference image, and F represents the frame-level features extracted by sending these frames to 2D Transformer, where F = [F1,…,F T ], the long-term motion module mainly uses the cross-segment temporal structure to enhance the frame-level feature representation: LMIM: Among them, L represents the LMIM module. In the long-term motion information, only the information of adjacent frames is considered in each long-term temporal modeling. By superimposing multiple LMIMs, the model can capture the temporal structure in long-term motion.

Images