A spatiotemporal self-similar continuous sign language recognition method

Abstract

The application discloses a spatiotemporal self-similarity continuous sign language recognition method, which comprises the following steps: acquiring a continuous sign language video, and pre-processing video frames; constructing a continuous sign language recognition model, including a visual backbone network, an auxiliary branch for local prediction, and a main branch for global prediction; from the second stage of the visual backbone network, a time channel attention module is arranged at the end of each subsequent stage; a spatiotemporal self-similarity module is connected after the time channel attention module connected at the third stage, and the output of the spatiotemporal self-similarity module is used as an input of the next stage; and the output of the last stage is processed through a time channel attention module, and the processed result is used as the input of the auxiliary branch and the main branch. The application can describe the nonlinear action evolution across multiple frames without explicit optical flow or displacement estimation, and the structured self-similarity has stronger adaptability to absolute appearance changes while retaining local geometric relationships.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, video understanding, and continuous sign language recognition, specifically to a spatiotemporally self-similar continuous sign language recognition method. This method is designed for continuous sign language videos that have not undergone explicit temporal segmentation, transing the video into a sequence of sign language words (gloss). Background Technology

[0002] Continuous sign language recognition requires identifying consecutive sequences of sign language words from unsegmented video. Compared to isolated sign language recognition, continuous sign language video presents challenges such as coordinated articulation, blurred movement boundaries, nonlinear hand trajectories, rapid finger changes, and coordination between facial and arm cues. Identical or similar sign language gestures can also exhibit significant differences in appearance depending on the signer, the shooting background, and the speed of the movement.

[0003] Existing techniques typically employ frame-by-frame visual backbone networks to extract features, followed by sequence modeling using one-dimensional temporal convolutions, recurrent neural networks, or connection-temporal classification losses. Some methods enhance visual representations through assisted supervision, knowledge distillation, or spatial attention; others acquire motion cues through correlations between adjacent frames, local trajectories, or region interactions. These approaches generally emphasize interactions between adjacent frames or within short timeframes, capable of describing instantaneous displacements but struggling to maintain a complete relational structure across multiple time offsets. Consequently, unified modeling of nonlinear hand trajectories, rapid finger changes, and weak motion frames in long sequences remains insufficient.

[0004] Furthermore, if similarity relationships are directly constructed across all feature channels, the differences in the contributions of different channels to the action will be ignored during the channel summation process. Channels with weak temporal variations or noise responses may weaken the role of action-sensitive channels. Therefore, a method is needed that enhances the temporal discriminativeness of channels before constructing spatiotemporal self-similarity relationships, preserves structured similarity patterns within the extended spatiotemporal neighborhood, and can be jointly optimized end-to-end with continuous sign language sequence prediction networks.

[0005] However, it still relies on task-specific trajectory or region association design. Summary of the Invention

[0006] To address the problems of existing technologies that rely heavily on the interaction between adjacent frames or local time, have difficulty describing the evolution of structured actions in long sequences, and fail to adequately distinguish the importance of channel time when constructing correlations, this invention provides a spatiotemporal self-similar continuous sign language recognition method.

[0007] The technical solution adopted by the present invention to solve the aforementioned technical problem is as follows: A spatiotemporal self-similar continuous sign language recognition method, the method comprising the following: Acquire continuous sign language video, and resize and normalize the video frames; A continuous sign language recognition model is constructed, which includes a visual backbone network, an auxiliary branch for local prediction, and a main branch for global prediction. The visual backbone network consists of multiple convolutional stages connected in series. Starting from the second stage of the visual backbone network, a branch is led out after each subsequent stage to set a temporal channel attention module. In the third stage, a spatiotemporal self-similarity module is connected after the temporal channel attention module. The output of the spatiotemporal self-similarity module is used as an input to the next stage. The output of the last stage is processed by a temporal channel attention module and used as the input to the auxiliary branch and the main branch. Video frames are input into a visual backbone network that includes multiple convolutional stages. Each stage extracts intermediate visual features with dimensions of batch size (B), channel size (C), time (T), height (H), and width (W). In the temporal channel attention module, global average pooling is performed on the intermediate visual features of the corresponding stage in the spatial dimension to form a channel-temporal descriptor; multiple one-dimensional temporal convolutional branches with different temporal receptive fields are used to extract multi-scale temporal dependencies, and the results of each branch are weighted and fused with learnable coefficients. Then, a channel-temporal attention map is generated through local cross-channel interaction and the Sigmoid function; the attention map is used to perform residual modulation on the intermediate visual features of the corresponding stage to obtain the temporally enhanced visual features. In the spatiotemporal self-similarity module, firstly, the visual features enhanced by the time channels are normalized to L2 along the channel dimension; for each time position and each spatial anchor point, the normalized feature inner product is calculated in an extended neighborhood containing multiple relative time offsets and local spatial displacements to obtain a dense spatiotemporal self-similarity body that retains the topology of the time offset dimension and the two-dimensional spatial displacement dimension; the local spatial displacement mesh of the dense spatiotemporal self-similarity body is flattened and embedded into a compact spatial relationship through a fully connected layer; then, it is fused by three-dimensional convolution to obtain a relational motion feature with the same number of channels as the visual features enhanced by the time channels; the relational motion feature is added to the residual of the visual features enhanced by the time channels according to a learnable scaling factor to obtain the fused feature; The continuous sign language recognition model is trained using a continuous sign language dataset. The trained continuous sign language recognition model outputs a sequence of sign language words that does not require pre-segmentation.

[0008] Furthermore, the auxiliary branch adopts one-dimensional temporal convolution, and the output of the one-dimensional temporal convolution is sequentially connected to the bidirectional recurrent sequence model and the classifier to form the main branch. During the training phase, the main connection temporal classification loss, the auxiliary branch connection temporal classification loss, and the self-distillation loss between the main branch and the auxiliary branch are jointly used for end-to-end optimization. During the inference phase, the class probabilities output by the bidirectional recurrent sequence model are decoded to output a sign language word sequence that does not require pre-segmentation.

[0009] Furthermore, the process of using this attention map to perform residual modulation on the intermediate visual features X of the corresponding stage is as follows: Broadcast the channel-time attention map A to the spatial dimension and use the first scaling factor. Residual modulation is performed to obtain visual features enhanced by the temporal channel. The specific formula is expressed as follows: (3) Furthermore, the process for determining the dense spatiotemporal self-similar body is as follows: Set the time neighborhood size L, and set the horizontal and vertical displacement window sizes U and V. For each target time and space anchor points In the case of relative time offset Local two-dimensional spatial displacement Calculate the normalized inner product of the extended neighborhood using the normalized features. :

[0010] in, , , ; Indicates rounding down; This represents the normalized features within the extended neighborhood of the c-channel; This indicates the spatial anchor point at the target time t in channel c. Normalized features; Normalized inner products obtained over the entire horizontal and vertical displacement window and over the entire time neighborhood are all within the extended neighborhood. Forming dense spacetime self-similar bodies .

[0011] Furthermore, the number of the three-dimensional convolutions is multiple, divided into two categories: one type of three-dimensional convolution is used for offset-independent spatial aggregation, denoted as A1, and the other type of three-dimensional convolution is used for multi-offset stitching and fusion, denoted as A2. Both types of structures adopt the Conv+ReLU+BN form. The number of three-dimensional convolutions A1 used for offset-independent spatial aggregation can be arbitrarily adjusted; the number of three-dimensional convolutions A2 used for multi-offset stitching and fusion is 1, and the output of A2 is used for residual injection.

[0012] The present invention also protects a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, can implement the steps of the method.

[0013] Compared with the prior art, the present invention has the following beneficial effects: This invention can describe nonlinear motion evolution across multiple frames without explicit optical flow or displacement estimation; structured self-similarity is more adaptable to absolute appearance changes while preserving local geometric relationships; temporal channel attention suppresses redundant responses with low parameter overhead, improving the recognizability of weak motions and fine-grained gestures; residual fusion preserves the original appearance information and allows the network to adaptively control the injection intensity of relational motion information; the main loss, auxiliary loss, and self-distillation loss together improve the consistency between local and global predictions.

[0014] This invention represents motion structures by preserving the topology of temporal offsets and spatial displacements, rather than simply outputting the displacements of adjacent frames or immediately weighting and aggregating related values. Before constructing the self-similar bodies, multi-scale temporal convolution and local cross-channel interaction are used to generate channel-temporal attention, so that the features involved in the inner product calculation highlight the motion-sensitive channels in advance. A hierarchical approach of "displacement grid projection - offset independent spatial clustering - multi-offset stitching fusion - learnable residual injection" is adopted to transform the high-dimensional similar bodies into motion representations that can be further processed by the visual backbone network.

[0015] This invention designs a temporal channel attention module and a continuous sequence prediction process for recognizing rapid hand movements, fine-grained hand shape changes, weak action frames, and long sequences in continuous sign language, thereby improving the accuracy and speed of continuous sign language recognition. Attached Figure Description

[0016] Figure 1 This is a diagram illustrating the overall framework of the spatiotemporal self-similar continuous sign language recognition method of the present invention.

[0017] Figure 2 This is a schematic diagram of the Time Channel Attention Module (TCAM) in this invention.

[0018] Figure 3 This is a schematic diagram of the spatiotemporal self-similar module STSM in this invention. Detailed Implementation

[0019] Specific embodiments of the present invention are given below. These specific embodiments are only used to further illustrate the present invention and do not limit the scope of protection of this application.

[0020] The following descriptions, to the extent that they can be implemented by those skilled in the art, illustrate the module connections, data transfer relationships, calculation steps, parameter selection, and training methods.

[0021] I. Definitions of Terms, Symbols, and Variables "Continuous sign language video" refers to a video sequence in which the start and end boundaries of each sign language word are not pre-marked; "Sign language word" is the sequence output unit of continuous sign language recognition; The "Temporal Channel Attention Module" refers to a module that models the changes of channels over time after spatial pooling. The "spatiotemporal self-similar module" refers to a module that calculates normalized feature relationships over multiple time offsets and local two-dimensional spatial displacements.

[0022] The above names will be used consistently throughout the text, and no English abbreviations will be used to replace the Chinese names.

[0023] Table 1. Definitions of Terms, Symbols, and Variables

[0024] A spatiotemporal self-similar continuous sign language recognition method includes the following steps: Step S1: Acquire continuous sign language video, resize and normalize the video frames, and perform one or more of random cropping, random horizontal flipping and temporal resampling for data augmentation during the training phase; input the video frames into a visual backbone network including multiple convolutional stages, with each stage connected sequentially, and each stage obtains intermediate visual features with batch B, channel C, time T, height H and width W dimensions respectively.

[0025] Step S2: Starting from the second stage of the visual backbone network, a temporal channel attention module (TCAM) is set after one or more stages in the second stage (Stage 2), third stage (Stage 3), and fourth stage (Stage 4). In the TCAM, global average pooling is performed on the intermediate visual features obtained in the corresponding stage in the spatial dimension to form a channel-temporal descriptor X. r The channel-time descriptor is extracted using multiple one-dimensional temporal convolutional branches with different temporal receptive fields (different branches have different padding and dilation coefficients, such as dilation coefficients of 1, 2, and 3). The results of each branch are weighted and fused using learnable coefficients, and then local cross-channel interaction is performed through channel attention to achieve dynamic recalibration of channel features. A channel-time attention map is then generated using the Sigmoid function. This attention map is then scaled using a first scaling factor. Residual modulation is performed on the intermediate visual features of the input to obtain the visual features enhanced in the temporal channel.

[0026] Step S3: A spatiotemporal self-similar module (STSM) is set after the temporal channel attention module (TCAM) in the third stage of the visual backbone network. First, the enhanced visual features (including target features and neighborhood features) along the channel dimension are normalized using the L2 norm. For each temporal location and each spatial anchor point, the normalized feature inner product is calculated in an extended neighborhood containing multiple relative temporal offsets and local spatial displacements, obtaining a dense spatiotemporal self-similar body that preserves the temporal offset dimension and the two-dimensional spatial displacement topology. Zero padding or equivalent boundary padding is used at the boundaries.

[0027] The local spatial displacement mesh of dense spatiotemporal self-similar bodies is flattened and formed into a compact embedding corresponding to each time offset through a fully connected layer. For each time offset, a 3D convolution that preserves the offset dimension is applied to aggregate the spatial context. The features of each offset are then concatenated along the channel dimension and fused through 3D convolution to obtain the visual features enhanced by the time channels. The relational motion feature Z has the same number of channels; the relational motion feature is scaled by the second scaling factor. Visual features enhanced by time channel The residuals are summed to obtain the fused feature Y. The first scaling factor and the second scaling factor are independent of each other and are set as learnable parameters.

[0028] Step S4: The fused feature Y is input into the fourth stage along with the output of the third stage. The output of the fourth stage is processed by the temporal channel attention module. The short-term dynamics are extracted using a one-dimensional temporal convolutional 1DCNN, the context representation is generated using a bidirectional recurrent sequence model BiLSTM, and the classifier outputs the probabilities of sign language word categories at each time position.

[0029] Step S5: During the training phase, a main connection temporal classification loss based on global prediction (main branch) is jointly adopted. Auxiliary branch connection temporal classification loss based on local prediction (auxiliary branch) and the self-distillation loss between the main branch and the auxiliary branch End-to-end optimization is performed; during the inference phase, the class probabilities output by the bidirectional cyclic sequence model BiLSTM are decoded to output a sign language word sequence that does not require pre-segmentation.

[0030] Example 1 In this embodiment, let the intermediate visual features output by the visual backbone network at a certain stage be: Where B is the batch size, C is the number of feature channels, T is the temporal length of the feature, and H and W represent the height and width of the feature map, respectively. The visual backbone network consists of multiple convolutional stages connected in sequence; each video frame first extracts spatial appearance features from the preceding convolutional stage, and then organizes them along the temporal dimension into five-dimensional features, which are the intermediate visual features. The temporal channel attention module is set at the output of the second, third, and fourth stages. The input of the temporal channel attention module is connected to the output of the corresponding convolutional stage, and its output is connected to the next convolutional stage or the spatiotemporal self-similarity module after the third stage.

[0031] In the temporal channel attention module, global average pooling is first performed on the input features (intermediate visual features) in the spatial dimension to obtain the channel-temporal descriptor. :

[0032] Among them, X c,t,h,w X represents the intermediate visual feature of the c-th channel, t-th time length, h-th height, and w-th width in a batch; r (c,t) represents the channel-time descriptor for the c-th channel and time length t; X r The input consists of multiple one-dimensional temporal convolutional branches with different kernel sizes or dilation rates. The outputs of each branch are fused using learnable coefficients. Then, local cross-channel interaction is performed through channel attention to achieve dynamic recalibration of channel features. The channel-temporal attention map A is generated using the Sigmoid function. (2) in, This represents the i-th one-dimensional temporal convolution branch. f represents the learnable coefficients corresponding to the i-th one-dimensional temporal convolution branch. ch This represents channel attention, where m represents the number of branches in the one-dimensional temporal convolution. This represents the Sigmoid function.

[0033] Broadcast the channel-time attention map A to the spatial dimension and use the first scaling factor. Residual modulation is performed to obtain the visual features X after temporal channel enhancement. att : (3) The spatiotemporal self-similarity module is placed after the temporal channel attention module in the third-stage output connection. It performs L2 normalization on the enhanced visual features along the channel dimension, resulting in the normalized features. : (4) Where k represents the k-th channel, Represents the normalized stable term. Represents the spatial anchor point of the c-th channel. Visual features enhanced by the time channel.

[0034] For each target time t and spatial anchor point In the case of relative time offset Local two-dimensional spatial displacement Calculate the normalized inner product of the extended neighborhood using the normalized features. :

[0035] in, , , L represents the size of the time neighborhood, and U and V represent the sizes of the horizontal and vertical displacement windows, respectively. Indicates rounding down; This represents the normalized features within the extended neighborhood of the c-channel; This indicates the spatial anchor point at the target time t in channel c. The normalized features.

[0036] Then, within the entire horizontal and vertical displacement window and within the entire time neighborhood, the normalized inner product of all extended neighborhoods. Formation of dense spatiotemporal self-similar bodies When the index crosses the boundary of the video or feature map, zero padding, boundary value duplication, or other equivalent boundary processing is used, but the boundary processing is consistent within the same embodiment.

[0037] For each relative time offset, The local spatial displacement mesh is flattened and projected into a compact spatial relationship embedding (displacement mesh projection) through a fully connected linear layer. Then, multiple 3D convolutions with a temporal depth of 1 (including convolutional layers Conv, ReLU function, and batch normalization BN) are used to aggregate the spatial context within each offset to avoid destroying the offset structure at this stage. The L temporal offset features are then concatenated along the channel dimension, and the relational motion feature Z, consistent with the number of input channels, is obtained through multiple cascaded 3D convolutions. Finally, a second scaling factor is applied. Perform residual fusion: (6) In this embodiment, two types of 3D convolutions are used: one type, denoted as A1, is used for offset-independent spatial aggregation, and the other type, denoted as A2, is used for multi-offset stitching and fusion. Both structures adopt the Conv+ReLU+BN form. The number of 3D convolutions A1 used for offset-independent spatial aggregation can be arbitrarily adjusted, and the number of channels is 128. In this embodiment, three A1 convolutions are used. The number of 3D convolutions A2 used for multi-offset stitching and fusion is 1. A Shape operation is set before Conv to facilitate subsequent residual alignment. The number of channels of Conv is 256, and the output of A2 is used for residual injection.

[0038] The fused feature Y is input to the fourth stage. The output of the fourth stage, after being processed by a temporal attention module, serves as the input to both the auxiliary and main branches. The auxiliary branch (1DCNN) is sequentially connected to a one-dimensional temporal convolutional layer and a temporal pooling layer; the main branch is sequentially connected to a bidirectional recurrent sequence model (BiLSTM) and a classifier. The output of the main branch is denoted as... The auxiliary branch output is denoted as Main connection time classification loss Temporal classification loss with auxiliary branches They are respectively: (7) (8) Where y represents the true label sequence, and p represents the total probability of obtaining the true label sequence y.

[0039] Self-distillation losses in main and auxiliary branches for: (9) in, D represents the distillation temperature; KL This represents the KL divergence.

[0040] End-to-end training adopts the following joint objective : (10) in, This represents the loss weight.

[0041] During inference, no sign language word boundary labeling is required. By using connection-series classification decoding (a type of classifier) ​​to fold and remove blanks from the category probabilities at each time position, a continuous sign language word sequence is obtained.

[0042] Example 2 In one feasible embodiment, an 18-layer residual network pre-trained with image data is used as the visual backbone network, comprising four stages; the temporal channel attention module is placed after the second, third, and fourth stages; and the spatiotemporal self-similarity module is placed after the temporal channel attention module in the third stage. The temporal neighborhood size L is set to 7, and the spatial displacement windows U and V are both set to 5, with a normalized stabilization term. Pick The input frame is scaled to 256×256 pixels, and randomly cropped to 224×224 pixels during training, with random horizontal flipping and temporal resampling possible. Center pruning is used during inference. The bidirectional recurrent sequence model employs a two-layer bidirectional long short-term memory network with a hidden dimension of 1024. The optimizer can be Adam, with the initial learning rate and weight decay determined based on the training set. The specific values ​​mentioned above are implementation examples and do not constitute a limitation on the scope of protection.

[0043] Word error rate Evaluation of recognition performance. In validation on the publicly available continuous sign language datasets PHOENIX2014, PHOENIX2014T, and CSL-Daily, the combination of a spatiotemporal self-similarity module and a temporal channel attention module achieved consistent improvement over the baseline without both modules.

[0044] Table 2 Validation results on public datasets

[0045] In this invention, the number of one-dimensional temporal convolutional branches, receptive field, attention generation method, visual backbone network, bidirectional recurrent sequence model, and decoding algorithm can all be replaced while maintaining the above technical concept; similarity calculation in the spatiotemporal self-similarity module can use normalized inner product, cosine similarity, or equivalent relational function with learnable metric; the spatial window can be rectangular, sparsely sampled, or multi-scale window; the temporal offset can be continuous symmetrical offset or adaptively selected according to action speed.

[0046] This invention, based on continuous video features, calculates normalized relationships within an extended neighborhood composed of multiple time offsets and two-dimensional spatial displacements, centered on each spatiotemporal anchor point, forming a dense spatiotemporal self-similar body that preserves both the time offset dimension and the spatial displacement topology. By injecting the residuals of relational motion features into the enhanced visual features of the temporal channel through learnable scaling factors, end-to-end co-optimization of appearance representation and relational motion representation is achieved.

[0047] This invention is based on a channel-temporal descriptor after spatial pooling. It generates channel-temporal attention by fusing multiple branches with different temporal receptive fields, learnable branches, and local cross-channel interactions, and performs residual enhancement on visual features.

[0048] This invention provides a continuous sign language recognition model that outputs a continuous sequence of sign language words without requiring sign language action boundary annotation or explicit displacement estimation. A joint objective is constructed using the main connection temporal classification loss, the auxiliary connection temporal classification loss, and the main-auxiliary branch self-distillation loss to constrain the consistency between the local temporal representation and the global sequence representation.

[0049] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A spatiotemporal self-similar continuous sign language recognition method, characterized in that, The method includes the following: Acquire continuous sign language video, and resize and normalize the video frames; A continuous sign language recognition model is constructed, which includes a visual backbone network, an auxiliary branch for local prediction, and a main branch for global prediction. The visual backbone network consists of multiple convolutional stages connected in series. Starting from the second stage of the visual backbone network, a branch is led out after each subsequent stage to set a temporal channel attention module. In the third stage, a spatiotemporal self-similarity module is connected after the temporal channel attention module. The output of the spatiotemporal self-similarity module is used as an input to the next stage. The output of the last stage is processed by a temporal channel attention module and used as the input to the auxiliary branch and the main branch. Video frames are input into a visual backbone network that includes multiple convolutional stages. Each stage extracts intermediate visual features with dimensions of batch size (B), channel size (C), time (T), height (H), and width (W). In the temporal channel attention module, global average pooling is performed on the intermediate visual features of the corresponding stage in the spatial dimension to form a channel-temporal descriptor; multiple one-dimensional temporal convolutional branches with different temporal receptive fields are used to extract multi-scale temporal dependencies, and the results of each branch are weighted and fused with learnable coefficients. Then, a channel-temporal attention map is generated through local cross-channel interaction and the Sigmoid function; the attention map is used to perform residual modulation on the intermediate visual features of the corresponding stage to obtain the temporally enhanced visual features. In the spatiotemporal self-similarity module, firstly, the visual features enhanced by the time channels are normalized to L2 along the channel dimension; for each time position and each spatial anchor point, the normalized feature inner product is calculated in an extended neighborhood containing multiple relative time offsets and local spatial displacements to obtain a dense spatiotemporal self-similarity body that retains the topology of the time offset dimension and the two-dimensional spatial displacement dimension; the local spatial displacement mesh of the dense spatiotemporal self-similarity body is flattened and embedded into a compact spatial relationship through a fully connected layer; then, it is fused by three-dimensional convolution to obtain a relational motion feature with the same number of channels as the visual features enhanced by the time channels; the relational motion feature is added to the residual of the visual features enhanced by the time channels according to a learnable scaling factor to obtain the fused feature; The continuous sign language recognition model is trained using a continuous sign language dataset. The trained continuous sign language recognition model outputs a sequence of sign language words that does not require pre-segmentation.

2. The method according to claim 1, characterized in that, The auxiliary branch employs one-dimensional temporal convolution, and the output of the one-dimensional temporal convolution is sequentially connected to the bidirectional recurrent sequence model and the classifier to form the main branch. During the training phase, end-to-end optimization is jointly performed using the main connection temporal classification loss, the auxiliary branch connection temporal classification loss, and the self-distillation loss between the main branch and the auxiliary branch. During the inference phase, the class probabilities output by the bidirectional recurrent sequence model are decoded to output a sign language word sequence that does not require pre-segmentation.

3. The method according to claim 1, characterized in that, The process of using this attention map to perform residual modulation on the intermediate visual features X of the corresponding stage is as follows: Broadcast the channel-time attention map A to the spatial dimension and use the first scaling factor. Residual modulation is performed to obtain visual features enhanced by the temporal channel. The specific formula is expressed as follows: (3)。 4. The method according to claim 1, characterized in that, The process for determining the dense spatiotemporal self-similar body is as follows: Set the time neighborhood size L, and set the horizontal and vertical displacement window sizes U and V. For each target time and space anchor points In the case of relative time offset Local two-dimensional spatial displacement Calculate the normalized inner product of the extended neighborhood using the normalized features. : ， in, , , ; Indicates rounding down; This represents the normalized features within the extended neighborhood of the c-channel; This indicates the spatial anchor point at the target time t in channel c. The normalized features; Normalized inner products obtained over the entire horizontal and vertical displacement window and over the entire time neighborhood are all within the extended neighborhood. Forming dense spacetime self-similar bodies .

5. The method according to claim 1, characterized in that, The number of 3D convolutions is multiple, divided into two categories: one type of 3D convolution is used for offset-independent spatial aggregation, denoted as A1, and the other type of 3D convolution is used for multi-offset stitching and fusion, denoted as A2. Both types of structures adopt the Conv+ReLU+BN form. The number of 3D convolutions A1 used for offset-independent spatial aggregation can be arbitrarily adjusted; the number of 3D convolutions A2 used for multi-offset stitching and fusion is 1, and the output of A2 is used for residual injection.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can perform the steps of the method described in any one of claims 1-5.

Images