Moving image decoding method based on global and local layered space-time attention network

Abstract

The invention discloses a moving image decoding method based on a global and local layered space-time attention network, and belongs to the technical field of data processing, and the method comprises the following steps: S1, collecting an original EEG signal, and carrying out the preprocessing of the original EEG signal; s2, partitioning the preprocessed original EEG signal to obtain a plurality of local domains; and S3, inputting the plurality of local domains into the global local layered space-time attention network to realize image decoding of the EEG signal. The moving image decoding accuracy has obvious advantages in an independent theme experiment, the standard deviation is at a low level, and the global and local layered space-time attention network is effective and stable in EEG decoding.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and particularly relates to a method for decoding motion images based on a global-local hierarchical spatio-temporal attention network. Background Art

[0002] The decoding of motion images (MI) based on electroencephalogram (EEG) has extensive potential applications as an important neuroengineering technology, and can provide new possibilities in fields such as brain-computer interfaces (BCIs) and rehabilitation medicine. Existing research has considered capturing global discriminant information in different dimensions of EEG signals.

[0003] The essence of a brain-computer interface is to interpret brain activity and convert it into commands to control external devices. Brain activity patterns can be recorded through various technologies, among which EEG signals are captured in a non-invasive form and represent the activities of cortical neurons. Due to its advantages such as low cost, low risk, and high temporal resolution, EEG is widely used in applications such as UAV maneuvering and stroke rehabilitation. A motion image is an activity of thinking about moving a part of the human torso without physical movement, and can be used to develop a brain-computer interface system. Therefore, the effective decoding of EEG patterns plays a crucial role in the application of motion images.

[0004] However, most previous studies have only learned the global patterns of the brain and ignored the influence of the collaborative cooperation between different brain functional regions. In addition, neuroscience has shown that there is complex information interaction between different brain functional regions. Therefore, by combining information interaction with deep learning, an efficient network has been developed that can greatly improve the decoding performance of motion images. Summary of the Invention

[0005] In order to solve the above problems, the present invention proposes a method for decoding motion images based on a global-local hierarchical spatio-temporal attention network.

[0006] The technical solution of the present invention is: A method for decoding motion images based on a global-local hierarchical spatio-temporal attention network includes the following steps:

[0007] S1. Collect the original EEG signal and preprocess the original EEG signal;

[0008] S2. Partition the preprocessed original EEG signal to obtain a number of local domains;

[0009] S3. Input the number of local domains into the global-local hierarchical spatio-temporal attention network to implement image decoding of the EEG signal.

[0010] Further, S3 includes the following sub-steps:

[0011] S31. Input several local domains into the hierarchical local spatio-temporal fine-grained encoder of the global-local hierarchical spatio-temporal attention network to extract fine-grained features;

[0012] S32. Use the high-order attention calibrator of the global-local hierarchical spatio-temporal attention network to strengthen the fine-grained features to obtain high-order spatio-temporal features;

[0013] S33. Use the kernel-level fusion decoder of the global-local hierarchical spatio-temporal attention network to fuse the high-order spatio-temporal features and update the global-local hierarchical spatio-temporal attention network;

[0014] S34. Use the updated global-local hierarchical spatio-temporal attention network to implement image decoding of EEG signals.

[0015] Further, S31 includes the following sub-steps:

[0016] S311. Based on several local domains, extract the first preliminary spatial representation and the second preliminary spatial representation;

[0017] S312. Normalize the first preliminary spatial representation and the second preliminary spatial representation to obtain a unified spatial representation;

[0018] S313. Extract the time preliminary representation according to several local domains;

[0019] S314. Use the unified spatial representation and the time preliminary representation as fine-grained features.

[0020] Further, in S311, the expression of the first preliminary spatial representation PS1 is:

[0021]

[0022] In the formula, C represents the number of channels, T represents the time step, B represents the bias vector, w r represents the weight of the sub-region, x r represents the input of the current time step, b r represents the bias vector of the sub-region, C r represents the number of channels of the sub-region;

[0023] In S311, the expression of the second preliminary spatial representation PS2 is:

[0024]

[0025] In S312, the expression of the unified spatial representation US is:

[0026]

[0027] Wherein, US1 represents the spatial representation of the first local domain, US2 represents the spatial representation of the second local domain, and US r represents the spatial representation of the r-th local domain, R represents the ReLU activation function, and r represents the local domain number;

[0028] Furthermore, in S313, the expression of the preliminary time representation PT is:

[0029]

[0030] Wherein, PT1 represents the dependency of the first local domain, PT2 represents the dependency of the second local domain, and PT r represents the dependency of the r-th local domain, r represents the local domain number, T represents the time step, B represents the bias vector, and C r represents the number of channels of the sub-region.

[0031] Furthermore, S32 includes the following sub-steps:

[0032] S321. Perform average pooling compression on the unified spatial representation;

[0033] S322. Construct a convolutional block for generating spatial attention calibration parameters based on the spatially represented average pooling compression;

[0034] S323. Generate a recalibrated high-order spatial representation based on the convolutional block for generating spatial attention calibration parameters;

[0035] S324. Extract dependency features of different durations and key information of different durations based on the preliminary time representation;

[0036] S325. Generate a recalibrated high-order time representation based on the dependency features of different durations and the key information of different durations;

[0037] S326. Use the recalibrated high-order spatial representation and the recalibrated high-order time representation as high-order spatio-temporal features.

[0038] Furthermore, in S321, the spatially represented average pooling compression Avg (B,T) has the following expression:

[0039]

[0040] Wherein, T represents the time step, B represents the bias vector, US r represents the spatial representation of the r-th local domain, C represents the number of channels, R represents the ReLU activation function, and r represents the local domain number;

[0041] S322. The convolutional block CP for generating spatial attention calibration parameters has the following expression:

[0042]

[0043] Wherein, σ sig represents the Sigmoid activation function, w r represents the weight of the sub-region, b r represents the bias vector of the sub-region;

[0044] In S323, the expression of the recalibrated high-order spatial representation RUS is:

[0045]

[0046] Wherein, RUS1 represents the high-order spatial representation after calibration of the first local domain, RUS2 represents the high-order spatial representation after calibration of the second local domain, RUS r represents the high-order spatial representation after calibration of the r-th local domain, C r represents the number of channels of the sub-region;

[0047] In S324, the dependent feature Q r with different durations is expressed as:

[0048]

[0049] Wherein, Linear represents the linear activation function, θ represents the linear layer parameter, US r represents the spatial representation of the r-th local domain, and Q represents the dependent feature;

[0050] In S324, the key information K r with different durations is expressed as:

[0051]

[0052] Wherein, K represents the key information;

[0053] In S325, the expression of the recalibrated high-order temporal representation RPT is:

[0054]

[0055] Wherein, RPT1 represents the high-order temporal representation after recalibration of the first local domain, RPT2 represents the high-order temporal representation after recalibration of the second local domain, RPT r represents the high-order temporal representation after recalibration of the r-th local domain.

[0056] Furthermore, S33 includes the following sub-steps:

[0057] S331. Generate a global high-order spatio-temporal representation based on the re-calibrated high-order spatial representation and the re-calibrated high-order temporal representation;

[0058] S332. Decode the global high-order spatio-temporal representation using a linear layer;

[0059] S333. Calculate the probability distribution of the decoded global high-order spatio-temporal representation;

[0060] S334. Calculate the cross-entropy loss function based on the probability distribution of the decoded global high-order spatio-temporal representation;

[0061] S335. Update the global local hierarchical spatio-temporal attention network using the cross-entropy loss function.

[0062] Further, in S331, the expression of the global high-order spatio-temporal representation GHR is:

[0063] GHR = ∑GHR r ∈R (10×C×B×T) ;

[0064]

[0065] where GHR r represents the global high-order spatio-temporal of the r-th local domain, R represents the ReLU activation function, r represents the local domain number, C represents the number of channels, T represents the time step, B represents the bias vector, RUS r represents the re-calibrated high-order spatial representation of the r-th local domain, C r represents the number of channels of the sub-region, RPT r represents the re-calibrated high-order temporal representation of the r-th local domain;

[0066] S332. The calculation formula of the decoded global high-order spatio-temporal representation Decode is:

[0067] Decode = Linear C×class

[0068] (Linear (5×C×B×T)×C (ReLU(Linear (10×C×B×T)×(5×C×B×T) (GHR; θ 1 )); θ 2 ); θ 3 );

[0069] where Linear represents the linear activation function, θ 1 represents the parameter of the first linear layer, θ 2 represents the parameter of the first linear layer, θ 3 represents the parameter of the first linear layer, class represents the number of sub-regions;

[0070] In S333, the probability distribution y of the decoded global high-order spatio-temporal representation pred is calculated as follows:

[0071] y pred = softmax(Decode) ∈ R class ;

[0072] In the formula, softmax(·) represents the activation function;

[0073] In S334, the expression of the cross-entropy loss function Loss is:

[0074]

[0075] In the formula, N represents the total number of samples, c represents the channel, represents the true label, represents the predicted label, and log(·) represents the logarithmic function.

[0076] The beneficial effects of the present invention are as follows:

[0077] (1) The present invention proposes a motion image decoding method based on a global-local hierarchical spatio-temporal attention network, which decodes motion images by considering the collaborative cooperation between local domains; strengthens fine-grained features through a spatio-temporal guided high-order attention calibrator, and hierarchically captures high-order spatio-temporal features from complex electroencephalogram signals; the present invention has achieved excellent results on public datasets and reveals the potential collaborative patterns between local regions under different motion image decoding tasks.

[0078] (2) The motion image decoding accuracy of the present invention has obvious advantages in independent subject experiments, and the standard deviation is at a low level. The global-local hierarchical spatio-temporal attention network of the present invention is effective and robust in EEG decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 is a flowchart of a motion image decoding method based on a global-local hierarchical spatio-temporal attention network;

[0080] Figure 2 is a schematic structural diagram of a global-local hierarchical spatio-temporal attention network. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0081] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0082] As Figure 1 shown, the present invention provides a motion image decoding method based on a global-local hierarchical spatio-temporal attention network, including the following steps:

[0083] S1. Collect the original EEG signals and preprocess the original EEG signals;

[0084] S2. Partition the preprocessed original EEG signals to obtain several local domains;

[0085] S3. Input the several local domains into the global-local hierarchical spatio-temporal attention network to implement image decoding of EEG signals.

[0086] As Figure 2 shown, the global-local hierarchical spatio-temporal attention network consists of a hierarchical local spatio-temporal fine-grained encoder (HLSTF), a spatio-temporal guided high-order attention calibrator (STG-HAC), and a kernel-level fusion decoder (KLF). First, a hierarchical local spatio-temporal fine-grained encoder is constructed to hierarchically learn the fine-grained features of each brain functional area. Then, a high-order attention calibrator is developed to strengthen the fine-grained features, so as to hierarchically capture high-order spatio-temporal features from complex EEG signals. Finally, by fusing these high-order spatio-temporal features at the kernel level, the influence of the potential cooperative cooperation between different brain functional areas on the classification effect is decoded.

[0087] In the embodiment of the present invention, S3 includes the following sub-steps:

[0088] S31. Input the several local domains into the hierarchical local spatio-temporal fine-grained encoder of the global-local hierarchical spatio-temporal attention network to extract fine-grained features;

[0089] S32. Use the high-order attention calibrator of the global-local hierarchical spatio-temporal attention network to strengthen the fine-grained features to obtain high-order spatio-temporal features;

[0090] S33. Use the kernel-level fusion decoder of the global-local hierarchical spatio-temporal attention network to fuse the high-order spatio-temporal features and update the global-local hierarchical spatio-temporal attention network;

[0091] S34. Use the updated global-local hierarchical spatio-temporal attention network to implement image decoding of EEG signals.

[0092] In the embodiment of the present invention, S31 includes the following sub-steps:

[0093] S311. Based on the several local domains, extract the first preliminary spatial representation and the second preliminary spatial representation;

[0094] S312. Normalize the first preliminary spatial representation and the second preliminary spatial representation to obtain a unified spatial representation;

[0095] S313. Extract the time preliminary representation according to the several local domains;

[0096] S314. Use the unified spatial representation and the time preliminary representation as the fine-grained features.

[0097] The present invention constructs a local time domain module to obtain time - fine - grained features of each local domain in a hierarchical manner, aiming to process sequence data by introducing a hierarchical structure. As the number of layers increases, the module can learn high - level abstract representations, capture the initial time representation of EEG signals, and thus abstract basic time semantic information. The calculation process is as follows:

[0098]

[0099] Due to the use of a bidirectional recurrent neural network, and represent forward and backward calculations respectively. PT k represents the result of the current hidden - layer calculation, and PT k-1 represents the result of the previous hidden - layer calculation. x r represents the input at the current time step, B represents the bias vector, T represents the time step, and || represents feature concatenation. g(·) and f(·) are both activation functions, and g(f(·)) represents a unidirectional calculation process. Specifically:

[0100]

[0101] In the formula, U represents the weight matrix of the current input , W represents the weight matrix of the previous value as the input of this value, represents the input of the previous step, and V represents the weight matrix of the output layer. Therefore, long - term dependencies and hierarchical structures in sequence data can be better captured.

[0102] In the embodiment of the present invention, in S311, the expression of the first preliminary spatial representation PS1 is:

[0103]

[0104] In the formula, C represents the number of channels, T represents the time step, B represents the bias vector, w r represents the weight of the sub - region, x r represents the input at the current time step, b r represents the bias vector of the sub - region, and C r represents the number of channels of the sub - region;

[0105] In S311, the expression of the second preliminary spatial representation PS2 is:

[0106]

[0107] In S312, the expression of the unified spatial representation US is:

[0108]

[0109] Wherein, US1 represents the spatial representation of the first local domain, US2 represents the spatial representation of the second local domain, and US r represents the spatial representation of the r-th local domain, R represents the ReLU activation function, and r represents the local domain number; the unified spatial representation can also be expressed as US = ReLU(BN(PS1 * PS2)), where BN represents the Batchnorm function.

[0110] In the embodiment of the present invention, in S313, the expression of the initial time representation PT is:

[0111]

[0112] Wherein, PT1 represents the dependency relationship of the first local domain, PT2 represents the dependency relationship of the second local domain, and PT r represents the dependency relationship of the r-th local domain, r represents the local domain number, T represents the time step, B represents the bias vector, and C r represents the number of channels of the sub-region.

[0113] In the embodiment of the present invention, S32 includes the following sub-steps:

[0114] S321. Perform average pooling compression on the unified spatial representation;

[0115] S322. Construct a convolutional block for generating spatial attention calibration parameters according to the spatially represented average pooling compression;

[0116] S323. Generate a recalibrated high-order spatial representation according to the convolutional block for generating spatial attention calibration parameters;

[0117] S324. Extract dependency features of different durations and key information of different durations according to the initial time representation;

[0118] S325. Generate a recalibrated high-order time representation according to the dependency features of different durations and the key information of different durations;

[0119] S326. Use the recalibrated high-order spatial representation and the recalibrated high-order time representation as high-order spatio-temporal features.

[0120] In the embodiment of the present invention, in S321, the spatially represented average pooling compression Avg (B,T) has the following expression:

[0121]

[0122] Wherein, T represents the time step, B represents the bias vector, and US rIt represents the spatial representation of the r-th local domain, C represents the number of channels, R represents the ReLU activation function, and r represents the local domain number;

[0123] In S322, the expression of the convolutional block CP for generating the spatial attention calibration parameter is:

[0124]

[0125] In the formula, σ sig represents the Sigmoid activation function, w r represents the weight of the sub-region, and b r represents the bias vector of the sub-region;

[0126] In S323, the expression of the recalibrated high-order spatial representation RUS is:

[0127]

[0128] In the formula, RUS1 represents the high-order spatial representation after calibration of the first local domain, RUS2 represents the high-order spatial representation after calibration of the second local domain, and RUS r represents the high-order spatial representation after calibration of the r-th local domain, and C r represents the number of channels of the sub-region; Multiply US by CP to obtain the recalibrated high-order spatial representation, RUS = US * CP ∈ R C×B×T .

[0129] In S324, in S324, the dependent feature Q r with different time durations is expressed as:

[0130]

[0131] In the formula, Linear represents the linear activation function, θ represents the linear layer parameter, and US r represents the spatial representation of the r-th local domain, and Q represents the dependent feature;

[0132] In S324, the key information K r with different time durations is expressed as:

[0133]

[0134] In the formula, K represents the key information;

[0135] Update the dependent feature with different time durations and the key information with different time durations to obtain the calibration parameter CP′, and its calculation formula is:

[0136] CP′ = Softmax (T) (Linear C×1 (Tanh(Q + K); θV )) ∈ R T×1 ;

[0137] Where θ V represents the parameters of this linear layer, Softmax (T) represents the calculation of the calibration parameters based on T, and Tanh(·) represents the activation function. Finally, PT and CP′ are multiplied to obtain the recalibrated high-order time representation.

[0138] In S325, the expression of the recalibrated high-order time representation RPT is:

[0139]

[0140] Where RPT1 represents the recalibrated high-order time representation of the first local domain, RPT2 represents the recalibrated high-order time representation of the second local domain, and RPT r represents the recalibrated high-order time representation of the rth local domain.

[0141] In the embodiment of the present invention, S33 includes the following sub-steps:

[0142] S331. Generate a global high-order spatio-temporal representation according to the recalibrated high-order spatial representation and the recalibrated high-order time representation;

[0143] S332. Decode the global high-order spatio-temporal representation using a linear layer;

[0144] S333. Calculate the probability distribution of the decoded global high-order spatio-temporal representation;

[0145] S334. Calculate the cross-entropy loss function according to the probability distribution of the decoded global high-order spatio-temporal representation;

[0146] S335. Update the global local hierarchical spatio-temporal attention network using the cross-entropy loss function.

[0147] In the embodiment of the present invention, in S331, the expression of the global high-order spatio-temporal representation GHR is:

[0148] GHR = ∑GHR r ∈ R (10×C×B×T) ;

[0149]

[0150] Where GHR r represents the global high-order spatio-temporal of the rth local domain, R represents the ReLU activation function, r represents the local domain number, C represents the number of channels, T represents the time step, B represents the bias vector, and RUS r represents the recalibrated high-order spatial representation of the rth local domain, Cr Indicates the number of channels in the sub-region, RPT r Indicates the high-order time representation after recalibration of the r-th local domain;

[0151] In S332, the calculation formula for the decoded global high-order spatio-temporal representation Decode is:

[0152] Decode = Linear C×class

[0153] (Linear (5×C×B×T)×C (ReLU(Linear (10×C×B×T)×(5×C×B×T) (GHR; θ 1 )); θ 2 )); θ 3 ));

[0154] In the formula, Linear represents the linear activation function, θ 1 represents the parameter of the first linear layer, θ 2 represents the parameter of the first linear layer, θ 3 represents the parameter of the first linear layer, class represents the number of sub-regions;

[0155] In S333, the probability distribution y of the decoded global high-order spatio-temporal representation pred has the following calculation formula:

[0156] y pred = softmax(Decode) ∈ R class ;

[0157] In the formula, softmax(·) represents the activation function;

[0158] In S334, the expression of the cross-entropy loss function Loss is:

[0159]

[0160] In the formula, N represents the total number of samples, c represents the channel, represents the true label, represents the predicted label, log(·) represents the logarithmic function.

[0161] The following is illustrated with specific embodiments.

[0162] Dataset Introduction: The Physionet dataset consists of raw time series EEG data from 109 volunteers. The data from the first 90 subjects with a sampling frequency of 160 Hz is used. Each subject needs to complete the following four tasks: (1) opening and clenching the left or right fist; (2) imagining opening and clenching the left or right fist; (3) opening and clenching the fist or foot; (4) imagining opening and clenching the fist or foot. Therefore, MI-EEG data of 90 subjects with four classifications can be obtained from this dataset. The Cho dataset consists of raw time series EEG data from 52 subjects. Each subject participated in the same experimental tasks of left and right hand motor imagery respectively. Therefore, binary classification MI-EEG data of 52 subjects can be obtained from this dataset.

[0163] Data Processing and Feature Calculation: The preprocessing of MI-EEG signals is a process of removing various types of noise and artifacts present in the signals before extracting relevant features. In the present invention, due to the Butterworth filter having a faster computational amount and less memory space requirement, it is selected to preprocess the MI-EEG signals. The complete signal is decomposed into five frequency bands: Delta (1 - 4 Hz), Theta (4 - 8 Hz), Alpha (8 - 12 Hz), Beta (12 - 30 Hz), and Gamma (30 - 50 Hz). Then, DE is also selected as an artificial feature.

[0164] The accuracy of the HSTA-HNet model has obvious advantages in independent subject experiments, and the standard deviation is at a low level, indicating that compared with other methods, the network of the present invention is effective and robust in EEG decoding. In addition, compared with the two classification experiments, the superior performance of HSTA-HNet in the four classification experiment is more prominent, which also indicates the great potential of the proposed network in multi-classification research applications.

[0165] In addition to comparing the classification accuracy and standard deviation, the computational cost also reflects the decoding performance of the model. Compared with other methods, HSTA-HNet shows superiority in terms of time consumption. The average test running times of HSTA-HNet on Cho and Physionet are 3.08 s and 3.28 s respectively, both lower than other models.

[0166] Those of ordinary skill in the art will realize that the embodiments described herein are to assist the reader in understanding the principles of the present invention. It should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on the technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention.

Claims

1. A motion image decoding method based on a global-local hierarchical spatio-temporal attention network, characterized in that, It includes the following steps: S1. Collect the original EEG signals and preprocess the original EEG signals; S2. Partition the preprocessed original EEG signals to obtain several local domains; S3. Input the several local domains into the global-local hierarchical spatio-temporal attention network to implement image decoding of the EEG signals.

2. The motion image decoding method based on the global-local hierarchical spatio-temporal attention network according to claim 1, wherein The S3 includes the following sub-steps: S31. Input the several local domains into the hierarchical local spatio-temporal fine-grained encoder of the global-local hierarchical spatio-temporal attention network to extract fine-grained features; S32. Use the high-order attention calibrator of the global-local hierarchical spatio-temporal attention network to strengthen the fine-grained features to obtain high-order spatio-temporal features; S33. Use the kernel-level fusion decoder of the global-local hierarchical spatio-temporal attention network to fuse the high-order spatio-temporal features and update the global-local hierarchical spatio-temporal attention network; S34. Use the updated global-local hierarchical spatio-temporal attention network to implement image decoding of the EEG signals.

3. The motion image decoding method based on the global-local hierarchical spatio-temporal attention network according to claim 2, wherein, The S31 includes the following sub-steps: S311. Based on the several local domains, extract the first preliminary spatial representation and the second preliminary spatial representation; S312. Normalize the first preliminary spatial representation and the second preliminary spatial representation to obtain a unified spatial representation; S313. According to the several local domains, extract the time preliminary representation; S314. Use the unified spatial representation and the time preliminary representation as the fine-grained features.

4. The motion image decoding method based on the global-local hierarchical spatio-temporal attention network according to claim 3, wherein, In the S311, the expression of the first preliminary spatial representation PS1 is: Wherein, C represents the number of channels, T represents the time step, B represents the bias vector, w r represents the weight of the sub-region, x r represents the input at the current time step, b r represents the bias vector of the sub-region, C r represents the number of channels of the sub-region; In the S311, the expression of the second preliminary spatial representation PS2 is: In the S312, the expression of the unified spatial representation US is: where, US1 represents the spatial representation of the first local domain, US2 represents the spatial representation of the second local domain, and US r represents the spatial representation of the r-th local domain, R represents the ReLU activation function, and r represents the local domain number.

5. The motion image decoding method based on the global-local hierarchical spatio-temporal attention network according to claim 3, wherein In the S313, the expression of the time preliminary representation PT is: Wherein, PT1 represents the dependency of the first local domain, PT2 represents the dependency of the second local domain, PT r represents the dependency of the r-th local domain, r represents the local domain number, T represents the time step, B represents the bias vector, C r represents the number of channels of the sub-region.

6. The motion image decoding method based on the global-local hierarchical spatio-temporal attention network according to claim 2, wherein The S32 includes the following sub-steps: S321. Perform average pooling compression on the unified spatial representation; S322. According to the spatially averaged-pooled representation, construct a convolutional block for generating spatial attention calibration parameters; S323. According to the convolutional block for generating spatial attention calibration parameters, generate a recalibrated high-order spatial representation; S324. According to the time preliminary representation, extract the dependent features of different durations and the key information of different durations; S325. According to the dependent features of different durations and the key information of different durations, generate a recalibrated high-order time representation; S326. Use the recalibrated high-order spatial representation and the recalibrated high-order time representation as the high-order spatio-temporal features.

7. The motion image decoding method based on the global-local hierarchical spatio-temporal attention network according to claim 6, wherein In the S321, the spatial representation Avg after average pooling compression (B,T) has the following expression: where T represents the time step, B represents the bias vector, US r represents the spatial representation of the r-th local domain, C represents the number of channels, R represents the ReLU activation function, and r represents the local domain number; In the S322, the expression of the convolutional block CP for generating spatial attention calibration parameters is: where, σ sig represents the Sigmoid activation function, w r represents the weight of the sub-region, b r represents the bias vector of the sub-region; In the S323, the expression of the recalibrated high-order spatial representation RUS is: Wherein, RUS1 represents the high-order spatial representation after calibration of the first local domain, RUS2 represents the high-order spatial representation after calibration of the second local domain, and RUS r represents the high-order spatial representation after calibration of the r-th local domain, and C r represents the number of channels of the sub-region; In the S324, the dependency feature Q with different durations r has the following expression: wherein, Linear represents a linear activation function, θ represents the parameters of the linear layer, and US r represents the spatial representation of the r-th local domain, and Q represents the dependent feature; In the S324, the key information K with different time lengths r has the following expression: In the formula, K represents the key information; In the S325, the expression of the recalibrated high-order time representation RPT is: Wherein, RPT1 represents the high-order time representation after recalibration of the first local domain, RPT2 represents the high-order time representation after recalibration of the second local domain, and RPT r represents the high-order time representation after recalibration of the r-th local domain.

8. The motion image decoding method based on the global-local hierarchical spatio-temporal attention network according to claim 2, wherein, The S33 includes the following sub-steps: S331. According to the recalibrated high-order spatial representation and the recalibrated high-order time representation, generate a global high-order spatio-temporal representation; S332. Use a linear layer to decode the global high-order spatio-temporal representation; S333. Calculate the probability distribution of the decoded global high-order spatio-temporal representation; S334. According to the probability distribution of the decoded global high-order spatio-temporal representation, calculate the cross-entropy loss function; S335. Update the global local hierarchical spatio-temporal attention network using the cross-entropy loss function.

9. The motion image decoding method based on the global-local hierarchical spatio-temporal attention network according to claim 8, wherein, In S331, the expression of the global high-order spatio-temporal representation GHR is: GHR = ∑GHR r ∈ R (10×C×B×T) ; where GHR r represents the global high-order spatio-temporal of the r-th local domain, R represents the ReLU activation function, r represents the local domain number, C represents the number of channels, T represents the time step, B represents the bias vector, and RUS r represents the calibrated high-order spatial representation of the r-th local domain, C represents the number of channels of the sub-region, and RPT r represents the recalibrated high-order time representation of the r-th local domain; In S332, the calculation formula of the decoded global high-order spatio-temporal representation Decode is: Decode=Linear C×class (Linear (5×C×B×T)×C (ReLU(Linear (10×C×B×T)×(5×C×B×T) (GHR; θ 1 )); θ 2 ); θ 3 ); where Linear represents the linear activation function, θ 1 represents the parameters of the first linear layer, θ 2 represents the parameters of the first linear layer, θ 3 represents the parameters of the first linear layer, and class represents the number of sub-regions; In S333, the probability distribution y of the decoded global high-order spatio-temporal representation pred is calculated as follows: y pred = softmax(Decode) ∈ R class ; In the formula, softmax(·) represents the activation function; In S334, the expression of the cross-entropy loss function Loss is: Where N represents the total number of samples, c represents the channel, represents the true label, represents the predicted label, and log(·) represents the logarithmic function.

Images