An ultrasound video recognition method and system based on diffusion attention

Abstract

The application discloses an ultrasonic video recognition method and system based on diffusion attention. The method comprises the following steps: acquiring an ultrasonic video; inputting the ultrasonic video into a trained recognition model to obtain a classification prediction result; wherein the recognition model uses multiple stages to obtain the overall representation of the ultrasonic video, each stage comprises a space-time block representation module, a diffusion window attention module and a space-time block merging module, the space-time block representation module is used for space-time division of the input ultrasonic video to obtain a discrete space-time block set; the diffusion window attention module establishes the distance dependence relationship between the space-time blocks by alternately using a fixed window attention mechanism and a diffusion window attention mechanism; and the space-time block merging module is used for down-sampling the features output by the diffusion window attention module. The application designs a diffusion window attention mechanism, which can enhance the perception field of the features, thereby quickly obtaining the overall information of the video without increasing the network complexity.

Description

Technical Field

[0001] The present invention relates to the field of video recognition technology, and more particularly to an ultrasonic video recognition method and system based on diffuse attention. Background Art

[0002] Processing clinical medical data, particularly ultrasound video, is one of the challenges of computer-assisted medical data analysis. Compared to traditional X-rays or CT scans, ultrasound video imaging is a fast, convenient, and radiation-free imaging technology, making ultrasound video analysis widely used in large-scale clinical diagnosis. However, ultrasound images and videos typically have low spatial resolution and are subject to noise interference such as artifacts and speckle noise. This can hinder on-site emergency diagnosis by medical personnel, prolonging patient diagnosis time and even leading to misdiagnosis.

[0003] In recent years, research on ultrasound video recognition based on deep learning has made significant progress. Currently, most deep learning-based video recognition methods can be categorized into three categories based on the level of processing: frame-based neural network (CNN) fusion prediction methods, video stream-based 3D convolutional neural network (3D convolutional neural network) methods, and Transformer-based methods. In frame-based neural network fusion prediction methods, a video is decomposed into a series of frames, each of which is then subjected to feature extraction using a pretrained CNN model. The extracted features are then fed into a fusion model, such as a recurrent neural network (RNN) or a long short-term memory (LSTM) network, to build an understanding and association of the video content. Ultimately, the fusion model uses the learned features for prediction and classification of the video content. Video stream-based 3D convolutional neural network (3D convolutional neural network) methods process the entire video stream more directly. This approach directly learns spatiotemporal features in the video by applying a 3D convolutional neural network in both the temporal and spatial dimensions. Compared to frame-based methods, video stream-based methods can better capture dynamic information and temporal relationships in the video, thereby improving the accuracy and efficiency of video recognition. The Transformer-based approach is a relatively new video recognition method in which video sequences are converted into sequential data and the spatiotemporal relationships in the video sequences are modeled through a multi-layer self-attention mechanism. This approach has good parallelism and scalability, and can effectively handle long-term dependencies.

[0004] In the prior art, the research scheme proposed by Born et al. (BORN J, WIEDEMANN N, COSSIO M, et al. Erratum: Born et al. Accelerating Detection of Lung Pathologies with Explainable Ultrasound Image Analysis. Appl. Sci. 2021, 11, 672 [J / OL]. Applied Sciences, 2022, 12 (8): 3869. DOI: 10.3390 / app12083869.) uses the VGG-16 classifier to obtain classification results for each frame of the video, and finally uses a weighted average method to fuse the results of each frame to obtain the classification results of the video. In this scheme, ultrasound video recognition uses a two-dimensional convolutional neural network to extract frame information, first performs frame-level prediction, and finally summarizes the prediction results of each frame and weights them to obtain the video-level prediction results.

[0005] Analysis reveals that the biggest challenge with existing frame-based ultrasound video processing methods is the difficulty in fusing video frame information into the overall video. As three-dimensional data, video contains not only scene information but also motion information. Frame-based processing methods struggle to capture temporal motion information through information fusion. Furthermore, videos often contain a vast number of frames, making frame-by-frame processing inefficient and difficult to apply to real-world medical scenarios. Existing classification and prediction methods based on video streams also need improvement in terms of computational efficiency and accuracy. Summary of the Invention

[0006] The purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide an ultrasonic video recognition method and system based on diffuse attention.

[0007] According to a first aspect of the present invention, a method for ultrasonic video recognition based on diffuse attention is provided. The method comprises the following steps:

[0008] Obtain ultrasound video;

[0009] Inputting the ultrasound video into a trained recognition model to obtain a classification prediction result;

[0010] The recognition model uses multiple stages to obtain the overall representation of the ultrasound video. Each stage includes a spatiotemporal block representation module, a diffuse window attention module, and a spatiotemporal block merging module. The spatiotemporal block representation module is used to perform spatiotemporal division on the input ultrasound video to obtain a discrete set of spatiotemporal blocks. The diffuse window attention module establishes a distance dependency between spatiotemporal blocks by alternating between a fixed window attention mechanism and a diffuse window attention mechanism. The spatiotemporal block merging module is used to downsample the features output by the diffuse window attention module.

[0011] Among them, the fixed window attention mechanism cuts the spatiotemporal matrix corresponding to the spatiotemporal block based on the set fixed window to obtain a local fixed window, and performs position encoding processing on the tokens in the local fixed window to convert them into a token sequence, and then uses the attention mechanism to interact with the information between tokens and establish dependency relationships between different tokens; the diffusion window attention mechanism interacts information from different local fixed windows in a diffusion window that is larger than the fixed window.

[0012] According to a second aspect of the present invention, there is provided an ultrasonic video recognition system based on diffuse attention. The system comprises:

[0013] Video acquisition module: used to acquire ultrasound video;

[0014] Classification prediction unit: used for inputting the ultrasound video into the trained recognition model to obtain a classification prediction result;

[0015] The recognition model uses multiple stages to obtain the overall representation of the ultrasound video. Each stage includes a spatiotemporal block representation module, a diffuse window attention module, and a spatiotemporal block merging module. The spatiotemporal block representation module is used to perform spatiotemporal division on the input ultrasound video to obtain a discrete set of spatiotemporal blocks. The diffuse window attention module establishes a distance dependency between spatiotemporal blocks by alternating between a fixed window attention mechanism and a diffuse window attention mechanism. The spatiotemporal block merging module is used to downsample the features output by the diffuse window attention module.

[0016] Among them, the fixed window attention mechanism cuts the spatiotemporal matrix corresponding to the spatiotemporal block based on the set fixed window to obtain a local fixed window, and performs position encoding processing on the tokens in the local fixed window to convert them into a token sequence, and then uses the attention mechanism to interact with the information between tokens and establish dependency relationships between different tokens; the diffusion window attention mechanism interacts information from different local fixed windows in a diffusion window that is larger than the fixed window.

[0017] Compared with the existing technology, the advantage of the present invention is that it combines diffuse window attention and fixed window attention to learn the global representation of the video. Compared with the existing frame-by-frame convolutional network method, the present invention crops the three-dimensional video into three-dimensional spatiotemporal blocks and learns the long-distance dependencies between these blocks by window division to establish a global representation of the video. In addition, the existing neural network processing method based on the entire video requires the model to be expanded in breadth, which increases the computational cost and also brings challenges to the learning and training of the model. Compared with the existing method of directly establishing global attention for all spatiotemporal blocks of the video, the present invention alternately uses fixed window attention and diffuse window attention modules. While reducing the amount of computation, it can still obtain an accurate overall representation of the video, thereby improving the accuracy and efficiency of ultrasound video recognition.

[0018] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 is a flowchart of an ultrasonic video recognition method based on diffuse attention according to an embodiment of the present invention;

[0021] Figure 2 is a framework diagram of an ultrasound video recognition model based on diffuse attention according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of spatiotemporal block segmentation according to an embodiment of the present invention;

[0023] Figure 4 1 is a schematic cross-sectional view of an improved local window and a diffusion cavity window according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0025] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0026] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0027] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0028] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0029] In general, this paper combines the advantages of high efficiency and low computational complexity of dilated convolution and proposes a diffuse window attention mechanism. On the basis of fixed window division, the idea of ​​dilated convolution kernel is used to design a diffuse window attention mechanism, which enables long-distance dependencies to be established between fixed windows. In this way, a good overall representation of the ultrasound video can be obtained while reducing the computational complexity.

[0030] See also Figure 1 As shown, the provided ultrasound video recognition method based on diffuse attention includes the following steps:

[0031] Step S110 : constructing an ultrasound video recognition model, which obtains the overall representation of the ultrasound video by alternately using fixed window attention and diffuse window attention.

[0032] Combine Figure 2 As shown in Figure 2, the ultrasound video recognition model based on diffuse attention can include multiple video information processing stages. Figure 2 The diagram shows four stages (marked as stage 1 to stage 4). Each stage contains three modules, namely the spatiotemporal block representation module (marked as patch embedding), the diffusion window attention module (marked as Dwin TransformerBlock) and the spatiotemporal block merging module (marked as Patch merging). First, the three-dimensional ultrasound video is divided into time and space to obtain a discrete set of spatiotemporal blocks. The information representation of the spatiotemporal blocks is obtained through a three-dimensional convolutional neural network. The spatiotemporal blocks are then divided into sets according to the designed fixed windows and sliding windows. The long-distance dependency of the spatiotemporal blocks in the set is established through the attention module. The high-spatiotemporal resolution information is then compressed to a low resolution (i.e., down-sampled) through the spatiotemporal block merging module. Through the multi-scale processing of these four stages, the overall representation of the ultrasound video is finally obtained.

[0033] For example, for video data of size T*H*W*3 (T, H, W represent time dimension, height and width respectively), the spatiotemporal block representation module divides the space and time according to the size of t*h*w*3 to obtain multiple spatiotemporal matrices of T / t, H / h, W / w. Since the spatiotemporal features are consistent and not affected by the specific spatiotemporal positions, a unified three-dimensional convolution operator can be used to obtain the spatiotemporal representation corresponding to each token. Figure 3 This is a schematic diagram of a spatiotemporal block. For example, for 8 frames in a video, each frame is divided into 8×8=64 grids. Assuming that the pixels of each grid are 4×4, when the patch size is (1, 4, 4), each small grid is a patch; when the patch size is (2, 4, 4), the grids at the same position in every two adjacent frames form a patch.

[0034] In one embodiment, a diffuse window attention module is designed based on the Video Swin Transformer model (LIU Z, NING J, CAO Y, et al. Video Swin Transformer [C / OL] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 3202-3211 [2023-10-06]). This module uses a fixed window attention mechanism and a diffuse window attention mechanism to improve the receptive field of view of the token. Existing video swin transformer blocks use attention to fuse features within the same window.

[0035] In the fixed window attention mechanism, the spatiotemporal matrix is ​​directly cut to obtain a local fixed window, and the tokens in the local fixed window are positionally encoded to convert them into a token sequence. The attention mechanism is then used to interact with the information between tokens and establish dependencies between different tokens.

[0036] For the diffusion window attention mechanism, influenced by the dilated convolution operator, information from different local windows is exchanged in a larger diffusion window, see Figure 4 The cross-section of the local window and the diffusion hole window is shown. By alternating between the fixed window attention mechanism and the diffusion window attention mechanism, the model ultimately obtains the global information of the video at a lower cost.

[0037] For example, the diffuse window attention mechanism uses a diffusion factor to increase the actual spatiotemporal interval of elements in the window, expanding the spatiotemporal range of the window while maintaining the number of tokens in the window. In the window attention mechanism, the establishment of dependencies between tokens expands the receptive field of a token to the union of the receptive fields of all tokens in the window. Alternating between fixed window attention and diffuse window attention ensures that each token used in calculating window attention comes from a different window in the previous layer, with a different receptive field. This continuously expands the receptive field of a token to a global range, capturing global information about the video.

[0038] Step S120: pre-training the recognition model using the set loss function.

[0039] Training datasets can be open-source datasets containing videos and annotations, such as datasets with video frames and action categories. Model training can use a variety of loss functions. For example, for video classification tasks, cross-entropy loss functions and mean squared error loss functions can be used. During training, model hyperparameters such as the learning rate and batch size can be adjusted to achieve optimal model performance.

[0040] Step S130: Acquire the target ultrasound video and perform classification prediction using the trained recognition model.

[0041] The trained recognition model can be used for video classification prediction tasks, that is, the collected target ultrasound video is input into the trained recognition model to obtain classification prediction results, such as for human action classification and recognition, image classification, video speech recognition, video text detection, etc.

[0042] Accordingly, the present invention also provides an ultrasound video recognition system based on diffuse attention, which is used to implement one or more aspects of the above method. For example, the system includes: a video acquisition module, which is used to acquire ultrasound video; a classification prediction unit, which is used to input the ultrasound video into a trained recognition model to obtain a classification prediction result. The recognition model uses multiple stages to obtain the overall representation of the ultrasound video, each stage includes a spatiotemporal block representation module, a diffuse window attention module and a spatiotemporal block merging module, the spatiotemporal block representation module is used to divide the input ultrasound video into time and space to obtain a discrete spatiotemporal block set; the diffuse window attention module establishes a distance dependency relationship between spatiotemporal blocks by alternating between a fixed window attention mechanism and a diffuse window attention mechanism; the spatiotemporal block merging module is used to downsample the features output by the diffuse window attention module. The fixed-window attention mechanism slices the spatiotemporal matrix corresponding to the spatiotemporal block based on a set fixed window to obtain a local fixed window. The tokens within the local fixed window are positionally encoded to convert them into a token sequence. The attention mechanism then uses this mechanism to exchange information between tokens and establish dependencies between different tokens. The diffuse window attention mechanism exchanges information from different local fixed windows within a diffuse window that is larger than the fixed window. Each unit can be implemented using a general-purpose processor, a dedicated processor, an FPGA, or a combination of software.

[0043] It should be noted that the ultrasound video recognition model training process described in the present invention can be performed offline on a server or in the cloud. Embedding the trained model into an electronic device enables real-time ultrasound video classification prediction. The electronic device can be a terminal device or a server, including any terminal device such as a mobile phone, tablet computer, personal digital assistant (PDA), in-vehicle computer, or smart wearable device (smartwatch, virtual reality glasses, virtual reality helmet, etc.). Servers include, but are not limited to, application servers or web servers, and can be standalone servers, cluster servers, or cloud servers.

[0044] In summary, compared with the prior art, the present invention has the following advantages:

[0045] 1) Compared with the video classification method based on frame processing, the present invention can directly process three-dimensional video data, quickly and efficiently obtain the overall information of the video, improve the accuracy of ultrasound video recognition and classification, and effectively classify diseases in ultrasound videos.

[0046] 2) The diffuse window attention mechanism proposed in this paper employs a novel window design approach that enhances the receptive field of view of features, allowing for rapid acquisition of the overall video information. The diffuse window not only extends beyond the spatial and temporal scope of the fixed window but also balances the token representations from different fixed windows. This allows for faster acquisition of the global receptive field without increasing network complexity, making training easier.

[0047] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0048] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0049] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0050] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0051] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0052] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0053] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0054] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0055] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for ultrasonic video recognition based on diffuse attention, comprising the following steps: Obtain ultrasound video; Inputting the ultrasound video into a trained recognition model to obtain a classification prediction result; The recognition model uses multiple stages to obtain the overall representation of the ultrasound video. Each stage includes a spatiotemporal block representation module, a diffuse window attention module, and a spatiotemporal block merging module. The spatiotemporal block representation module is used to perform spatiotemporal division on the input ultrasound video to obtain a discrete set of spatiotemporal blocks. The diffuse window attention module establishes a distance dependency between spatiotemporal blocks by alternating between a fixed window attention mechanism and a diffuse window attention mechanism. The spatiotemporal block merging module is used to downsample the features output by the diffuse window attention module. Among them, the fixed window attention mechanism cuts the spatiotemporal matrix corresponding to the spatiotemporal block based on the set fixed window to obtain a local fixed window, and performs position encoding processing on the tokens in the local fixed window to convert them into a token sequence, and then uses the attention mechanism to interact with the information between tokens and establish dependency relationships between different tokens; the diffusion window attention mechanism interacts information from different local fixed windows in a diffusion window that is larger than the fixed window.

2. The method according to claim 1, characterized in that The plurality of stages are set to four, and each stage corresponds to a different scale.

3. The method according to claim 1, wherein The diffusion window attention module is built based on the VideoSwin Transformer model.

4. The method according to claim 1, wherein The classification prediction result is a disease classification result.

5. The method according to claim 1, wherein The diffusion window attention mechanism increases the actual spatiotemporal interval of elements in the window through a diffusion factor, expanding the spatiotemporal range of the window while keeping the number of tokens in the window unchanged. The establishment of dependencies between tokens makes the receptive field of a token expand to the union of the receptive fields of all tokens in the window.

6. An ultrasonic video recognition system based on diffuse attention, comprising: Video acquisition module: used to acquire ultrasound video; Classification prediction unit: used for inputting the ultrasound video into the trained recognition model to obtain a classification prediction result; The recognition model uses multiple stages to obtain the overall representation of the ultrasound video. Each stage includes a spatiotemporal block representation module, a diffuse window attention module, and a spatiotemporal block merging module. The spatiotemporal block representation module is used to perform spatiotemporal division on the input ultrasound video to obtain a discrete set of spatiotemporal blocks. The diffuse window attention module establishes a distance dependency between spatiotemporal blocks by alternating between a fixed window attention mechanism and a diffuse window attention mechanism. The spatiotemporal block merging module is used to downsample the features output by the diffuse window attention module. Among them, the fixed window attention mechanism cuts the spatiotemporal matrix corresponding to the spatiotemporal block based on the set fixed window to obtain a local fixed window, and performs position encoding processing on the tokens in the local fixed window to convert them into a token sequence, and then uses the attention mechanism to interact with the information between tokens and establish dependency relationships between different tokens; the diffusion window attention mechanism interacts information from different local fixed windows in a diffusion window that is larger than the fixed window.

7. The system according to claim 6, wherein: The diffusion window attention module is built based on the VideoSwin Transformer model.

8. The system according to claim 6, wherein: The diffusion window attention mechanism increases the actual spatiotemporal interval of elements in the window through a diffusion factor, expanding the spatiotemporal range of the window while keeping the number of tokens in the window unchanged. The establishment of dependencies between tokens makes the receptive field of a token expand to the union of the receptive fields of all tokens in the window.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images