Ultrasound image prediction method and device based on multi-level spatiotemporal feature recurrent neural network

Abstract

The present invention discloses an ultrasound image prediction method and device based on a multi-layer spatiotemporal feature recurrent neural network. The method first pre-processes the ultrasound scanning effect, splits the image into images with time series information and performs noise reduction processing, locates the standard cross-section images required for the doctor's diagnosis in the time series, and uses them as training data for the ultrasound image prediction method; uses the transformer architecture with a variable window multi-head attention mechanism to extract the spatial features of the medical ultrasound image, and further learns the association between the changes in the spatial features and the changes in the time series features of the image data based on the long short-term memory neural network, so that the model predicts the standard cross-section images required for the doctor's diagnosis through the current input ultrasound image. The present invention reduces the medical knowledge barriers required for manual ultrasound examinations, improves the efficiency of doctors' scanning examinations, and is suitable for scenarios such as ultrasound imaging examinations and ultrasound physician training in areas with a shortage of medical human resources.

Description

Technical Field

[0001] The present invention relates to the fields of medical image processing and recurrent neural networks, and in particular to an ultrasound image prediction method and device based on a multi-level spatiotemporal feature recurrent neural network. Background Art

[0002] Medical ultrasound screening requires extensive medical expertise. The uneven distribution of medical resources across the country has led to insufficient ultrasound screening coverage in underdeveloped regions. Traditional AI algorithms and technologies are unable to effectively assist medical personnel lacking specialized knowledge in scanning and screening specific body parts. Automatic prediction of diagnostic image sections based on learning ultrasound image feature change patterns can serve as an auxiliary tool for human ultrasound scanning training and examinations, lowering the medical knowledge threshold required for ultrasound image interpretation and diagnostic image acquisition, increasing ultrasound screening coverage in rural areas, and thus alleviating the uneven distribution of medical resources. During the scanning process, ultrasound doctors collect standard cross-sectional images required for diagnosis by identifying the approximate scanning area, observing changes in medical features, and then assessing and saving diagnostic images. Image recurrent networks can be used to learn changes in ultrasound image features over a continuous time series. Long-short-term memory (LSTM) neural networks are commonly used for natural language generation tasks with temporal properties, learning to understand semantic changes in preceding text and inferring and generating subsequent text. Images are composed of continuous images in the same time series. After extracting features from the image through a convolutional neural network, it is combined with recurrent network training to predict and generate natural images that are later in the time series. It is the earliest neural network model for image prediction.

[0003] When it comes to image feature extraction, convolutional neural networks (CNNs) are more sensitive to local features in images, have higher computational efficiency, and are better at capturing subtle changes in local features. Transformers, based on an attention mechanism, capture global dependencies, namely the relationships between local image features and the overall image and other local features, and are more capable of modeling long-range dependencies. To better extract medical features from ultrasound images, Transformers incorporate a hierarchical CNN construction method, where the downsampling factor of feature maps changes with the number of layers. Subsequent research will use a windowed attention mechanism to reduce the computational effort of a single Transformer module, improving the Transformer module's ability to extract medical image features with a minimal increase in computational effort. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention proposes a method and device for ultrasound image prediction based on a multi-level spatiotemporal feature recurrent neural network. This method studies and develops a data processing method for predicting standard section diagnostic images in retrospective ultrasound image data. The method mainly locates the ultrasound standard section images required for diagnosis in the image data to obtain the temporal position of the gold standard image in the image. The prediction of the ultrasound standard section image is completed by an on-demand combination of image patch embedding, a multi-level spatiotemporal feature recurrent network, image patch downsampling, image patch upsampling, and image reconstruction layers. The multi-level feature recurrent network is based on a long short-term memory network. It uses a variable window sampling Transformer to replace the traditional CNN to extract the spatiotemporal features of medical images. It also uses a spatial mean loss function to calculate the temporal mean of several frames before and after the target prediction image. The calculation results are compared with the gold standard using a loss function that includes spatial structural constraints and training.

[0005] The object of the present invention is achieved through the following technical solution: a method for ultrasound image prediction based on a multi-level spatiotemporal feature recurrent neural network, the method comprising the following steps:

[0006] S1, establish a retrospective human ultrasound scan image database and perform data pairing between the gold standard of diagnostic images and the corresponding images;

[0007] S2, uses the ultrasound image label positioning algorithm to determine the temporal position of the label in the corresponding image, and performs noise reduction and cropping on the image;

[0008] S3: Build an ultrasound image prediction model, including: image embedding patches, a multi-level spatiotemporal feature recurrent network, image patch up- and down-sampling layers, and an image reconstruction layer. The final output is a human ultrasound scan image that meets the doctor's diagnostic criteria. The multi-level spatiotemporal feature recurrent network consists of a long-short-term recurrent network nested within a Transformer with variable window sampling and a spatial mean loss function.

[0009] S4, inputting the processed image data set into the ultrasound image prediction model for training, and performing ultrasound image prediction based on the trained model.

[0010] Furthermore, in step S2, each image file corresponds to a gold standard image for diagnosis, and the positioning algorithm combines the RMSE index and the structural similarity index to calculate the frame number position of the gold standard image in the time series of the corresponding image.

[0011] Furthermore, in step S3, the image embedding patch, image patch upsampling layer and downsampling layer are respectively based on the image vectorization processing, image patch expansion and image patch splicing technology used by Transformer, and the image reconstruction layer is a two-dimensional deconvolution operation.

[0012] Furthermore, in step S3, the variable window sampling transforms the original global self-attention mechanism operation into a calculation within a given window size range, and allows image feature information to be transferred between adjacent windows by moving the window.

[0013] Furthermore, in step S3, the spatial mean loss function is based on the positioning algorithm and selects the image output of the current frame and the image prediction results of several previous frames for mean processing as the prediction result of the current frame, which is used to simulate the characteristics of ultrasound images in the actual scanning process. It is a loss function that includes image spatial features and image temporal change rules.

[0014] Furthermore, in step S3, the ultrasound image prediction model can include multiple multi-level spatiotemporal feature recurrent networks connected in series.

[0015] Furthermore, in step S3, the multi-level spatiotemporal feature recurrent network can include multiple Transformers with variable window sampling for image feature extraction, and the number of Transformers varies according to the specific task and computing resources.

[0016] Furthermore, in step S4, the ultrasound image prediction model uses the Adam optimizer to complete network training, independently selects the initial learning rate, and continuously changes the learning rate according to the gradient change during training.

[0017] In the second aspect, the present invention also provides an ultrasound image prediction device based on a multi-layer spatiotemporal feature recurrent neural network, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the ultrasound image prediction method based on a multi-layer spatiotemporal feature recurrent neural network.

[0018] In a third aspect, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the ultrasound image prediction method based on a multi-layer spatiotemporal feature recurrent neural network.

[0019] The beneficial effects of the present invention are as follows: the present invention pairs retrospective medical ultrasound image data through an ultrasound image label positioning algorithm to construct a dynamic image data database; uses a multi-level spatiotemporal feature recurrent network composed of a long short-term memory network and an improved Transformer to predict and generate images required for the doctor's diagnosis during the scanning process, and develops a variable window sampling mechanism during Transformer feature extraction to simultaneously extract local and global features of the image with a small amount of computation; further simulates the doctor's thinking of selecting diagnostic images, and improves the accuracy of the target predicted image through the spatial mean loss function. This invention can reduce the knowledge barriers for medical staff when performing human medical ultrasound scanning, alleviate the problem of lack of talent among grassroots ultrasound doctors, and enable ultrasound image inspection and screening to be more widely used; at the same time, the results of this method can be used as auxiliary materials for ultrasound scanning guidance training to improve training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is the overall flow chart of the present invention.

[0022] Figure 2 This is the architecture diagram of the multi-level spatiotemporal feature recurrent network in this invention, which includes the main innovative algorithms of variable window sampling and spatial mean loss function.

[0023] Figure 3 This is a structural diagram of an ultrasound image prediction device based on a multi-level spatiotemporal feature recurrent neural network according to the present invention. DETAILED DESCRIPTION

[0024] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0025] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0026] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0027] The present invention provides an ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network. Figure 1 As shown, the method includes the following steps:

[0028] S1, establish a retrospective human ultrasound scan image database;

[0029] In this database, the gold standard of diagnostic images and the corresponding images need to maintain a one-to-one correspondence. The paired image data are read frame by frame according to the acquired time series and saved as image files.

[0030] S2, obtain the temporal position of the gold standard in the image, perform image preprocessing, and establish an ultrasound image prediction dataset;

[0031] The temporal position of the gold standard in the image is obtained, and image preprocessing is performed to establish an ultrasound image prediction dataset, including: using an ultrasound image label positioning algorithm to determine the temporal position of the gold standard in the corresponding image, and using the video frame corresponding to the temporal position and the 24 video frames before it as the ultrasound image prediction data for this scan, of which the first 15 frames are input frames and the last 10 frames are prediction frames; using Gaussian filtering to denoise the gold standard image and the ultrasound image prediction data, and then scaling the high-quality image to 320×240 resolution. The data enhancement method is the same as the general data enhancement method for deep learning; and dividing the ultrasound image prediction data into a training set, a validation set, and a test set according to a 6:2:2 data partitioning method to establish an ultrasound image prediction dataset.

[0032] The ultrasound image tag positioning algorithm is used to calculate the frame number position of the gold standard image in the time series of the image to which it belongs. The positioning algorithm formula is:

[0033]

[0034] in, Represents the similarity index of mixed RMSE and SSIM images, represents a custom hyperparameter between 0 and 1. represents the RMSE indicator, Indicates the structural similarity index. In the specific implementation case is 0.1.

[0035] S3, predicts the standard cross-sectional images required for diagnosis by doctors through ultrasound images;

[0036] like Figure 2 As shown, the ultrasound image prediction model includes: image embedding patches, multi-level spatiotemporal feature recurrent networks, image patch up- and down-sampling layers, and image reconstruction layers, which are combined as needed.

[0037] The image patch embedding, image patch upsampling layer, and image patch downsampling layer utilize the image vectorization, image patch expansion, and image patch concatenation techniques used in the general Transformer, respectively. The image reconstruction layer utilizes a two-dimensional deconvolution operation. In a specific implementation, two multi-level spatiotemporal feature recurrent networks are embedded before and after the image patch downsampling layer.

[0038] S4, construction of multi-level spatiotemporal feature recurrent network;

[0039] The multi-level spatiotemporal feature recurrent network includes: a Transformer image feature extraction with variable window sampling, a long short-term memory neural network, and a spatial mean loss function;

[0040] The multi-level spatiotemporal feature recurrent network is based on an improvement to the long short-term memory neural network. It primarily replaces the traditional CNN with a Transformer with variable window sampling for spatiotemporal feature extraction. By capturing the relationship between local image features and global features, the model has stronger analytical capabilities for long-range dependencies. In a specific implementation case, each multi-level spatiotemporal feature recurrent network embeds two Transformer modules with variable window sampling.

[0041] The variable window sampling transforms the original global self-attention mechanism operation into a calculation within a given window size, and by moving the window, image feature information can be transferred between adjacent windows. In the specific implementation case, the window size is 2.

[0042] Furthermore, based on the ultrasound image label positioning algorithm, the image output of the current frame and the image output results of the previous two frames are respectively averaged, and the final average sum is used as the prediction result of the current frame. The formula is:

[0043]

[0044] in, Represents the index of the video frame sequence, For the Frame image output, Expressed as The average output of the frame.

[0045] The spatial mean loss function is different from the general single-image prediction result and the gold standard for loss calculation. Instead, the prediction results from the first frame to the second-to-last frame are spliced to obtain a prediction vector, and the real frames from the second frame to the last frame are spliced to obtain a real frame vector. The loss calculation using the prediction vector and the real frame vector can include the image spatial features and the image temporal change law. The formula of the loss function is:

[0046]

[0047] In specific implementation cases, represents the concatenated vector of the prediction results of frames 1 to 24, The splicing vector representing the real frames from the 2nd to the 25th frame is:

[0048]

[0049]

[0050] S5, ultrasound image prediction model training;

[0051] The ultrasound image prediction model uses the Adam optimizer in Torch, which employs an adaptive learning rate optimization algorithm. During training, the learning rate of each parameter is adjusted by calculating the exponential moving average of the first-order and second-order moment estimates of the gradient. In the specific implementation, the learning rate is 0.0001 and the batch size is 2.

[0052] In one embodiment, after training and verification converge simultaneously, the network model structure and its hyperparameters are saved. When the neural network algorithm is called, the network model can be read and input data can be directly predicted according to the saved hyperparameters.

[0053] Corresponding to the aforementioned embodiment of an ultrasound image prediction method based on a multi-layer spatiotemporal feature recurrent neural network, the present invention also provides an embodiment of an ultrasound image prediction system based on a multi-layer spatiotemporal feature recurrent neural network. This system includes a data matching module, an image processing module, a model building module, and an image prediction module. The implementation of each module in the system is similar to the specific implementation steps of the aforementioned ultrasound image prediction method based on a multi-layer spatiotemporal feature recurrent neural network.

[0054] The data pairing module is used to establish a retrospective human ultrasound scan image database and perform data pairing between the gold standard of diagnostic images and the corresponding images;

[0055] The image processing module is used to use the ultrasound image label positioning algorithm to determine the temporal position of the label in the corresponding image, and at the same time perform noise reduction and cropping on the image; the positioning algorithm combines the RMSE index and the structural similarity index to calculate the frame number position of the gold standard image in the time series of the image to which it belongs.

[0056] The model construction module is used to construct an ultrasound image prediction model, including: image embedding patches, multi-level spatiotemporal feature recurrent networks, image patch upsampling and downsampling layers, and image reconstruction layers. The final output result is a human ultrasound scan image that meets the doctor's diagnostic standards, wherein the multi-level spatiotemporal feature recurrent networks can be connected in series; the multi-level spatiotemporal feature recurrent network is composed of a long-term and short-term recurrent network nested in a Transformer containing variable window sampling and spatial mean loss function, which is used for image feature extraction. The number of these networks varies according to the specific task and computing power resources; the image embedding patches, image patch upsampling layers, and downsampling layers are respectively based on the image vectorization processing, image patch expansion, and image patch splicing technology used by Transformer, and the image reconstruction layer is a two-dimensional deconvolution operation. The variable window sampling transforms the original global self-attention mechanism operation into a calculation within a given window size range, and allows image feature information to be transferred in adjacent windows by moving the window. The spatial mean loss function is based on the positioning algorithm and selects the image output of the current frame and the image prediction results of several previous frames for average processing as the prediction result of the current frame. It is used to simulate the characteristics of ultrasound images in the actual scanning process. It is a loss function that includes image spatial characteristics and image temporal change rules.

[0057] The image prediction module is used to input the processed image dataset into the ultrasound image prediction model for training, and then perform ultrasound image prediction based on the trained model. The ultrasound image prediction model uses the Adam optimizer to complete network training, autonomously selects the initial learning rate, and continuously adjusts the learning rate based on gradient changes during training.

[0058] Corresponding to the aforementioned embodiment of an ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network, the present invention also provides an embodiment of an ultrasound image prediction device based on a multi-level spatiotemporal feature recurrent neural network.

[0059] See also Figure 3 An embodiment of the present invention provides an ultrasound image prediction device based on a multi-layer spatiotemporal feature recurrent neural network, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement an ultrasound image prediction method based on a multi-layer spatiotemporal feature recurrent neural network in the above embodiment.

[0060] The embodiment of an ultrasound image prediction device based on a multi-layer spatiotemporal feature recurrent neural network provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown in the figure, a hardware structure diagram of an ultrasound image prediction device based on a multi-level spatiotemporal feature recurrent neural network provided by the present invention is provided in any device with data processing capabilities, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0061] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0062] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0063] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, an ultrasound image prediction method based on a multi-layer spatiotemporal feature recurrent neural network in the above embodiment is implemented.

[0064] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0065] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network.

[0066] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. An ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network, characterized in that: The method comprises the following steps: S1, establish a retrospective human ultrasound scan image database and perform data pairing between the gold standard of diagnostic images and the corresponding images; S2, uses the ultrasound image label positioning algorithm to determine the temporal position of the label in the corresponding image, and performs noise reduction and cropping on the image; S3: Build an ultrasound image prediction model, including: image embedding patches, a multi-level spatiotemporal feature recurrent network, image patch up- and down-sampling layers, and an image reconstruction layer. The final output is a human ultrasound scan image that meets the doctor's diagnostic criteria. The multi-level spatiotemporal feature recurrent network consists of a long-short-term recurrent network nested within a Transformer with variable window sampling and a spatial mean loss function. S4, inputting the processed image data set into the ultrasound image prediction model for training, and performing ultrasound image prediction based on the trained model.

2. The ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network according to claim 1, characterized in that: In step S2, each image file corresponds to a gold standard image for diagnosis, and the positioning algorithm combines the RMSE index and the structural similarity index to calculate the frame number position of the gold standard image in the time series of the corresponding image.

3. The ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network according to claim 1, characterized in that: In step S3, the image embedding patch, image patch upsampling layer and downsampling layer are respectively based on the image vectorization processing, image patch expansion and image patch splicing technology used by Transformer, and the image reconstruction layer is a two-dimensional deconvolution operation.

4. The ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network according to claim 1, characterized in that: In step S3, the variable window sampling converts the original global self-attention mechanism operation into a calculation within a given window size range, and allows image feature information to be transferred between adjacent windows by moving the window.

5. The ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network according to claim 1, characterized in that: In step S3, the spatial mean loss function is based on the positioning algorithm and selects the image output of the current frame and the image prediction results of several previous frames for average processing as the prediction result of the current frame. It is used to simulate the characteristics of ultrasound images in the actual scanning process. It is a loss function that includes image spatial features and image temporal change rules.

6. The ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network according to claim 1, characterized in that: In step S3, the ultrasound image prediction model can include multiple multi-level spatiotemporal feature recurrent networks connected in series.

7. The ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network according to claim 1, characterized in that: In step S3, the multi-level spatiotemporal feature recurrent network can include multiple Transformers with variable window sampling for image feature extraction, and the number of Transformers varies according to the specific task and computing resources.

8. The ultrasound image prediction method based on a multi-level spatiotemporal feature recurrent neural network according to claim 1, characterized in that: In step S4, the ultrasound image prediction model uses the Adam optimizer to complete network training, independently selects the initial learning rate, and continuously changes the learning rate according to the gradient changes during training.

9. An ultrasound image prediction device based on a multi-layer spatiotemporal feature recurrent neural network, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements an ultrasound image prediction method based on a multi-layer spatiotemporal feature recurrent neural network as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, an ultrasound image prediction method based on a multi-layer spatiotemporal feature recurrent neural network as described in any one of claims 1 to 8 is implemented.

Images