Ultrasonic image synthesis method based on sequence-to-sequence model and blood flow information

Abstract

The invention discloses an ultrasonic image synthesis method based on a sequence-to-sequence model and blood flow information. The method comprises the steps that a data set is constructed by obtaining a standard gray level ultrasonic image video, a corresponding contrast enhancement ultrasonic image video and a blood flow image of a diseased region of a clinical patient in the past; training an ultrasonic image synthesis model by using the data set; collecting a standard gray level ultrasonic image video and a corresponding blood flow image of a current clinical patient, and inputting the standard gray level ultrasonic image video and the corresponding blood flow image into the trained ultrasonic image synthesis model; and the ultrasonic image synthesis model converts the standard grayscale ultrasonic image video into a corresponding contrast-enhanced ultrasonic image video under the guidance of blood flow information contained in the blood flow image, and outputs and displays the contrast-enhanced ultrasonic image video. The blood flow information of the patient is integrated into the ultrasonic image generation process, the clinical correlation and accuracy of the generated image can be remarkably improved, dependence on a contrast agent is reduced, the quality and time sequence consistency of the image are improved, and a more effective tool is provided for clinical diagnosis.

Description

Technical Field

[0001] The present invention belongs to the field of generative algorithms of medical image processing and deep learning technology, and specifically relates to an ultrasound image synthesis method based on a sequence-to-sequence (Seq2Seq) model and blood flow information. Background Art

[0002] In the field of medical imaging, ultrasound imaging is an important diagnostic tool, especially playing a key role in the diagnosis of neurological diseases. Contrast-enhanced ultrasound imaging technology can enhance the visualization of lesions and surrounding tissues, improve image clarity and visualization, and thus help doctors detect lesions and monitor disease progression more accurately. However, existing contrast-enhanced ultrasound imaging technology relies on the use of ultrasound contrast agents, which also brings potential health risks. For example, the accumulation of contrast agents in the body may cause adverse reactions in patients and cause long-term effects on patients' health. In addition, traditional ultrasound enhanced examinations require multiple injections of contrast agents, which not only increases the complexity and cost of clinical operations, but also prolongs the examination time. Therefore, how to reduce the use of contrast agents while improving image quality has become a research hotspot in the field of medical imaging.

[0003] In recent years, the Sequence-to-Sequence Model (Seq2Seq) model has been gradually introduced into the conversion task of medical imaging due to its successful application in the field of natural language processing. This type of Seq2Seq model can convert the input medical image into a high-quality target image by adopting an encoder-decoder architecture. Although existing studies have shown that the Seq2Seq model has potential in ultrasound image generation, most studies mainly rely on the information of the image itself for training, lack the use of blood flow information, and fail to effectively capture the key information in the dynamic physiological process, resulting in the generated contrast-enhanced ultrasound images. Poor performance in clinical applications lacks the necessary details and structure, affecting the accuracy of clinical diagnosis.

[0004] The existing contrast-enhanced ultrasound image synthesis technology mainly has the following problems: 1. Reliance on image intrinsic information: Many existing sequence-to-sequence (Seq2Seq) models and generative adversarial network (GAN) methods rely mainly on the information of the input image itself for training. As a result, the generated ultrasound images often lack sufficient details and clinical relevance, making it difficult to meet the needs of actual medical diagnosis. Especially in mammography, accurate detection and localization of lesions are crucial, and image intrinsic information alone may not be able to reliably capture the characteristics of lesions, thus affecting the accuracy of diagnosis.

[0005] 2. Lack of utilization of blood flow information: Many current ultrasound imaging technologies fail to integrate blood flow information into the ultrasound image generation process. Blood flow information is usually obtained through Doppler ultrasound. Blood flow information is crucial for understanding the functional status of blood vessels and surrounding tissues, but traditional contrast-enhanced ultrasound imaging methods often ignore it and do not make full use of it. This results in the generated contrast images may lack important hemodynamic characteristics, affecting the comprehensive assessment and diagnostic accuracy of breast lesions. Without the support of blood flow information, the generated images may not be reliable enough in clinical applications.

[0006] 3. Insufficient temporal consistency: Existing ultrasound imaging technology often has difficulty maintaining temporal consistency when generating image sequences, resulting in a lack of temporal coherence in the generated images. This is particularly important when conducting long-term observations or real-time diagnosis. Breast contrast agent ultrasound requires accurate capture of dynamic physiological processes, and if the generated images have obvious inconsistencies between different time frames, it will reduce its practicality in clinical practice and affect physicians’ assessment and decision-making of patients’ conditions. Summary of the invention

[0007] Based on this, the present invention provides an ultrasound image synthesis method based on a sequence-to-sequence (Seq2Seq) model and blood flow information, so as to directly convert the original ultrasound image into a high-quality contrast-enhanced ultrasound image (CEUS), which is beneficial to significantly improve the image quality, clarity and timing consistency of the generated contrast-enhanced ultrasound image.

[0008] In order to solve the above technical problems and achieve the above technical effects, the present invention is implemented through the following technical solutions: A method for synthesizing ultrasound images based on a sequence-to-sequence model and blood flow information comprises the following steps: Step 1) Obtain several standard grayscale ultrasound image videos of lesions of previous clinical patients and the corresponding contrast-enhanced ultrasound image videos and blood flow images, and use them to construct a data set for model training; Step 2) using the data set, training the constructed ultrasound image synthesis model, and training the model to automatically convert the standard grayscale ultrasound image video into the corresponding contrast-enhanced ultrasound image video under the guidance of the corresponding blood flow information; Step 3) collecting the standard grayscale ultrasound image video and the corresponding blood flow image of the current clinical patient, and simultaneously inputting them into the trained ultrasound image synthesis model; Step 4) the ultrasound image synthesis model performs enhancement processing on the standard grayscale ultrasound image video under the guidance of the blood flow information contained in the blood flow image, thereby synthesizing the corresponding contrast-enhanced ultrasound image video; Step 5) output and display the synthesized contrast-enhanced ultrasound image video.

[0009] Furthermore, in step 1), the method for constructing the data set is: 1) Collect several standard grayscale ultrasound image videos of the lesion sites of previous clinical patients; 2) Collect contrast-enhanced ultrasound image videos of the lesion site of each previous clinical patient; 3) Collect blood flow images of the lesion site of each previous clinical patient; 4) Standard grayscale ultrasound image videos, contrast-enhanced ultrasound image videos, and blood flow images belonging to the same previous clinical patient are sorted and paired to construct the data set.

[0010] Furthermore, in step 2), the ultrasound image synthesis model uses a sequence-to-sequence (Seq2Seq) model with an attention mechanism as the basic architecture, and its specific structure includes: A first encoder with a ConvLSTM layer is responsible for reading an input blood flow image sequence, and performing time series processing on the blood flow image sequence through the ConvLSTM layer to which it belongs, so as to output an additional vector, wherein the additional vector contains blood flow information of the input blood flow image sequence; The second encoder with the ConvLSTM layer is responsible for reading each input standard grayscale ultrasound image sequence, and performing time series processing on each standard grayscale ultrasound image sequence through the ConvLSTM layer to which it belongs, so as to output a context vector, wherein the context vector contains ultrasound information of the input standard grayscale ultrasound image sequence; A decoder with a ConvLSTM layer is responsible for taking the context vector output by the second encoder as a basic input, taking the additional vector output by the first encoder as a guide signal, and performing time-series processing on the context vector through the ConvLSTM layer to which it belongs, and finally gradually decoding and outputting a contrast-enhanced ultrasound image sequence corresponding to the input standard grayscale ultrasound image sequence; An attention mechanism is responsible for helping the decoder to refine the encoded features from ultrasound information and blood flow information when decoding the context vector, so as to enhance the ability of the ultrasound image synthesis model to selectively focus on relevant areas affected by hemodynamics, thereby improving the quality and clinical relevance of the generated contrast-enhanced ultrasound image sequence. ‌‌ Furthermore, the internal structures of the second encoder and the decoder are respectively: The first encoder and the second encoder both include two convolutional layers and five residual blocks connected in sequence; The decoder includes four residual blocks and three super-convolutional layers connected in sequence, and an upsampling layer arranged between two adjacent super-convolutional layers.

[0011] Furthermore, in step 2), the training method of the ultrasound image synthesis model is: 1) Data preparation; Selecting several standard grayscale ultrasound image videos from the data set, and randomly selecting a certain number of video frames from each of the selected standard grayscale ultrasound image videos as standard grayscale ultrasound images for training; At the same time, in the data set, a contrast-enhanced ultrasound image and a blood flow image corresponding to the randomly selected standard grayscale ultrasound image are selected; 2) Data preprocessing; Performing preprocessing including clipping and normalization on the selected standard grayscale ultrasound images, contrast-enhanced ultrasound images and the corresponding blood flow images; 3) Sequence transformation; converting the preprocessed standard grayscale ultrasound image, contrast-enhanced ultrasound image and the corresponding blood flow image into a standard grayscale ultrasound image sequence, a contrast-enhanced ultrasound image sequence and the corresponding blood flow image sequence respectively; 4) Introduction of attention mechanism; An attention mechanism is introduced to help the decoder refine the encoded features from ultrasound information and blood flow information when decoding the context vector, so as to enhance the ability of the ultrasound image synthesis model to selectively focus on relevant areas affected by hemodynamics, thereby improving the quality and clinical relevance of the generated contrast-enhanced ultrasound image sequence. 5) Model training: Taking a preprocessed standard grayscale ultrasound image sequence as a basic input, a preprocessed contrast-enhanced ultrasound image sequence as a target output, and a preprocessed and labeled blood flow image sequence as an additional input, the ultrasound image synthesis model is guided by the blood flow image sequence to learn a mapping relationship from the standard grayscale ultrasound image sequence to the contrast-enhanced ultrasound image sequence, so as to achieve prediction and generation of the contrast-enhanced ultrasound image sequence; 6) Loss calculation; Using a structural coherence loss function, calculating the difference between the contrast-enhanced ultrasound image sequence and the standard grayscale ultrasound image sequence, and adjusting the parameters of the first encoder, the second encoder and the decoder according to the calculated loss to minimize the loss; 7) Evaluation and adjustment; After the training is completed, a specific evaluation index is used to evaluate the performance of the ultrasound image synthesis model, and the ultrasound image synthesis model is further adjusted according to the evaluation result. Furthermore, the key factors comprehensively considered by the structural coherence loss function include: 1) Smoothing and transition issues; 2) Structural integrity issues; 3) Time consistency problem.

[0012] 7. The ultrasound image synthesis method based on sequence-to-sequence model and blood flow information according to claim 5, characterized in that the evaluation index includes: 1) Peak signal-to-noise ratio; 2) Structural similarity index measurement.

[0013] 3) Visual inspection results.

[0014] Further, the specific method of step 3) to step 5) is: 1) Collect the standard grayscale ultrasound image video and corresponding blood flow image of the current clinical patient, and split the standard grayscale ultrasound image video into a single standard grayscale ultrasound image frame by frame; 2) converting all standard grayscale ultrasound images and blood flow images into corresponding standard grayscale ultrasound image sequences and blood flow image sequences respectively, and then simultaneously inputting the trained ultrasound image synthesis model; 3) The second encoder in the ultrasound image synthesis model reads each input standard grayscale ultrasound image sequence, and performs time series processing on each standard grayscale ultrasound image sequence through the ConvLSTM layer to which it belongs, thereby outputting a context vector, wherein the context vector contains ultrasound information of the input standard grayscale ultrasound image sequence; 4) The first encoder in the ultrasound image synthesis model reads the blood flow image sequence, and performs time series processing on the blood flow image sequence through the ConvLSTM layer to which it belongs, thereby outputting an additional vector, wherein the additional vector contains blood flow information of the input blood flow image sequence; 5) The encoder in the ultrasound image synthesis model uses the context vector output by the second encoder as a basic input, uses the additional vector output by the first encoder as a guide signal, and performs time-series processing on the context vector through the ConvLSTM layer to which it belongs, and finally gradually decodes and outputs a contrast-enhanced ultrasound image sequence corresponding to the input standard grayscale ultrasound image sequence; 6) Convert all synthesized contrast-enhanced ultrasound image sequences into contrast-enhanced ultrasound images, and finally display multiple contrast-enhanced ultrasound images or a contrast-enhanced ultrasound image video to provide an intuitive display for doctors.

[0015] A computer device comprises: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to execute operations corresponding to the ultrasound image synthesis method based on a sequence-to-sequence model and blood flow information.

[0016] A computer-readable storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the ultrasound image synthesis method based on a sequence-to-sequence model and blood flow information.

[0017] The beneficial effects of the present invention are: 1. This paper proposes an advanced sequence-to-sequence (Seq2Seq) model that can directly convert raw ultrasound images into high-quality contrast-enhanced ultrasound images (CEUS). The model is designed to improve the clarity and temporal consistency of the generated images, ensuring that the generated images are reliable in clinical diagnosis. By optimizing the architecture of the encoder and decoder, the limitations of the current traditional Seq2Seq model in terms of generation quality and clinical applicability are addressed.

[0018] 2. The core innovation of this invention is to integrate patient-specific blood flow information as an additional input guide into the sequence-to-sequence (Seq2Seq) model, so as to directly convert the original ultrasound image into a high-quality contrast-enhanced ultrasound image (CEUS). This approach enables the sequence-to-sequence (Seq2Seq) model to more accurately capture lesion information, so that the generated images can more accurately reflect physiological dynamics, and improve the clinical relevance and accuracy of the generated images. This innovative technical point fills the gap in the existing ultrasound image generation method that does not consider blood flow dynamics, and significantly improves the diagnostic ability of blood vessels and surrounding tissues.

[0019] 3. The present invention also introduces a novel structural consistency loss function, which aims to enhance the spatiotemporal consistency and smoothness of the generated image sequence. This loss function comprehensively considers the smoothness and transition problems, structural integrity problems and temporal consistency problems, ensuring that the generated image is not only visually similar to the target image, but also maintains structural integrity and temporal consistency. This innovative technical point can effectively improve the reliability of image sequences in dynamic monitoring and comparative analysis, thereby improving the quality and usability of generated images and providing higher diagnostic accuracy for clinical applications. This is crucial for clinical diagnosis because doctors need to maintain the structural integrity of images between different frames.

[0020] 4. The present invention also shows great potential in reducing dependence on contrast agents. By generating contrast-enhanced ultrasound images directly from conventional ultrasound images, the risks and examination costs of patients can be reduced. This has important clinical value, especially for patient groups who need frequent examinations. Therefore, compared with traditional methods, the present invention is expected to provide a more reliable and effective solution for medical imaging applications.

[0021] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is a schematic diagram of the steps of the ultrasound image synthesis method based on the sequence-to-sequence model and blood flow information of the present invention; Figure 2 The architecture diagram of the ultrasound image synthesis model constructed for the present invention; Figure 3 10 paired image data in the dataset shown in the embodiments of the present invention; Figure 4 A schematic diagram of the steps of training an ultrasound image synthesis model according to the present invention; Figure 5 These are the sample output results of 10 groups of ultrasound image synthesis models shown in an embodiment of the present invention, where each group of samples is, from left to right, the original input standard grayscale ultrasound image, the generated output contrast-enhanced ultrasound image, and the ground truth contrast-enhanced ultrasound image.

[0023] Figure 6 The comparison result of generating contrast-enhanced ultrasound images with and without blood flow guidance in an embodiment of the present invention is as follows: Figure 6 a represents the original standard grayscale ultrasound image, Figure 6 b represents the contrast-enhanced ultrasound image generated without blood flow guidance. Figure 6 c represents the contrast-enhanced ultrasound image generated by blood flow guidance. Figure 6 d represents the contrast-enhanced ultrasound image of the real image; Figure 7 Another comparison result of generating contrast-enhanced ultrasound images with and without blood flow guidance in an embodiment of the present invention is as follows: Figure 7 a represents the original standard grayscale ultrasound image, Figure 7b represents the contrast-enhanced ultrasound image generated without blood flow guidance. Figure 7 c represents the contrast-enhanced ultrasound image generated by blood flow guidance. Figure 7 d shows the contrast-enhanced ultrasound image of the real image. DETAILED DESCRIPTION

[0024] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings so that the purpose, features and advantages of the invention can be more clearly understood. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0025] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0026] Unless the context requires otherwise, throughout the specification and claims, the word "comprise" and variations such as "include" and "have" should be construed in an open, inclusive sense, ie, should be interpreted as "including, but not limited to."

[0027] References throughout the specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0028] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should be noted that the term "or" is generally employed in its sense including "and / or" unless the context clearly dictates otherwise.

[0029] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0030] See also Figure 1 As shown, a method for synthesizing ultrasound images based on a sequence-to-sequence model and blood flow information mainly includes the following steps: Step 1) Obtain several standard grayscale ultrasound image videos of lesion sites of previous clinical patients and the corresponding contrast-enhanced ultrasound image videos and blood flow images, and use them to construct a data set for model training.

[0031] Step 2) Using the data set, the constructed ultrasound image synthesis model is trained to train the model to automatically convert the standard grayscale ultrasound image video into the corresponding contrast-enhanced ultrasound image video under the guidance of the corresponding blood flow information.

[0032] Step 3) Collect the standard grayscale ultrasound image video and the corresponding blood flow image of the current clinical patient, and input them into the trained ultrasound image synthesis model at the same time.

[0033] Step 4) The ultrasound image synthesis model enhances the standard grayscale ultrasound image video under the guidance of the blood flow information contained in the blood flow image, thereby synthesizing the corresponding contrast-enhanced ultrasound image video.

[0034] Step 5) output and display the synthesized contrast-enhanced ultrasound image video.

[0035] The ultrasound image synthesis model of the present invention adopts a sequence-to-sequence (Seq2Seq) model with an attention mechanism as a basic architecture to achieve the conversion of standard grayscale ultrasound images to contrast-enhanced ultrasound images.

[0036] The Seq2Seq model is actually an Encoder-Decoder network. Its input is a sequence and its output is also a sequence. The Encoder converts the input sequence into a vector of fixed length, and then converts the vector into the output sequence we want through the Decoder.

[0037] See also Figure 2 As shown, the specific structure of the ultrasound image synthesis model of the present invention mainly includes: a first encoder with a ConvLSTM layer, a second encoder with a ConvLSTM layer, a decoder with a ConvLSTM layer and an attention mechanism, etc. The specific tasks they undertake are: 1. The first encoder is responsible for reading the input blood flow image sequence and performing time-series processing on the blood flow image sequence through the ConvLSTM layer to which it belongs, thereby outputting an additional vector of fixed length, wherein the additional vector contains the blood flow information (physiological data) of the input blood flow image sequence. The specific method may be, The patient's blood flow information is extracted from the blood flow image and used as a text control condition. By marking the region of interest, the corresponding valid box is made to serve as an additional input of the model, so that the model can adapt to small sample medical datasets and have user-friendly controllability.

[0038] 2. The second encoder is responsible for reading each input standard grayscale ultrasound image sequence, and performing time series processing on each standard grayscale ultrasound image sequence through its ConvLSTM layer, thereby outputting a context vector of a fixed length, wherein the context vector contains the ultrasound information of the input standard grayscale ultrasound image sequence.

[0039] 3. The decoder is responsible for taking the context vector output by the second encoder as the basic input, the additional vector output by the first encoder as the guide signal, and performing time series processing on the context vector through the ConvLSTM layer to which it belongs, and finally gradually decoding and outputting a contrast-enhanced ultrasound image sequence corresponding to the input standard grayscale ultrasound image sequence.

[0040] 4. Attention mechanism, The attention mechanism is responsible for helping the decoder to refine the encoded features from the ultrasound information and blood flow information when decoding the context vector, so as to enhance the ability of the ultrasound image synthesis model to selectively focus on relevant areas affected by hemodynamics, thereby improving the quality and clinical relevance of the generated contrast-enhanced ultrasound image sequence.

[0041] Previous Seq2Seq models for ultrasound image conversion often fail to fully consider dynamic physiological factors, such as blood flow, which is critical for accurate diagnostic imaging. Traditional models focus primarily on information inherent to ultrasound images, resulting in generated images that lack clinical reliability and relevance. In addition, previous methods did not directly address the conversion from conventional ultrasound images to contrast-enhanced ultrasound (CEUS) images, ignoring the importance of real-time physiological data in this process. These limitations result in generated contrast-enhanced ultrasound (CEUS) images that are unreliable and of little use for clinical diagnosis.

[0042] To address these shortcomings, the present invention innovatively integrates patient-specific blood flow information into the encoder and adopts a robust computational framework, the Seq2Seq model, to accommodate the integration of blood flow information. This integration can adaptively adjust the conversion process based on real-time physiological data, thereby enhancing the fidelity and clinical relevance of the generated CEUS images. By integrating blood flow information, a more comprehensive representation of the underlying physiological condition is provided, which is critical for accurate and reliable diagnostic imaging.

[0043] In the present invention, the ConvLSTM layer is a deep learning model that combines a convolutional neural network (CNN) and a long short-term memory network (LSTM), which is specifically used to process time series data and can capture the spatiotemporal correlation in ultrasound data. Therefore, the integration of the ConvLSTM layer further enhances the ability of the ultrasound image synthesis model of the present invention to capture the spatiotemporal correlation in ultrasound data, which is crucial for maintaining the temporal coherence and structural integrity of the generated image.

[0044] (1); In formula (1), x t represents the input at time step t, h t represents the hidden state, c t represents the cell state, i t represents the input gate, f t represents the forget gate, o t Represents the output gate, σ() represents the sigmoid function, tanh() represents the hyperbolic tangent function, W, U, V represent weight matrices, b represents the bias term, and e represents element-by-element multiplication.

[0045] The core of the present invention is to use the patient's blood flow image sequence as an additional input to the encoder. The blood flow information contained in the blood flow image sequence will serve as a guiding signal to affect the attention mechanism and feature extraction process. Figure 2 As shown, the blood flow image sequence is processed by a separate branch of the ConvLSTM layer. By encoding the blood flow information contained in the blood flow image sequence together with the original standard grayscale ultrasound image sequence, the ultrasound image synthesis model of the present invention effectively adjusts its conversion to reflect the real-time changes in hemodynamics, which is crucial for accurate diagnostic imaging.

[0046] Given a series of inputs X={Iblood,Ius,0,Ius,1,…,Ius,T-1}, where Iblood represents the blood flow image of the area, I represents the ultrasound image at time point T, and T is the time window size, then we pass them through a series of encoder structures to obtain the corresponding embedding E={eblood,eus,0,eus,1,…,eus,T-1}. See Figure 2 As shown in Figure 1, each encoder contains two convolutional layers and five residual blocks connected in sequence.

[0047] Next, we divide the encoder into two parts, Eblood=eblood, Eus={Eus,0,Eus,1,…,eus,T-1}, and pass them to two branches respectively. These branches have the same structure, but are used to process blood flow information and ultrasound information respectively. The general calculation formula of the attention mechanism is as follows: (2); The present invention adds an attention mechanism module after the ConvLSTM layer to which the encoder belongs to further refine the encoded features from the ultrasound input and blood guidance information. This attention mechanism enhances the ability of the ultrasound image synthesis model of the present invention to selectively focus on relevant areas affected by hemodynamics, thereby improving the quality and clinical relevance of the generated CEUS images. Mathematically, the attention mechanism operates by calculating attention weights, which guide the model to focus on important features. The refined calculation formula of the attention mechanism is as follows: (3); In formula (3), Q = K = V = concat [zblood; zus] represents the query matrix, key matrix and value matrix, d represents the dimension of K, and δ () represents the softmax function. This operation calculates the weighted sum of the value matrix V based on the similarity between the query matrix Q and the key matrix K.

[0048] Finally, at the decoder stage, see Figure 2 As shown, the decoder includes 4 residual blocks and 3 super-convolutional layers connected in sequence, and an upsampling layer arranged between two adjacent super-convolutional layers. The residual blocks and super-convolutional layers in the decoder are helpful for the reconstruction of contrast-enhanced ultrasound images.

[0049] Figure 2 The detailed structures of the second encoder and the decoder are demonstrated in the framework to ensure the preservation of morphological details and texture consistency, which are crucial for clinical interpretation and diagnostic accuracy.

[0050] After building the ultrasound image synthesis model of the present invention, it is necessary to first collect and construct a data set for training the ultrasound image synthesis model of the present invention. The specific method is as follows: 1) Collect several standard grayscale ultrasound image videos of the lesion sites of previous clinical patients; 2) Collect contrast-enhanced ultrasound image videos of the lesion site of each previous clinical patient; 3) Collect blood flow images of the lesion site of each previous clinical patient; 4) Standard grayscale ultrasound image videos, contrast-enhanced ultrasound image videos, and blood flow images belonging to the same previous clinical patient are sorted and paired to construct the data set.

[0051] In this study, we used an in-house dataset consisting of 219 patients with breast lesions who underwent ultrasound examinations in our clinic. Figure 3 As shown, the internal data set includes: standard grayscale ultrasound image videos, and contrast-enhanced ultrasound image videos and blood flow images corresponding to each standard grayscale ultrasound image video scanning site.

[0052] The internal data set also includes: comprehensive clinical information of the 219 previous clinical patients, such as gender, age, lesion location and classification, pathological results, and other relevant clinical data. Figure 3 As shown, 10 paired data from 219 previous clinical patients are displayed. The 10 paired data in the figure are from left to right: standard grayscale ultrasound image, contrast-enhanced ultrasound image, and blood flow image.

[0053] The standard grayscale ultrasound image videos are collected through routine clinical ultrasound examinations of patients with breast lesions. These image videos provide basic input for the ultrasound image synthesis model of the present invention.

[0054] Contrast-enhanced ultrasound image videos (CEUS) were acquired using a machine equipped with a LA2-9A probe. These image videos are used as target outputs for training the ultrasound image synthesis model of the present invention.

[0055] Blood flow images are obtained using Doppler ultrasound. These images provide additional input to guide the model, helping it generate higher quality and more accurate contrast-enhanced ultrasound image videos.

[0056] See also Figure 4 As shown, after constructing the data set for model training, the present invention further provides a method for training the ultrasound image synthesis model, which specifically includes the following steps: 1. Data preparation Several standard grayscale ultrasound image videos are selected from the data set, and a certain number of video frames (for example, 80%) are randomly selected from each selected standard grayscale ultrasound image video as standard grayscale ultrasound images for training.

[0057] At the same time, in the data set, contrast-enhanced ultrasound images and blood flow images corresponding to the randomly selected standard grayscale ultrasound images are selected.

[0058] 2. Data preprocessing The selected standard grayscale ultrasound images, contrast-enhanced ultrasound images and corresponding blood flow images are preprocessed including cropping and normalization. The specific method is: first, all standard grayscale ultrasound images, contrast-enhanced ultrasound images and blood flow images used as training inputs are uniformly adjusted to a standard resolution; for example, uniformly adjusted to 128*256 pixels; then all the intensity values ​​of standard grayscale ultrasound images, contrast-enhanced ultrasound images and blood flow images used as training inputs are normalized to a consistent range; finally, all standard grayscale ultrasound images used as training inputs are aligned with their corresponding contrast-enhanced ultrasound images and blood flow images to ensure accurate pairing; so far, all images used as training inputs are standardized to the same format and resolution to ensure the uniformity and compatibility of the entire data set.

[0059] 3. Sequence transformation The preprocessed standard grayscale ultrasound images, contrast-enhanced ultrasound images and the corresponding blood flow images are converted into standard grayscale ultrasound image sequences, contrast-enhanced ultrasound image sequences and the corresponding blood flow image sequences respectively.

[0060] 4. Introduction of attention mechanism An attention mechanism is introduced to help the decoder refine the encoded features from ultrasound information and blood flow information when decoding the context vector, so as to enhance the ability of the ultrasound image synthesis model to selectively focus on relevant areas affected by hemodynamics, thereby improving the quality and clinical relevance of the generated contrast-enhanced ultrasound image sequence. 5. Model training Using the preprocessed standard grayscale ultrasound image sequence as the basic input, the preprocessed contrast-enhanced ultrasound image sequence as the target output, and the preprocessed and labeled blood flow image sequence as the additional input, the network learns how to restore the original ultrasound image from noise under the supervision of control conditions, thereby improving the quality and clinical usability of the generated images. Specifically, the ultrasound image synthesis model learns the mapping relationship from the standard grayscale ultrasound image sequence to the contrast-enhanced ultrasound image sequence under the guidance of the blood flow image sequence, so as to achieve the prediction and generation of the contrast-enhanced ultrasound image sequence. Figure 5 As shown, 10 groups of sample output results of the ultrasound image synthesis model are shown, and each group of samples is, from left to right: the original input standard grayscale ultrasound image, the generated output contrast-enhanced ultrasound image, and the ground truth contrast-enhanced ultrasound image.

[0061] 6. Loss calculation The structural coherence loss function is used to calculate the difference between the contrast enhanced ultrasound image sequence and the standard grayscale ultrasound image sequence, and the parameters of the first encoder, the second encoder and the decoder are adjusted according to the calculated loss to minimize the loss. Generating high-quality, smooth, and temporally coherent contrast-enhanced ultrasound image sequences is a challenging task. Traditional loss functions often have limitations, such as lack of attention to structural information or poor performance in temporal coherence. To address these issues, this paper proposes a new loss function, called structural coherence loss (SCL). It comprehensively considers the following key factors to improve the quality of generated images.

[0062] 1) Smoothing and transition issues; The generated image sequence may have sudden changes or non-smooth transitions, affecting the continuity and visual quality of the image. 1 The exponential form of the loss is defined as: (4); In formula (4), represents the predicted value, Represents the true value.

[0063] L 1 The loss applies a Gaussian kernel to smooth the transitions between generated images, reducing abrupt changes and ensuring smoothness of the image sequence.

[0064] 2) Structural integrity issues; Relying solely on L 1 The loss may ignore the structural information of the image, resulting in the lack of structural accuracy and completeness of the generated image. To address this problem, the present invention introduces the structural similarity index (SSIM) loss L 2 , which is defined as: (5); In formula (5), represents the predicted value, Represents the true value.

[0065] SSIM loss L 2 The brightness, contrast and structure of the image are taken into account to better maintain and reconstruct the structural integrity of the image and enhance the perceptual quality of the generated image.

[0066] 3) Time consistency issues; The generated image sequence may lack consistency between consecutive frames, resulting in significant differences between frames, affecting the fluidity and coherence of the video. To solve this problem, the present invention introduces the mutual information (MI) loss L 3 , defined as: (6); In formula (6), represents the predicted value, Represents the true value.

[0067] The coherence loss function L of the present invention total That is, the loss L 1 , loss L 2 and loss L 3 Combination by certain weights: (7).

[0068] 7. Evaluate and adjust After the training is completed, a specific evaluation index is used to evaluate the performance of the ultrasound image synthesis model, and the ultrasound image synthesis model of the present invention is further adjusted according to the evaluation result.

[0069] Considering the generated contrast-enhanced images and the real images, the present invention thoroughly verifies the performance of the ultrasound image synthesis model of the present invention using several evaluation metrics, including the peak signal-to-noise ratio (PSNR) and the structural similarity index metric (SSIM), which are calculated as follows: (8); In formula (8), p max is the maximum pixel value of the generated image, and μ is the average pixel value.

[0070] (9); In formula (9), μ x and μ y are the average pixel values ​​of x and y respectively, and is the variance of x and y, σ is the covariance, C 1 =0.01×255, C 2 = 0.03×255 is a constant to stabilize the division of the weak denominator.

[0071] In addition, the present invention also performed a visual inspection to evaluate the prediction performance of the ultrasound image synthesis model of the present invention.

[0072] The impact of flow guidance was evaluated and the results showed that the inclusion of blood flow information significantly improved the performance of the model. Figure 6 As shown, Figure 6 a represents the original standard grayscale ultrasound image, Figure 6 b represents the contrast-enhanced ultrasound image generated without blood flow guidance. Figure 6 c represents the contrast-enhanced ultrasound image generated by blood flow guidance. Figure 6 d represents the contrast-enhanced ultrasound image of the real image, Figure 6 c and Figure 6 The box in d indicates the location of the lesion. Figure 7 As shown, Figure 7 a represents the original standard grayscale ultrasound image, Figure 7 b represents the contrast-enhanced ultrasound image generated without blood flow guidance. Figure 7 c represents the contrast-enhanced ultrasound image generated by blood flow guidance. Figure 7 d represents the contrast-enhanced ultrasound image of the real image, Figure 7 c and Figure 7 The box in d indicates the location of the lesion.

[0073] PSNR improved from 21.71 to 24.75, indicating that the images generated under the guidance of blood flow information are closer to the real images in terms of noise and clarity. SSIM increased from 0.993 to 0.994, indicating that the structural similarity of the generated images was improved, reflecting that important clinical features were better preserved. The improvement in structural coherence loss was the most significant, decreasing from 0.0024 to 0.0015. This metric is critical because it measures the temporal consistency of the image sequence, ensuring that the generated CEUS images are not only structurally accurate but also consistent over time, which is critical for clinical interpretation.

[0074] The present invention further evaluated the impact of single and multiple blood flow guidance on model performance, and the results showed that the single blood flow guided model outperformed the multiple blood flow guided model in all indicators. The single blood flow guided model had the highest PSNR (24.75), the highest SSIM (0.994), and the lowest structural coherence loss (0.0015). In contrast, the multiple blood flow guided model performed poorly, with a PSNR of 18.71, an SSIM of 0.973, and a structural coherence loss of 0.0124, which were significantly higher than the multiple blood flow guided model.

[0075] This performance difference can be attributed to the increased complexity and potential noise introduced by multiple blood flow inputs, which can overwhelm the model and hinder its ability to focus on relevant features. In contrast, a single blood flow guide provides a more streamlined and focused input, allowing the model to better capture and utilize key information, thereby improving the quality of generated images.

[0076] These evaluation metrics, combined with visual inspection, provide a robust validation framework to assess the effectiveness and generalization ability of the proposed ultrasound image synthesis model in generating high-quality contrast-enhanced ultrasound images. The ultrasound image synthesis model constructed by the present invention was developed using PyTorch and trained on an NVIDIA RTXA4000 GPU. The training was performed using the Adam optimizer with 1,000,000 steps, a batch size of 1, a learning rate of 0.0002, and a total of 200 epochs.

[0077] During training, the constructed data set can be divided into a training set and a validation set according to a certain ratio, for example, 80% of the data is used as the training set and 20% of the data is used as the validation set. The training set is used for parameter optimization and updating network weights, and the validation set is used to evaluate the performance and generalization ability of the model.

[0078] After the ultrasound image synthesis model is trained, the specific method of using the ultrasound image synthesis model to predict contrast-enhanced ultrasound images is as follows: 1. Collect the standard grayscale ultrasound image video and the corresponding blood flow image of the current clinical patient, and split the standard grayscale ultrasound image video into a single standard grayscale ultrasound image frame by frame.

[0079] 2. Convert all standard grayscale ultrasound images and blood flow images into corresponding standard grayscale ultrasound image sequences and blood flow image sequences respectively, and then input them into the trained ultrasound image synthesis model at the same time.

[0080] 3. The second encoder in the ultrasound image synthesis model reads each input standard grayscale ultrasound image sequence, and performs time series processing on each standard grayscale ultrasound image sequence through its ConvLSTM layer, thereby outputting a context vector of a fixed length, wherein the context vector contains the ultrasound information of the input standard grayscale ultrasound image sequence.

[0081] 4. The first encoder in the ultrasound image synthesis model reads the blood flow image sequence and performs time series processing on the blood flow image sequence through the ConvLSTM layer to which it belongs, thereby outputting an additional vector of a fixed length, wherein the additional vector contains the blood flow information (physiological data) of the input blood flow image sequence.

[0082] 5. The encoder in the ultrasound image synthesis model takes the context vector output by the second encoder as the basic input, and the additional vector output by the first encoder as the guide signal, and performs time series processing on the context vector through the ConvLSTM layer to which it belongs, and finally gradually outputs a contrast-enhanced ultrasound image sequence corresponding to the input standard grayscale ultrasound image sequence, and the contrast-enhanced ultrasound image sequence is generated based on the control blood flow information input.

[0083] 6. Convert all synthesized contrast-enhanced ultrasound image sequences into contrast-enhanced ultrasound images, and finally display multiple contrast-enhanced ultrasound images or a contrast-enhanced ultrasound image video on the screen, providing doctors with intuitive display to support clinical diagnosis and research applications.

[0084] The present invention also provides a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned ultrasound image synthesis method based on sequence-to-sequence model and blood flow information.

[0085] The present invention also provides a computer-readable storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned ultrasound image synthesis method based on a sequence-to-sequence model and blood flow information.

[0086] From the overall effect point of view, the present invention provides an ultrasound image generation model controlled by blood flow information. Through preprocessing, the data set is organized into data suitable for model input, and control conditions are extracted from the original information. The ultrasound image generation model can quickly convert the original standard grayscale image into a corresponding contrast-enhanced ultrasound image according to the patient's specific blood flow dynamics. The present invention utilizes the powerful learning ability of the Seq2Seq model, and has the characteristics of global consistency, local diversity and controllability of the image. It can assist in the training of clinicians and provide a large amount of synthetic data support for the training of deep learning models.

[0087] From the perspective of partial effects, the blood flow information integration method proposed in the present invention enables the ultrasound image generation model to make full use of limited data sets for training, and accurately synthesizes ultrasound images with controllable appearance through blood flow information, which is easy to operate and user-friendly. By adapting encoders to different control conditions, the ultrasound image generation model can effectively extract the characteristics of blood flow information and learn the relationship between control conditions and generated images. The ultrasound image generation model architecture adopts the Seq2Seq model and multiple attention mechanisms, thereby improving the generation effect and the clinical relevance of the image.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for synthesizing ultrasound images based on a sequence-to-sequence model and blood flow information, characterized in that: The following steps are involved: Step 1) Obtain several standard grayscale ultrasound image videos of lesions of previous clinical patients and the corresponding contrast-enhanced ultrasound image videos and blood flow images, and use them to construct a data set for model training; Step 2) using the data set, training the constructed ultrasound image synthesis model, and training the model to automatically convert the standard grayscale ultrasound image video into the corresponding contrast-enhanced ultrasound image video under the guidance of the corresponding blood flow information; Step 3) collecting the standard grayscale ultrasound image video and the corresponding blood flow image of the current clinical patient, and simultaneously inputting them into the trained ultrasound image synthesis model; Step 4) the ultrasound image synthesis model performs enhancement processing on the standard grayscale ultrasound image video under the guidance of the blood flow information contained in the blood flow image, thereby synthesizing the corresponding contrast-enhanced ultrasound image video; Step 5) output and display the synthesized contrast-enhanced ultrasound image video.

2. The ultrasound image synthesis method based on sequence-to-sequence model and blood flow information according to claim 1, characterized in that: In step 1), the method for constructing the data set is: 1) Collect several standard grayscale ultrasound image videos of the lesion sites of previous clinical patients; 2) Collect contrast-enhanced ultrasound image videos of the lesion site of each previous clinical patient; 3) Collect blood flow images of the lesion site of each previous clinical patient; 4) Standard grayscale ultrasound image videos, contrast-enhanced ultrasound image videos, and blood flow images belonging to the same previous clinical patient are sorted and paired to construct the data set.

3. The ultrasound image synthesis method based on sequence-to-sequence model and blood flow information according to claim 1, characterized in that: In step 2), the ultrasound image synthesis model uses a sequence-to-sequence (Seq2Seq) model with an attention mechanism as the basic architecture, and its specific structure includes: A first encoder with a ConvLSTM layer is responsible for reading an input blood flow image sequence, and performing time series processing on the blood flow image sequence through the ConvLSTM layer to which it belongs, so as to output an additional vector, wherein the additional vector contains blood flow information of the input blood flow image sequence; The second encoder with the ConvLSTM layer is responsible for reading each input standard grayscale ultrasound image sequence, and performing time series processing on each standard grayscale ultrasound image sequence through the ConvLSTM layer to which it belongs, so as to output a context vector, wherein the context vector contains ultrasound information of the input standard grayscale ultrasound image sequence; A decoder with a ConvLSTM layer is responsible for taking the context vector output by the second encoder as a basic input, taking the additional vector output by the first encoder as a guide signal, and performing time-series processing on the context vector through the ConvLSTM layer to which it belongs, and finally gradually decoding and outputting a contrast-enhanced ultrasound image sequence corresponding to the input standard grayscale ultrasound image sequence; The attention mechanism is responsible for helping the decoder to refine the encoded features from the ultrasound information and blood flow information when decoding the context vector, so as to enhance the ability of the ultrasound image synthesis model to selectively focus on relevant areas affected by hemodynamics, thereby improving the quality and clinical relevance of the generated contrast-enhanced ultrasound image sequence.

4. The ultrasound image synthesis method based on sequence-to-sequence model and blood flow information according to claim 3, characterized in that: The internal structures of the second encoder and the decoder are respectively: The first encoder and the second encoder both include two convolutional layers and five residual blocks connected in sequence; The decoder includes four residual blocks and three super-convolutional layers connected in sequence, and an upsampling layer arranged between two adjacent super-convolutional layers.

5. The ultrasound image synthesis method based on sequence-to-sequence model and blood flow information according to claim 3, characterized in that: In step 2), the training method of the ultrasound image synthesis model is: 1) Data preparation; Selecting several standard grayscale ultrasound image videos from the data set, and randomly selecting a certain number of video frames from each of the selected standard grayscale ultrasound image videos as standard grayscale ultrasound images for training; At the same time, in the data set, a contrast-enhanced ultrasound image and a blood flow image corresponding to the randomly selected standard grayscale ultrasound image are selected; 2) Data preprocessing; Performing preprocessing including clipping and normalization on the selected standard grayscale ultrasound images, contrast-enhanced ultrasound images and the corresponding blood flow images; 3) Sequence transformation; converting the preprocessed standard grayscale ultrasound image, contrast-enhanced ultrasound image and the corresponding blood flow image into a standard grayscale ultrasound image sequence, a contrast-enhanced ultrasound image sequence and the corresponding blood flow image sequence respectively; 4) Introduction of attention mechanism; An attention mechanism is introduced to help the decoder refine the encoded features from ultrasound information and blood flow information when decoding the context vector, so as to enhance the ability of the ultrasound image synthesis model to selectively focus on relevant areas affected by hemodynamics, thereby improving the quality and clinical relevance of the generated contrast-enhanced ultrasound image sequence. 5) Model training; Taking a preprocessed standard grayscale ultrasound image sequence as a basic input, a preprocessed contrast-enhanced ultrasound image sequence as a target output, and a preprocessed and labeled blood flow image sequence as an additional input, the ultrasound image synthesis model is guided by the blood flow image sequence to learn a mapping relationship from the standard grayscale ultrasound image sequence to the contrast-enhanced ultrasound image sequence, so as to achieve prediction and generation of the contrast-enhanced ultrasound image sequence; 6) Loss calculation; Using a structural coherence loss function, calculating the difference between the contrast-enhanced ultrasound image sequence and the standard grayscale ultrasound image sequence, and adjusting the parameters of the first encoder, the second encoder and the decoder according to the calculated loss to minimize the loss; 7) Evaluation and adjustment; After the training is completed, a specific evaluation index is used to evaluate the performance of the ultrasound image synthesis model, and the ultrasound image synthesis model is further adjusted according to the evaluation result.

6. The ultrasound image synthesis method based on sequence-to-sequence model and blood flow information according to claim 5, characterized in that: The key factors comprehensively considered by the structural coherence loss function include: 1) Smoothing and transition issues; 2) Structural integrity issues; 3) Time consistency problem.

7. The ultrasound image synthesis method based on sequence-to-sequence model and blood flow information according to claim 5, characterized in that: The evaluation indicators include: 1) Peak signal-to-noise ratio; 2) Structural similarity index measurement; 3) Visual inspection results.

8. The ultrasound image synthesis method based on sequence-to-sequence model and blood flow information according to claim 3, characterized in that: The specific method of step 3) to step 5) is: 1) Collect the standard grayscale ultrasound image video and corresponding blood flow image of the current clinical patient, and split the standard grayscale ultrasound image video into a single standard grayscale ultrasound image frame by frame; 2) converting all standard grayscale ultrasound images and blood flow images into corresponding standard grayscale ultrasound image sequences and blood flow image sequences respectively, and then simultaneously inputting the trained ultrasound image synthesis model; 3) The second encoder in the ultrasound image synthesis model reads each input standard grayscale ultrasound image sequence, and performs time series processing on each standard grayscale ultrasound image sequence through the ConvLSTM layer to which it belongs, thereby outputting a context vector, wherein the context vector contains ultrasound information of the input standard grayscale ultrasound image sequence; 4) The first encoder in the ultrasound image synthesis model reads the blood flow image sequence, and performs time series processing on the blood flow image sequence through the ConvLSTM layer to which it belongs, thereby outputting an additional vector, wherein the additional vector contains blood flow information of the input blood flow image sequence; 5) The encoder in the ultrasound image synthesis model uses the context vector output by the second encoder as a basic input, uses the additional vector output by the first encoder as a guide signal, and performs time-series processing on the context vector through the ConvLSTM layer to which it belongs, and finally gradually decodes and outputs a contrast-enhanced ultrasound image sequence corresponding to the input standard grayscale ultrasound image sequence; 6) Convert all synthesized contrast-enhanced ultrasound image sequences into contrast-enhanced ultrasound images, and finally display multiple contrast-enhanced ultrasound images or a contrast-enhanced ultrasound image video to provide an intuitive display for doctors.

9. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to perform operations corresponding to the ultrasound image synthesis method based on a sequence-to-sequence model and blood flow information as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the ultrasound image synthesis method based on a sequence-to-sequence model and blood flow information as described in any one of claims 1-8.

Images