Ultrasound image processing method, device, equipment and medium based on style transfer

Abstract

The present invention discloses a method, apparatus, device, and medium for ultrasound image processing based on style transfer. The method comprises the following steps: collecting multi-source ultrasound images and annotated images of the multi-source ultrasound images from a hospital; extracting content information and style information from the multi-source ultrasound images; determining the migration ratio of the content information and style information based on a migration parameter, and generating a style-transferred ultrasound image based on the migration ratio; inputting the multi-source ultrasound images and the style-transferred ultrasound images into an evaluation network to obtain a binarized ultrasound image; adaptively determining the migration parameter value based on the evaluation results of the binarized ultrasound image and the annotated images of the multi-source ultrasound image, and generating a style-transferred ultrasound image. This method addresses the problem of reduced generalization of fetal head ultrasound segmentation methods due to data obtained from different machines, settings, and operations. It introduces two dimensions of information, content information and style information, to achieve better multi-source ultrasound data segmentation performance.

Description

Technical Field

[0001] The present invention relates to the fields of translational medicine and artificial intelligence technology, and in particular to an ultrasound image processing method, device, equipment and medium based on style transfer. Background Art

[0002] Algorithms play a crucial role in the current field of medical artificial intelligence, with data having a significant impact on them. Currently, most algorithms are developed specifically to solve a specific problem. While these algorithms can achieve good results on their datasets, different datasets often differ to varying degrees, and a method proposed based on one dataset may not be equally effective on another. On the one hand, the generalization performance of the method is limited by the differences in the quantity and quality of different datasets. On the other hand, the proposed method may not account for the differences between datasets generated by various factors (such as different machines, different settings, different operators, etc.).

[0003] Current style transfer methods explicitly aim to enhance content information within images with a specific style. Consequently, the trained models are limited to the specific target style and cannot generalize to new styles. Alternatively, they rely on complex adversarial training to learn the target domain, resulting in overly complex and cumbersome models. Therefore, a style transfer-based ultrasound image processing method is urgently needed to eliminate differences between multi-source datasets and ensure model robustness.

[0004] Invention content information

[0005] The present invention aims to address the aforementioned shortcomings in the prior art by providing an ultrasound image processing method, apparatus, device, and medium based on style transfer. The overall network architecture consists of two modules: a data reconstruction unit, a backbone module, and an adaptive unit. The backbone module uses a multi-layer network structure to extract multi-scale features from ultrasound images. The data reconstruction unit and the adaptive unit incorporate contextual information into the features to address intra-class inconsistencies in the segmentation results. The overall network is connected sequentially in the following order: data reconstruction unit, backbone module, and adaptive unit. The adaptive unit feeds back style transfer parameters to the data reconstruction unit. The data reconstruction unit utilizes a transfer method based on Fourier domain transform. The source domain image is adjusted in phase and amplitude space, addressing the problem of uneven cross-domain segmentation. The adaptive unit automatically adjusts parameter values ​​that influence style and content information through a discriminator, eliminating the need for manual parameter setting. Furthermore, the present invention can be readily transferred to other data, making it a universal data processing method. Therefore, the present invention can significantly reduce the data dependency of different models, reducing the workload of ultrasound physicians in data acquisition and processing, and adaptively matching different models.

[0006] A first object of the present invention is to provide an ultrasound image processing method based on style transfer, the ultrasound image processing method comprising the following steps:

[0007] S1. Collect multi-source ultrasound images and annotated images of multi-source ultrasound images;

[0008] S2, extracting content information and style information of multi-source ultrasound images;

[0009] S3. Determine a migration ratio of the content information and the style information according to the migration parameter, and generate an ultrasound image after style migration according to the migration ratio;

[0010] S4. Input the multi-source ultrasound image and the ultrasound image after style transfer into the evaluation network to obtain a binarized ultrasound image. Adaptively determine the migration parameter values ​​based on the evaluation results of the binarized ultrasound image and the annotated image of the multi-source ultrasound image, and determine the ultrasound image after style transfer.

[0011] Furthermore, in step S1, data obtained from different medical instruments, different medical instrument settings, and different operating processes are collected. Since there may be large differences in grayscale value distribution and color contrast, it is necessary to collect multi-source ultrasound images and perform image annotation on the collected multi-source ultrasound images.

[0012] Furthermore, the multi-source ultrasound images are first processed in the Fourier domain. The resulting ultrasound image content information is amplitude information in the Fourier domain, and the image style information is phase information in the Fourier domain. Amplitude information is relatively intuitive. While phase information is less intuitive, it can reveal a lot of information that amplitude information cannot clearly express. Using the Fourier domain, information in ultrasound images can be learned from multiple perspectives.

[0013] Furthermore, the data reconstruction unit uses a migration method based on Fourier domain transform, treating data from different sources in the multi-source ultrasound image as the source domain image and target domain image, respectively. Fast Fourier transform is used to extract the amplitude information and phase information of the source domain image and target domain image, respectively. Using style migration parameters, the amplitude information of the central portion of the source domain image is replaced with the amplitude information of the central portion of the target domain image, and the phase information of the central portion of the target domain image is replaced with the phase information of the central portion of the target domain image. The processed amplitude information and phase information are converted into an ultrasound image using an inverse Fourier transform (IDFT). The data reconstruction unit primarily extracts information from the image and, through migration between data from different sources, brings the distribution of data from two different sources closer to each other.

[0014] After obtaining the style-transferred ultrasound image data, the original target domain ultrasound image and the style-transferred ultrasound image are input into the backbone module. The backbone module uses deep learning to extract features from the input ultrasound image data. This module learns the corresponding information in the ultrasound image, thereby obtaining a nonlinear expression for the one-to-one correspondence between the ultrasound image and its corresponding annotated image. The backbone module iterates through successive iterations to continuously approximate the nonlinear expression that best fits the current data. The backbone module primarily learns the low-dimensional information of the ultrasound image through downsampling and convolutional blocks. This approach uses a feedback mechanism to automatically adjust parameters without requiring manual parameter setting.

[0015] Furthermore, the loss value L dice The loss value is used to adjust the style transfer parameter index. The binary ultrasound image obtained by the backbone module contains information about the location and region of the corresponding anatomical structures in the multi-source ultrasound image. Combined with the annotation information of the multi-source ultrasound image in S1, the performance of the backbone module can be evaluated. Continuous iteration continuously optimizes the ultrasound image after style transfer input to the backbone module, thereby continuously optimizing the model and determining the ultrasound image after style transfer.

[0016] A second object of the present invention is to provide an ultrasonic image processing device based on style transfer, the ultrasonic image processing device comprising:

[0017] A collection and annotation module is used to collect multi-source ultrasound images and annotated images of the multi-source ultrasound images;

[0018] An information extraction module, used to extract content information and style information of multi-source ultrasound images;

[0019] a transfer generation module, configured to determine a transfer ratio of the content information and the style information according to a transfer parameter, and generate an ultrasound image after style transfer according to the transfer ratio;

[0020] The evaluation processing module is used to input the multi-source ultrasound image and the ultrasound image after style transfer into the evaluation network to obtain a binarized ultrasound image, adaptively determine the migration parameter values ​​based on the evaluation results of the binarized ultrasound image and the annotated image of the multi-source ultrasound image, and determine the ultrasound image after style transfer.

[0021] A third object of the present invention is to provide a computer device comprising a processor and a memory for storing a program executable by the processor, wherein when the processor executes the program stored in the memory, an ultrasound image processing method based on style transfer is implemented.

[0022] A fourth object of the present invention is to provide a storage medium storing a program, which, when executed by a processor, implements an ultrasound image processing method based on style transfer.

[0023] The invention has the following advantages and effects compared with the prior art:

[0024] This invention adjusts the source domain image in phase and amplitude space, solving the problem of uneven cross-domain segmentation. A discriminator automatically adjusts the parameter values ​​that influence style and content information, eliminating the need for manual parameter setting. Furthermore, this method can be readily transferred to other data, making it a universal data processing method. Therefore, this invention can significantly reduce the data dependency of different models, reducing the workload of sonographers in data acquisition and processing, and adaptively matching different models. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 is a flow chart of an ultrasound image processing method based on style transfer disclosed in the present invention;

[0027] Figure 2 is a schematic diagram of the network structure of an evaluation network according to an embodiment of the present invention;

[0028] Figure 3 is a schematic diagram of data generation according to an embodiment of the present invention;

[0029] Figure 4 It is the effect diagram of different migration parameters of the present invention;

[0030] Figure 5 This is a schematic diagram of data distribution before data processing in an embodiment of the present invention;

[0031] Figure 6 is a schematic diagram of data distribution after data processing in an embodiment of the present invention;

[0032] Figure 7 is a structural block diagram of an ultrasonic image processing device in Example 2 of the present invention;

[0033] Figure 8 This is a structural block diagram of the computer device in Example 3 of the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0035] Example 1:

[0036] This embodiment discloses an ultrasound image processing method based on style transfer, which mainly includes the following steps:

[0037] S1. Collect multi-source ultrasound images and annotated images of multi-source ultrasound images;

[0038] The data collected in step S1 are from multi-source ultrasound images acquired medically, and are specifically in the form of data obtained from different medical instruments, different medical instrument settings, and different operating procedures.

[0039] S2, extracting content information and style information of multi-source ultrasound images;

[0040] The content information is the amplitude information in the Fourier domain, and the style information is the phase information in the Fourier domain. The process of extracting the content information and style information is as follows.

[0041] S21. Extract content information and phase information of the multi-source ultrasound image respectively.

[0042]

[0043] Among them, F(x i ) represents the Fourier transform of (u, v), j is an imaginary number, h and w represent the height and width of the image respectively. (u, v) represents the horizontal and vertical coordinates of the two-dimensional matrix point position in the multi-source ultrasound image. After the Fourier transform, the image is symmetrical around the frequency rectangle. Define M α 、 For the two-dimensional mask in the Fourier domain, use the two-dimensional mask M α 、M β The non-zero region (i.e., the region where amplitude and phase information is retained) is divided by the value of , where the initial values ​​of α and β are random numbers between 0 and 1:

[0044]

[0045] S22. For a given target domain image and source domain image, the following formula is used to process the content information and style information, and the source domain image after style transfer is output. This is the image result after style transfer.

[0046]

[0047] Specifically, the source domain image represents the ultrasound image of the current style, and the target domain image represents the ultrasound image of the desired style. The phase and amplitude information mapping is determined by the migration parameter to determine the mapping ratio.

[0048] Among them, F -1 Represents the inverse Fourier transform, which maps the amplitude and phase of the spectrum signal back to the original two-dimensional matrix space. and Represent the phase information and amplitude information of the source domain image respectively, and Represent the phase information and amplitude information of the target domain image respectively, Represents matrix multiplication. It can be seen from the formula that the source image phase The low-frequency part and source image amplitude part of the target image Then, replace the modified The spectral representation of the image is mapped back to Its content information is close to and will be similar to The specific results are shown as follows. Figure 3 . Among them, different parameters (i.e., content information mapping parameter α and style information mapping parameter β) are used to map the source image to the target image to generate the preprocessed source image. α (from left to right) and β (from top to bottom) increase from 0 to 0.25, and the final preprocessed source image is marked with a green rectangle. A, B, C, and D represent examples from four different source ultrasound datasets. Figure 3 Figure 1 shows example images obtained after adaptive style transfer. In the first row, the four original images are from datasets A, B, C, and D. In the second column, image A is the target image, and the source domain images (i.e., B, C, and D) are mapped to image A, obtaining preprocessed images (i.e., B#, C#, and D#) after adaptive style transfer. In the following columns, B, C, and D are used as target images, respectively. Images different from the target images are considered source images and preprocessed using adaptive style transfer. # Represents the mapped image based on the target image, that is, the image after style transfer.

[0049] S3. Determine a migration ratio of the content information and the style information according to the migration parameter, and generate an ultrasound image after style migration according to the migration ratio;

[0050] In step S3, different style information datasets will be generated according to different scale parameters, such as Figure 4The new dataset style information is generated for one set of different source data according to different scale parameters, where α and β represent two style information migration parameters.

[0051] S4. Input the multi-source ultrasound image and the ultrasound image after style transfer into the evaluation network to obtain a binarized ultrasound image. Adaptively determine the migration parameter values ​​based on the evaluation results of the binarized ultrasound image and the annotated image of the multi-source ultrasound image, and determine the ultrasound image after style transfer.

[0052] In step S4, the U-Net convolutional neural network is used to evaluate the segmentation effect of the model. The Adam optimizer is used in the model, the initial learning rate is set to 0.0003, the number of iterations is set to 200, and the training batch size is set to 16.

[0053] During training, the model loss on the validation set is continuously monitored to adjust the learning rate and other parameters appropriately. For greater accuracy, the best Dice coefficient result on the validation set is used as the final model. All data is resized to 256×256 pixels during training to accommodate the hardware environment, and all data is divided into three groups with an approximate ratio of 4:1:1: training, validation, and testing. Images with the same parent data are assigned to the same dataset to avoid the influence of dataset similarity on the results. Finally, the network output consists of a probability map representing the background and foreground, which is used to evaluate model performance.

[0054] In this embodiment, step S4 is adjusted according to the different scale parameters initially obtained in step S3. Given an adaptive dataset and labels, a semantic segmentation network with can be trained by minimizing the cross entropy loss.

[0055] In this embodiment, the adaptive mode process is as follows. The adaptive mode is executed before model training. (s) and target dataset X (t) Merge into the mixed generated dataset X mix =X (s) ∪X (t) Generate a new domain transfer dataset X in adaptive mode (f) We used a composite data processing approach:

[0056]

[0057] Where i represents the number of iterations in the training process. represents the dataset after style transfer, represents the source domain dataset, represents the number of images in the dataset, represents the Fourier domain adaptive function. θ is a custom parameter that operates based on the number of datasets and training batches. The generated new dataset is fed into the network, and the model quality is judged using the Acc and Dice metrics. In adaptive mode, a custom hybrid optimizer is used to update the phase and amplitude adjustment parameters based on the gradient information propagated back through the network to further reduce the loss value.

[0058] In summary, this embodiment provides an ultrasound image processing method based on style transfer, which visualizes the significant changes in the data distribution after processing through the t-SNE method. Where △ and * are two data sets from different sources. Figure 5 This is a diagram of data distribution before style migration. Figure 6 This is a diagram of data distribution after style transfer. Figure 5 and Figure 6 It can be clearly seen that the domain gap of the dataset after style transfer becomes smaller, which can effectively reduce the disadvantage of poor model generalization caused by the domain gap of data obtained from different medical instruments, different medical instrument settings and different operation processes, and obtain the most appropriate style transfer image.

[0059] Example 2:

[0060] like Figure 7 As shown, this embodiment provides an ultrasound image processing device based on style transfer, which includes a collection and annotation module 701, an information extraction module 702, a transfer generation module 703, and an evaluation and processing module 704. The specific functions of each module are as follows:

[0061] A collection and annotation module 701 is used to collect multi-source ultrasound images and annotated images of the multi-source ultrasound images;

[0062] An information extraction module 702 is configured to extract content information and style information of a multi-source ultrasound image;

[0063] a transfer generation module 703, configured to determine a transfer ratio of the content information and the style information according to a transfer parameter, and generate an ultrasound image after style transfer according to the transfer ratio;

[0064] The evaluation processing module 704 is configured to input the multi-source ultrasound image and the ultrasound image after style transfer into an evaluation network to obtain a binarized ultrasound image, adaptively determine the values ​​of the transfer parameters based on the evaluation results of the binarized ultrasound image and the annotated image of the multi-source ultrasound image, and determine the ultrasound image after style transfer.

[0065] The specific implementation of each module in this embodiment can be found in the above-mentioned embodiment 1, and will not be described one by one here; it should be noted that the device provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0066] Example 3:

[0067] This embodiment provides a computer device, which can be a computer, such as Figure 8 As shown, a processor 802, a memory, an input device 803, a display 804, and a network interface 805 are connected via a system bus 801. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 806 and an internal memory 807. The non-volatile storage medium 806 stores an operating system, a computer program, and a database. The internal memory 807 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 802 executes the computer program stored in the memory, an ultrasound image processing method proposed in the above-mentioned embodiment 1 is implemented, which includes the following steps:

[0068] S1. Collect multi-source ultrasound images and annotated images of multi-source ultrasound images;

[0069] S2, extracting content information and style information of multi-source ultrasound images;

[0070] S3. Determine a migration ratio of the content information and the style information according to the migration parameter, and generate an ultrasound image after style migration according to the migration ratio;

[0071] S4. Input the multi-source ultrasound image and the ultrasound image after style transfer into the evaluation network to obtain a binarized ultrasound image. Adaptively determine the migration parameter values ​​based on the evaluation results of the binarized ultrasound image and the annotated image of the multi-source ultrasound image, and determine the ultrasound image after style transfer.

[0072] Example 4:

[0073] This embodiment provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the ultrasound image processing method of the above embodiment 1 is implemented as follows:

[0074] S1. Collect multi-source ultrasound images and annotated images of multi-source ultrasound images;

[0075] S2, extracting content information and style information of multi-source ultrasound images;

[0076] S3. Determine a migration ratio of the content information and the style information according to the migration parameter, and generate an ultrasound image after style migration according to the migration ratio;

[0077] S4. Input the multi-source ultrasound image and the ultrasound image after style transfer into the evaluation network to obtain a binarized ultrasound image. Adaptively determine the migration parameter values ​​based on the evaluation results of the binarized ultrasound image and the annotated image of the multi-source ultrasound image, and determine the ultrasound image after style transfer.

[0078] The storage medium described in this embodiment can be a magnetic disk, an optical disk, a computer memory, a random access memory (RAM), a USB flash drive, a mobile hard disk, or other media.

[0079] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for processing ultrasound images based on style transfer, characterized in that: The ultrasonic image processing method comprises the following steps: S1. Collect multi-source ultrasound images and annotated images of multi-source ultrasound images; S2, extracting content information and style information of multi-source ultrasound images; S3. Determine a migration ratio of the content information and the style information according to the migration parameter, and generate an ultrasound image after style migration according to the migration ratio; S4. Inputting the multi-source ultrasound image and the ultrasound image after style transfer into an evaluation network to obtain a binarized ultrasound image, adaptively determining migration parameter values ​​based on evaluation results of the binarized ultrasound image and the annotated image of the multi-source ultrasound image, and determining the ultrasound image after style transfer; The network structure of the evaluation network is a data reconstruction unit, a backbone module, and an adaptive unit connected in sequence, wherein the adaptive unit feeds back the style migration parameters to the data reconstruction unit; The data reconstruction unit adopts a migration method based on Fourier domain transform, uses data from different sources in the multi-source ultrasound image as the source domain image and the target domain image, and adjusts the source domain image in the content and style space according to the migration ratio based on the content information and style information of the target domain image. The process is as follows: input the source domain image and the target domain image, use fast Fourier transform (FFT) to extract the phase information and amplitude information of the source domain image and the target domain image, map the phase information and amplitude information of the target domain image to the source domain image according to the migration parameters, and restore the source domain image after style migration through inverse Fourier transform (IDFT); The style-transferred source domain image and the original target domain image are input into the backbone module, where the backbone module uses a feature extraction network to extract multi-scale features from the style-transferred source domain image and the original target domain image and outputs a segmented binary ultrasound image; The input of the adaptive unit is the binarized ultrasound image output by the backbone module and the annotated image of step S1. According to the loss values ​​of the binarized ultrasound image and the annotated image of S1, the gradient information is back-propagated to update the phase adjustment parameters and amplitude adjustment parameters of the data reconstruction unit to further reduce the loss value of the adaptive unit, thereby determining the migration parameter value, and returning to step S3.

2. The ultrasonic image processing method based on style transfer according to claim 1, characterized in that: The data collected in step S1 are from multi-source ultrasound images acquired medically, and are specifically in the form of data obtained from different medical instruments, different medical instrument settings, and different operating procedures.

3. The ultrasonic image processing method based on style transfer according to claim 1, characterized in that: The specific structure of the backbone module is as follows: The connections from the input layer to the output layer are: conv2d+BN+Relu with a convolution kernel size of 3x3 and 64 kernels; maxpool layer with a stride of 2; conv2d+BN+Relu with a convolution kernel size of 3x3 and 128 kernels; maxpool layer with a stride of 2; conv2d+BN+Relu with a convolution kernel size of 3x3 and 256 kernels; maxpool layer with a stride of 2; conv2d+BN+Relu with a convolution kernel size of 3x3 and 512 kernels; maxpool layer with a stride of 2; convolution kernel size of 3x3 and 1024 kernels. conv2d+BN+Relu; deconv layer with a stride of 2; conv2d+BN+Relu with a convolution kernel size of 3x3 and 512 kernels; deconv layer with a stride of 2; conv2d+BN+Relu with a convolution kernel size of 3x3 and 256 kernels; deconv layer with a stride of 2; conv2d+BN+Relu with a convolution kernel size of 3x3 and 128 kernels; deconv layer with a stride of 2; conv2d+BN+Relu with a convolution kernel size of 3x3 and 64 kernels; convolution layer conv with a convolution kernel size of 1x1 and 1 kernel.

4. The ultrasonic image processing method based on style transfer according to claim 3, characterized in that: The loss value L dice The loss function is calculated as follows: Wherein, X and Y represent the binary ultrasound image output by the backbone module and the annotated image in step S1, respectively.

5. The ultrasonic image processing method based on style transfer according to claim 1, characterized in that: In step S4, the evaluation network corresponding to the best loss value on the validation set is used as the final model, and the generated new data set is input into the evaluation network during the iteration of the training process to obtain the processed ultrasound image.

6. An ultrasonic image processing device based on the ultrasonic image processing method based on style transfer according to any one of claims 1 to 5, characterized in that: The ultrasonic image processing device comprises: A collection and annotation module is used to collect multi-source ultrasound images and annotated images of the multi-source ultrasound images; An information extraction module, used to extract content information and style information of multi-source ultrasound images; a transfer generation module, configured to determine a transfer ratio of the content information and the style information according to a transfer parameter, and generate an ultrasound image after style transfer according to the transfer ratio; The evaluation processing module is used to input the multi-source ultrasound image and the ultrasound image after style transfer into the evaluation network to obtain a binarized ultrasound image, adaptively determine the migration parameter values ​​based on the evaluation results of the binarized ultrasound image and the annotated image of the multi-source ultrasound image, and determine the ultrasound image after style transfer.

7. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the ultrasound image processing method based on style transfer according to any one of claims 1 to 5 is implemented.

8. A storage medium storing a program, characterized in that: When the program is executed by a processor, the ultrasound image processing method based on style transfer according to any one of claims 1 to 5 is implemented.

Images