An ultrasonic image improvement method based on artificial intelligence

Abstract

The present invention discloses an ultrasonic image improvement method based on artificial intelligence, which belongs to the field of medical image processing, and includes: step S1, segmenting the ultrasonic image to obtain a first structural feature map and a first tissue feature map; step S2, binarizing the first structural feature map and performing morphological expansion processing; and decomposing the first tissue feature map into multiple layers, removing the tissue features of the middle layer, and obtaining a second tissue feature map; step S3, migrating the features of the tomographic image and the magnetic resonance image to the ultrasonic image to obtain a third structural feature map and a third tissue feature map; step S6, fusing the third structural feature map, the third tissue feature map and the ultrasonic image. The beneficial effect of the present invention is that by extracting the features of the tomographic image and the magnetic resonance image and migrating them to the ultrasonic image, and performing fusion processing, the speckle noise of the ultrasonic image is suppressed, and the structural expression of the image is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to an ultrasonic image improvement method based on artificial intelligence. Background Art

[0002] B-ultrasound, as the main method of ultrasonic examination, is an important means of medical examination. Due to its low cost and convenience, and almost no radiation damage, more and more doctors choose to use B-ultrasound as a means of initial screening of patients, making the scope and region of application of ultrasound become wider.

[0003] However, the spatial resolution of the ultrasound beam is limited, the reflection surface of the tissue is not smooth, the echo signals generated by different reflection sources have different phases, they may be superimposed or cancelled when the beam is formed, and the noise of the electronic circuit and other factors lead to the presence of a lot of speckle noise in the ultrasound image. Due to the presence of speckle noise, the contrast of the ultrasound image and the detail information available in the tissue are reduced, making it difficult to identify and analyze the image details, thus interfering with the doctor's diagnosis.

[0004] In order to improve the quality of ultrasound images, different regions are obtained by segmenting ultrasound images in the prior art, and different image processing methods are used for different regions. For example, according to the tissue structure contained in the ultrasound image, the ultrasound image is segmented into a structural region and a non-structural region, and the structural region is anisotropically processed and the non-structural region is isotropically processed to achieve the purpose of enhancing the edge of the ultrasound image, thereby improving the quality of the ultrasound image. However, this processing method will cause obvious differences between different regions, and the real details may be lost, and the ultrasound image does not appear natural. Therefore, in view of the above problems, it is urgent to design an ultrasound image improvement method based on artificial intelligence to meet the needs of actual use. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an ultrasound image improvement method based on artificial intelligence.

[0006] The technical problem solved by the present invention can be achieved by adopting the following technical solutions:

[0007] The present invention provides an ultrasonic image improvement method based on artificial intelligence, comprising:

[0008] Step S1, extracting structural information of an ultrasonic image, and segmenting the ultrasonic image according to the extracted structural information to obtain a first structural feature map and a first tissue feature map;

[0009] Step S2, binarizing the first structural feature map, and performing morphological dilation processing on the processed first structural feature map to obtain a second structural feature map; and

[0010] Decomposing the first tissue feature map into multiple layers, and removing tissue features of a middle layer of the first tissue feature map to obtain a second tissue feature map;

[0011] Step S3, providing a tomographic image corresponding to the ultrasound image, annotating the corresponding structural features on the tomographic image through the trained structural optimization network, and migrating the annotated structural features to the second structural feature map to obtain a third structural feature map; and

[0012] Providing a magnetic resonance image corresponding to the ultrasound image, annotating corresponding tissue features on the magnetic resonance image through a trained tissue optimization network, and migrating the annotated tissue features to the second tissue feature map to obtain a third tissue feature map;

[0013] Step S4, fusing the third structural feature map, the third tissue feature map and the ultrasound image to obtain the optimized ultrasound image.

[0014] Preferably, in step S1, extracting structural information of the ultrasound image specifically includes:

[0015] The ultrasonic image is extracted based on a Sobel operator or a Canny operator to obtain the structural information.

[0016] Preferably, in the step S1, a segmentation network is used to segment and extract tissue in the ultrasound image to obtain the first structural feature map and the first tissue feature map.

[0017] Preferably, the segmentation network includes a combination of one or more of a deep learning segmentation network, a fully convolutional network, and a semantic segmentation network.

[0018] Preferably, in step S2, binarization processing is performed on the first structural feature map, specifically including:

[0019] Step S21, processing the first structural feature graph based on a Sobel operator or a Prewit operator to obtain a structural factor corresponding to each structural feature in the first structural feature graph;

[0020] Step S22, normalizing the structure factors respectively;

[0021] Step S23 , comparing the normalized structure factor with a preset threshold, and when the absolute value of the structure factor is less than the preset threshold, setting the structure factor of the corresponding structural feature to 0.

[0022] Preferably, in step S2, decomposing the first tissue feature map into multiple layers and removing the tissue features of the middle layer of the first tissue feature map to obtain a second tissue feature map specifically includes:

[0023] The tissue feature map is decomposed by wavelet transform or pyramid segmentation to obtain multi-layer tissue features, and then the tissue features of the middle layer are removed by soft threshold filtering.

[0024] Preferably, in step S3, the trained structure optimization network and the trained organization optimization network respectively include:

[0025] A multi-layer encoding module, each layer of the encoding module comprises a coding unit and a maximum pooling layer, and the input end of the coding unit of the latter layer is connected to the output end of the maximum pooling layer of the previous layer;

[0026] A self-attention cascade module, wherein the self-attention cascade module includes multiple layers of self-attention sub-modules, and the self-attention sub-modules are respectively connected to the output end of the encoding module corresponding to each layer;

[0027] A multi-layer decoding module, each layer of the decoding module comprises a first two-dimensional convolutional layer and an upsampling layer, the first multi-dimensional convolutional layer is connected to the output end of the self-attention submodule corresponding to each layer and the output end of the corresponding upsampling layer, and the input end of the upsampling layer of the previous layer is connected to the output end of the first two-dimensional convolutional layer of the next layer;

[0028] A second two-dimensional convolutional layer, wherein an input end of the second two-dimensional convolutional layer is connected to an output end of the encoding module of the last layer, and an output end of the second two-dimensional convolutional layer is connected to an input end of the decoding module of the last layer;

[0029] The third two-dimensional convolutional layer is connected to the decoding module in the first layer.

[0030] Preferably, the encoding module includes a combination of one or more of a deep convolutional neural network, a residual network, and a dense convolutional network.

[0031] Preferably, the decoding module comprises a two-dimensional matrix convolution layer and a transmission two-dimensional convolution layer.

[0032] Preferably, in step S4, the third structural feature map, the third tissue feature map and the ultrasound image are fused to obtain the optimized ultrasound image. The specific fusion formula is as follows:

[0033] I'=alpha*(S”+T”)+(1-alpha)*I

[0034] in,

[0035] I represents the original ultrasound image;

[0036] alpha represents the fusion factor;

[0037] S" represents the third structural characteristic diagram;

[0038] T" represents the third tissue characteristic diagram;

[0039] I' represents the optimized ultrasound image.

[0040] The beneficial effects of the present invention are:

[0041] The present invention divides the ultrasound image into structural features and tissue features, extracts the image features of the tomography image and migrates them to the structural features of the ultrasound image, extracts the image features of the magnetic resonance image and migrates them to the tissue features of the ultrasound image, and performs fusion processing to suppress the speckle noise of the ultrasound image and improve the structural expression of the image; the migration is performed through the deep learning method, the vertical boundary structure is strengthened, the structure originally parallel to the sound beam is made clearer, and the artifacts caused by the acoustic shadow are effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of an ultrasonic image improvement method based on artificial intelligence in the present invention;

[0043] Figure 2 It is a flowchart of a specific embodiment of performing binarization processing on the first structural feature map in step S2 of the present invention;

[0044] Figure 3 It is a structural block diagram of a specific embodiment of the trained structure optimization network or the organization optimization network in step S3 of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0048] The present invention provides an ultrasonic image improvement method based on artificial intelligence, which belongs to the field of medical image processing. Figure 1 As shown, the following steps are included:

[0049] Step S1, extracting structural information of an ultrasonic image, and segmenting the ultrasonic image according to the extracted structural information to obtain a first structural feature map and a first tissue feature map;

[0050] As a preferred implementation, in step S1, extracting structural information of an ultrasound image, where the ultrasound image includes a B-ultrasound image, specifically includes:

[0051] The ultrasonic image is extracted based on the Sobel operator or the Canny operator to obtain structural information.

[0052] As a preferred implementation, in step S1, a segmentation network is used to segment and extract tissue in the ultrasound image to obtain a first structural feature map and a first tissue feature map.

[0053] As a preferred implementation, the segmentation network includes a combination of one or more of a deep learning segmentation network, a fully convolutional network, and a semantic segmentation network.

[0054] Specifically, in step S1, the structural information of the ultrasound image is first extracted based on the Sobel operator or the Canny operator, and then the ultrasound image is segmented according to the structural information of the ultrasound image through a segmentation network to obtain structural features and tissue features, i.e., a first structural feature map and a first tissue feature map. Preferably, the segmentation network can be a deep learning segmentation network (unity networking, Unet), and / or a fully convolutional network (Fully Convolutional Networks, FCN), and / or a semantic segmentation network Deeplab, etc.

[0055] Step S2, binarizing the first structural feature map, and performing morphological dilation processing on the processed first structural feature map to obtain a second structural feature map;

[0056] Specifically, in step S2, the features in the first structural feature map are binarized, such as Figure 2 As shown, the specific processing steps are as follows:

[0057] Step S21, processing the first structural feature graph based on the Sobel operator or the Prewit operator to obtain a structural factor S(x, y) corresponding to each structural feature in the first structural feature graph, wherein x and y are used to represent the coordinates of each structural feature in the first structural feature graph;

[0058] Step S22, normalizing the structure factors respectively, and normalizing the range of the structure factors to between [-1, 1];

[0059] In step S23, the normalized structure factor is compared with a preset threshold value, and when the absolute value of the structure factor is less than the preset threshold value, the structure factor of the corresponding structural feature is configured to be 0, wherein the preset threshold value is generally 0.08. If the absolute value of the structure factor |S(x, y)| is less than 0.08, the structure factor is set to 0, that is, the structure factor of the corresponding structural feature is 0.

[0060] After binarization, a morphological dilation operation is performed to obtain the second structural feature map.

[0061] Step S2 also includes: decomposing the first tissue feature map into multiple layers, and removing the tissue features of the middle layer of the first tissue feature map to obtain a second tissue feature map;

[0062] As a preferred implementation, in step S2, the tissue feature map is decomposed by wavelet transform or pyramid segmentation to obtain multi-layer tissue features, and then the tissue features of the middle layer are removed by soft threshold filtering.

[0063] In step S3, a tomographic image (such as a CT image) is provided. The tomographic image corresponds one-to-one to the above-mentioned ultrasound image. The corresponding image features on the tomographic image are annotated one by one through a pre-trained structural optimization network. These annotations correspond to spatial points in the ultrasound image. Through the spatial point-to-point correspondence, the image features on the tomographic image are migrated to the structural features of the ultrasound image to obtain a third structural feature map after migration.

[0064] Furthermore, the structure optimization network needs to be trained in advance. The training process is as follows:

[0065] Providing tomographic images and ultrasound images, preprocessing the tomographic images and ultrasound images, including Gauss filtering of ultrasound images, manually labeling the tomographic images and ultrasound images, respectively. The tomographic images and ultrasound images each contain a coordinate system, and the coordinates of each structural feature in the ultrasound image correspond one-to-one to the tomographic image;

[0066] After training with a preset segmentation network, such as a deep learning segmentation network (unity networking, Unet), and / or a fully convolutional network (Fully Convolutional Networks, FCN), and / or a semantic segmentation network such as Deeplab;

[0067] The entity DoG of the tomographic image is extracted. During the training process, the entity DoG of the tomographic image is used as the ground truth of the training network.

[0068] As a preferred embodiment, in step S3, Figure 3 As shown in Figure 1, the trained structure optimization network specifically includes the following parts:

[0069] Multi-layer coding module, each layer of coding module includes a coding unit Basic Block and a maximum pooling layer Max pool, and the input end of the coding unit of the latter layer is connected to the output end of the maximum pooling layer of the previous layer;

[0070] Self-attention cascade module, the self-attention cascade module includes multiple layers of self-attention sub-modules SE Lager, and the self-attention sub-modules are respectively connected to the output end of the encoding module corresponding to each layer;

[0071] Multi-layer decoding module, each layer of decoding module includes a first two-dimensional convolution layer Conv2D and an upsampling layer upsample, the first multi-dimensional convolution layer is connected to the output end of the corresponding self-attention submodule of each layer and the output end of the corresponding upsampling layer, and the input end of the upsampling layer of the previous layer is connected to the output end of the first two-dimensional convolution layer of the next layer;

[0072] The second two-dimensional convolution layer Conv2D 3*3, the input end of the second two-dimensional convolution layer is connected to the output end of the last layer encoding module, and the output end of the second two-dimensional convolution layer is connected to the input end of the last layer decoding module;

[0073] The third two-dimensional convolutional layer Conv2D 3*3 is connected to the first layer decoding module.

[0074] Specifically, in this embodiment, the structure optimization network includes a multi-layer encoding module, a self-attention cascade module, and a multi-layer decoding module;

[0075] Among them, the multi-layer encoding module specifically includes a first encoding module, a second encoding module, a third encoding module, and a fourth encoding module. Each layer of the encoding module includes a coding unit and a maximum pooling layer, and the input end of the encoding unit of the latter layer is connected to the output end of the maximum pooling layer of the previous layer, that is, the input end of the encoding unit in the second encoding module is connected to the output end of the maximum pooling layer of the first encoding module, and so on. The connection relationship of the encoding modules of the remaining layers is not repeated;

[0076] The self-attention cascade module includes multiple layers of self-attention submodules, specifically including: a first self-attention submodule, a second self-attention submodule, a third self-attention submodule, and a fourth self-attention submodule, each of which is connected to the output end of the encoding module corresponding to the same layer, that is, the first self-attention submodule is connected to the output end of the first encoding module, and the second self-attention submodule is connected to the output end of the second encoding module;

[0077] The multi-layer decoding module specifically includes a first decoding module, a second decoding module, a third decoding module, and a fourth decoding module, each decoding module includes a first two-dimensional convolution layer and an upsampling layer, and the first multi-dimensional convolution layer is connected to the output end of the self-attention submodule corresponding to the same layer and the output end of the corresponding upsampling layer, and the input end of the upsampling layer of the previous layer is connected to the output end of the first two-dimensional convolution layer of the next layer, for example, the input end of the upsampling layer in the first decoding module is connected to the output end of the first two-dimensional convolution layer in the second decoding module, and the first two-dimensional convolution layer in the first decoding module includes a first input end and a second input end, the first input end is connected to the output end of the first self-attention submodule corresponding to the same layer, and the second input end is connected to the upsampling layer in the corresponding first decoding module;

[0078] As a preferred implementation, the encoding module includes a combination of one or more of a deep convolutional neural network VGG, a residual neural network (Residual Neural Network, ResNet), and a dense convolutional network Densenet.

[0079] As a preferred implementation, the decoding module may also use a two-dimensional matrix convolution layer and a transmission two-dimensional convolution layer to replace the first two-dimensional convolution layer and the upsampling layer.

[0080] In step S3, it also includes: providing a magnetic resonance image (such as an MRI image) corresponding to the ultrasound image, annotating the corresponding tissue features on the magnetic resonance image through the trained tissue optimization network, and migrating the annotated tissue features to the second tissue feature map to obtain a third tissue feature map;

[0081] Among them, the training method and specific structure of the tissue optimization network are roughly the same, except that the structure optimization network in step S3 performs feature migration on the tomography image, and the tissue optimization network performs feature migration on the magnetic resonance image. The tissue optimization network will not be described in detail here.

[0082] Through the above technical solution, a self-attention cascade module is set as an inter-layer skip connection layer, and the output of each maximum pooling layer is connected to the input of the corresponding two-dimensional convolutional layer; through the skip connection, the network can fuse feature maps at multiple scales, accelerate the network convergence speed and restore better image details.

[0083] Step S4, fusing the third structural feature map, the third tissue feature map and the ultrasound image to obtain an optimized ultrasound image.

[0084] As a preferred implementation, in step S4, the third structural feature map, the third tissue feature map and the ultrasound image are fused to obtain an optimized ultrasound image. The specific fusion formula is as follows:

[0085] I'=alpha*(S”+T”)+(1-alpha)*I

[0086] in,

[0087] I represents the original ultrasound image;

[0088] alpha represents the fusion factor;

[0089] S” represents the third structural characteristic diagram;

[0090] T” indicates the third tissue characteristic diagram;

[0091] I' represents the optimized ultrasound image.

[0092] The beneficial effects of the present invention are:

[0093] The present invention divides the ultrasound image into structural features and tissue features, extracts the image features of the tomography image and migrates them to the structural features of the ultrasound image, extracts the image features of the magnetic resonance image and migrates them to the tissue features of the ultrasound image, and performs fusion processing to suppress the speckle noise of the ultrasound image and improve the structural expression of the image; the migration is performed through the deep learning method, the vertical boundary structure is strengthened, the structure originally parallel to the sound beam is made clearer, and the artifacts caused by the acoustic shadow are effectively reduced.

[0094] The above description is only a preferred embodiment of the present invention, and does not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. An ultrasound image improvement method based on artificial intelligence, It is characterized in that include: Step S1, extracting structural information of an ultrasonic image, and segmenting the ultrasonic image according to the extracted structural information to obtain a first structural feature map and a first tissue feature map; Step S2, binarizing the first structural feature map, and performing morphological dilation processing on the processed first structural feature map to obtain a second structural feature map; as well as Decomposing the first tissue feature map into multiple layers, and removing tissue features of a middle layer of the first tissue feature map to obtain a second tissue feature map; Step S3, providing a tomographic image corresponding to the ultrasound image, annotating the corresponding structural features on the tomographic image through the trained structural optimization network, and migrating the annotated structural features to the second structural feature map to obtain a third structural feature map; and Providing a magnetic resonance image corresponding to the ultrasound image, annotating corresponding tissue features on the magnetic resonance image through a trained tissue optimization network, and migrating the annotated tissue features to the second tissue feature map to obtain a third tissue feature map; Step S4, fusing the third structural feature map, the third tissue feature map and the ultrasound image to obtain the optimized ultrasound image.

2. The ultrasonic image improvement method based on artificial intelligence according to claim 1, It is characterized in that In step S1, extracting structural information of the ultrasound image specifically includes: The ultrasonic image is extracted based on a Sobel operator or a Canny operator to obtain the structural information.

3. The ultrasonic image improvement method based on artificial intelligence according to claim 1, It is characterized in that In the step S1, a segmentation network is used to segment and extract tissue in the ultrasound image to obtain the first structural feature map and the first tissue feature map.

4. The ultrasonic image improvement method based on artificial intelligence according to claim 3, It is characterized in that The segmentation network includes a combination of one or more of a deep learning segmentation network, a fully convolutional network, and a semantic segmentation network.

5. The ultrasonic image improvement method based on artificial intelligence according to claim 1, It is characterized in that In the step S2, binarization processing is performed on the first structural feature map, specifically including: Step S21, processing the first structural feature graph based on a Sobel operator or a Prewit operator to obtain a structural factor corresponding to each structural feature in the first structural feature graph; Step S22, normalizing the structure factors respectively; Step S23 , comparing the normalized structure factor with a preset threshold, and when the absolute value of the structure factor is less than the preset threshold, setting the structure factor of the corresponding structural feature to 0.

6. The ultrasonic image improvement method based on artificial intelligence according to claim 1, It is characterized in that In the step S2, decomposing the first tissue feature map into multiple layers and removing the tissue features of the middle layer of the first tissue feature map to obtain a second tissue feature map specifically includes: The tissue feature map is decomposed by wavelet transform or pyramid segmentation to obtain multi-layer tissue features, and then the tissue features of the middle layer are removed by soft threshold filtering.

7. The ultrasonic image improvement method based on artificial intelligence according to claim 1, It is characterized in that In step S3, the trained structure optimization network and the trained organization optimization network respectively include: A multi-layer encoding module, each layer of the encoding module comprises a coding unit and a maximum pooling layer, and the input end of the coding unit of the latter layer is connected to the output end of the maximum pooling layer of the previous layer; A self-attention cascade module, wherein the self-attention cascade module includes multiple layers of self-attention sub-modules, and the self-attention sub-modules are respectively connected to the output end of the encoding module corresponding to each layer; A multi-layer decoding module, each layer of the decoding module comprises a first two-dimensional convolution layer and an upsampling layer, the first two-dimensional convolution layer is connected to the output end of the self-attention submodule corresponding to each layer and the output end of the corresponding upsampling layer, and the input end of the upsampling layer of the previous layer is connected to the output end of the first two-dimensional convolution layer of the next layer; A second two-dimensional convolutional layer, wherein an input end of the second two-dimensional convolutional layer is connected to an output end of the encoding module of the last layer, and an output end of the second two-dimensional convolutional layer is connected to an input end of the decoding module of the last layer; The third two-dimensional convolutional layer is connected to the decoding module in the first layer.

8. The ultrasonic image improvement method based on artificial intelligence according to claim 7, It is characterized in that The encoding module includes a combination of one or more of a deep convolutional neural network, a residual network, and a dense convolutional network.

9. The ultrasonic image improvement method based on artificial intelligence according to claim 7, It is characterized in that The decoding module includes a two-dimensional matrix convolution layer and a transmission two-dimensional convolution layer.

10. The ultrasonic image improvement method based on artificial intelligence according to claim 1, It is characterized in that In step S4, the third structural feature map, the third tissue feature map and the ultrasound image are fused to obtain the optimized ultrasound image. The specific fusion formula is as follows: I'=alpha*(S''+T'')+(1-alpha)*I in, I represents the original ultrasound image; alpha represents the fusion factor; S'' represents the third structural characteristic graph; T'' represents the third tissue characteristic diagram; I' represents the optimized ultrasound image.

Images