Organ ultrasound image quality evaluation method and system based on multilayer activation response fusion deep neural network

Abstract

The invention provides an organ ultrasound image quality evaluation method and system based on a multilayer activation response fusion deep neural network. The method comprises the following steps: S1, acquiring an acquired organ ultrasound image; s2, inputting the organ ultrasound image into a pre-trained deep neural network model to obtain an activation response graph which represents fusion of activation response features of the image at different levels; and S3, performing stipulation calculation on the activation response diagram to obtain a quality evaluation estimated value of the organ ultrasonic image. According to the organ ultrasound image quality evaluation method, through a deep neural network model structure and an efficient data expansion means, the model can capture more sufficient features to evaluate the image quality. Meanwhile, the generalization and the stability of the model are sufficiently guaranteed through the model structure and a large number of positive and negative samples.

Description

Technical Field

[0001] The present invention relates to the field of image quality evaluation, and particularly to a method for evaluating the quality of organ ultrasound images. Background Art

[0002] In the processing of organ ultrasound images, after a doctor completes an ultrasound scan of a patient's organ tissue, it is often necessary to further analyze these collected ultrasound images later. Correspondingly, how to quickly screen out valuable images from numerous collected images (such as how much organ tissue information is included) is a valuable and practical image analysis task.

[0003] Past image quality evaluation methods are based on traditional image algorithms, with relatively slow calculation processes and low accuracy and stability of calculation results. Some quality evaluation algorithms based on deep neural networks may often contain more parameters, or the training process is simple and crude. For example: some networks need to perform image cropping operations such as roi pooling on images, and then perform quality evaluation on the roi region images. Such operations are not "end-to-end" in themselves, resulting in low accuracy and stability.

[0004] In clinical applications, low-quality ultrasound images may lead to misdiagnosis or missed diagnosis. Therefore, establishing a scientific and reliable quality assessment mechanism is crucial for improving the level of ultrasound diagnosis. However, how to ensure the accuracy of the assessment while adapting to ultrasound images collected from different organs and different devices, and realizing the generality and robustness of the assessment method, is a complex and urgent technical problem facing researchers. Summary of the Invention

[0005] The object of the present invention is to address the problems existing in the existing image quality assessment methods, and propose a method for evaluating the quality of organ ultrasound images based on a multi-layer activation response fusion deep neural network.

[0006] The technical solution of the present invention is as follows: The present invention provides a method for evaluating the quality of organ ultrasound images based on a multi-layer activation response fusion deep neural network, the method comprising the following steps: S1. Obtain the collected organ ultrasound images; S2. Input the organ ultrasound images into a pre-trained deep neural network model to obtain activation response maps, which represent the fusion of activation response features of the images at different levels; S3. Perform reduction calculation on the activation response maps to obtain the quality evaluation estimate of the organ ultrasound images.

[0007] Further, S1 also includes: obtaining an organ ultrasound segmentation image corresponding to the organ ultrasound image, where the organ ultrasound segmentation image provides a mask for subsequent regularization calculation.

[0008] Further, before inputting the organ ultrasound image into the pre-trained deep neural network model in S2, it also includes: performing scaling and cropping on the organ ultrasound image, and the size of the processed organ ultrasound image is the same as the output size of the activation response map.

[0009] Further, the deep neural network model in S2 is designed with an encoder-decoder structure, including a compression path and an expansion path; A plurality of downsampling modules are set in the compression path to extract multi-scale features of the organ ultrasound image; A plurality of upsampling modules and skip connections are set in the expansion path to gradually restore the image resolution and fuse multi-scale features.

[0010] Further, S2 specifically includes: Inputting the preprocessed organ ultrasound image into the pre-trained deep neural network model, and extracting multi-scale features through the encoder compression path; Restoring the image resolution of the extracted multi-scale features through the upsampling modules and skip connections in the expansion path to obtain activation response maps output at each scale, where the activation response map represents the image features extracted by the model at this layer; Performing fusion processing on the output activation response maps to obtain the final activation response region.

[0011] Further, the regularization calculation of the multi-layer activation response in S3 includes: S31: Performing sigmoid activation on the activation response map output by the deep neural network model to obtain sigmoid activation values; S32: If there is an organ ultrasound segmentation image, using the organ ultrasound segmentation image as a mask, and performing probability normalization calculation softmax on the activation response map within the mask region to obtain a probability heat map within the mask region; If there is no organ ultrasound segmentation image, performing probability normalization calculation softmax on the activation response map of the entire image to obtain a probability heat map of the entire image; S33: Performing regularization calculation based on the sigmoid activation values and the probability heat map, and using the result as a quality evaluation estimation parameter.

[0012] Further, the sigmoid activation in S31 uses the following formula:

[0013] Where, , Represents an activation-response graph.

[0014] Furthermore, the calculation in S33 adopts the following formula:

[0015] in: Indicates the number of pixels, Refers to the number The sigmoid activation value of the pixel point, Refers to the number The probability of a pixel.

[0016] Further, after obtaining the quality evaluation estimate of the organ ultrasound image, the following steps are performed: Compare the quality evaluation estimate with a preset threshold, and if it is greater than the threshold, determine it as a high-quality image, otherwise determine it as a low-quality image; Repeat the above steps for all organ ultrasound images in an examination sequence to obtain a quality evaluation estimate for each frame of the image, and calculate the average quality score of the entire sequence as the overall quality evaluation result of the examination; The quality evaluation estimate is compared with a preset threshold value. If the quality evaluation estimate is greater than the preset threshold value, it is considered that the organ ultrasound image has qualified quality.

[0017] A system used in an organ ultrasound image quality assessment method based on a multi-layer activation response fusion deep neural network comprises: An image acquisition module, used for acquiring an ultrasonic image of an organ; A deep neural network processing module, used for inputting the organ ultrasound image into a pre-trained deep neural network model to obtain an activation response map, wherein the activation response map represents a fusion of activation response features of the image at different levels; The quality evaluation calculation module is used to perform a reduction calculation on the activation response map to obtain a quality evaluation estimate of the organ ultrasound image.

[0018] Beneficial effects of the present invention: The organ ultrasound image quality assessment method of the present invention uses a deep neural network model structure and efficient data expansion means to enable the model to capture more sufficient features to assess image quality. At the same time, the model structure and a large number of positive and negative samples ensure sufficient generalization and stability of the model.

[0019] The method of the present invention obtains the collected organ ultrasound images, inputs them into a pre-trained deep neural network model to obtain an activation response map, and performs normalization calculation on it to obtain an image quality evaluation value. Specifically, a deep neural network model designed with an encoder-decoder structure, including a compression path and an expansion path, extracts and fuses multi-scale features; performs normalization calculation by setting a mask area to obtain a quality evaluation value; finally, compares the quality evaluation value with a preset threshold to judge the image quality and evaluate the quality of the entire inspection sequence.

[0020] The present invention can effectively evaluate the quality of organ ultrasound images, provide reliable image quality assurance for medical diagnosis, and improve the accuracy and reliability of ultrasound examinations. The calculation speed of this method is fast, and the inference time of the quantized model on an NVIDIA graphics card is only 8 ms for one time.

[0021] The method of the present invention directly goes from low resolution to high resolution, performs activation response fusion in the decoding stage, and does not introduce any additional operations such as ROI (full-image operation); this method only requires the network model to learn the representation from blurred 0 to clear 1, involves fewer parameters, and the training process is efficient and reliable.

[0022] Other features and advantages of the present invention will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By describing the exemplary embodiments of the present invention in more detail in conjunction with the drawings, the above and other objects, features, and advantages of the present invention will become more obvious. Among them, in the exemplary embodiments of the present invention, the same reference numerals generally represent the same components.

[0024] Figure 1 Shows a flowchart of the organ ultrasound image quality evaluation method based on a multi-layer activation response fusion deep neural network of the present invention; Figure 2 Shows a schematic structural diagram of the deep neural network model of the present invention; Figure 3 Shows a schematic diagram of the normalization calculation in the embodiment of the present invention; Figure 4 Shows a schematic diagram of the quality evaluation results of a batch of continuously collected ultrasound images in the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0025] The following will describe the preferred embodiments of the present invention in more detail with reference to the drawings. Although the preferred embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein.

[0026] Embodiment 1 Figure 1 The flowchart of the organ ultrasound image quality evaluation method based on the multi-layer activation response fusion deep neural network of the present invention is shown.

[0027] As Figure 1 shown, the present invention provides an organ ultrasound image quality evaluation method based on a multi-layer activation response fusion deep neural network, including: S1. Obtain the collected organ ultrasound image; S2. Input the organ ultrasound image into a pre-trained deep neural network model to obtain an activation response map, where the activation response map represents the fusion of activation response features of the image at different levels; S3. Perform a reduction calculation on the activation response map to obtain a quality evaluation estimate of the organ ultrasound image.

[0028] In this embodiment, by obtaining high-quality ultrasound images, a reliable data source can be provided for subsequent deep neural network processing; inputting the organ ultrasound image into a pre-trained deep neural network model to obtain multi-layer activation responses. The activation response is the response of the network to the input image at each level, reflecting the feature representation of the image at that level. Through multi-layer activation responses, the network can comprehensively capture the feature information of the image. Perform a reduction calculation on the activation response map to obtain a quality evaluation estimate of the organ ultrasound image. The reduction calculation is a process of summarizing or compressing multi-layer activation responses, aiming to transform complex features at multiple levels into a single quality score, which can reasonably fuse features at different levels to ensure the accuracy and stability of the score. Through the reduction calculation, the quality of the ultrasound image can be quickly evaluated, helping doctors screen out valuable images.

[0029] In one example, obtain the organ ultrasound segmentation image corresponding to the organ ultrasound image, and the organ ultrasound segmentation image is used to provide a template for subsequent reduction calculation.

[0030] In this embodiment, the organ ultrasound segmentation image is used to provide a template for subsequent reduction calculation. The segmentation image is the result of precisely dividing the organ region in the original ultrasound image, which can clearly identify the boundaries and internal structures of the organ. This distinction not only provides a clear regional scope for subsequent quality evaluation but also can avoid the interference of non-target regions on the evaluation results.

[0031] Taking the segmentation image as a template provides a basis for the reduction calculation. The reduction calculation needs to perform statistics or feature extraction on specific regions in the image, and the segmentation image ensures that these calculations are only performed on the target organ region, excluding the influence of background noise, which can significantly improve the accuracy and stability of the quality evaluation.

[0032] In one example, the organ ultrasound image is scaled and cropped, and the size of the processed organ ultrasound image is the same as the output size of the activation response map.

[0033] In this embodiment, the scaling process is used to ensure the alignment of the input image and the output of the network model in the spatial dimension, facilitating subsequent activation response calculation. By uniformly processing ultrasound images of different sizes and resolutions into the same input size, the sensitivity of the model to different input sizes can be reduced, thereby improving the performance of the model on different datasets. In addition, the cropping operation can remove noise and irrelevant information in the image, enabling the model to focus more on the feature learning of the target region, further enhancing the stability and robustness of the model.

[0034] In one example, the deep neural network model is designed with an encoder-decoder structure, including a compression path and an expansion path; A plurality of downsampling modules are arranged in the compression path to extract multi-scale features of the organ ultrasound image; A plurality of upsampling modules and skip connections are arranged in the expansion path to gradually restore the image resolution and fuse multi-scale features. The processes performed include: Input the preprocessed organ ultrasound image into a pre-trained deep neural network model, and extract multi-scale features through the encoder compression path; Restore the image resolution of the extracted multi-scale features through the upsampling modules and skip connections in the expansion path to obtain the activation response maps output at each scale. The activation response maps represent the image features extracted by the model at this layer; Perform a fusion process on the output activation response maps to obtain the final activation response region.

[0035] In this embodiment, when the preprocessed organ ultrasound image is input into the pre-trained deep neural network model, the model first extracts multi-scale features through the encoder compression path. The encoder consists of multiple convolutional layers and pooling layers. The convolutional layers are used to extract local features in the image, while the pooling layers gradually reduce the resolution of the feature map through downsampling operations while retaining important feature information. Through this multi-scale feature extraction method, different levels of detail information in the image can be captured, providing a rich feature basis for subsequent quality evaluation.

[0036] After extracting the multi-scale features, the image resolution is restored through the upsampling module and jump connection of the extended path. The upsampling module uses deconvolution or interpolation operations to gradually restore the low-resolution feature map to the resolution of the original image. The jump connection fuses the feature map in the encoder path with the feature map in the decoder path, so that the model can retain the multi-scale features extracted in the encoder path while restoring the image resolution; it can effectively avoid information loss and enhance the model's ability to capture image details.

[0037] When fusing the output activation response maps, the activation response maps of different scales are integrated by weighted summation or cascading. The multi-layer fusion model can comprehensively consider the local details and global structure in the image to obtain the final activation response area. The final activation response area is represented in the form of a heat map. The highlighted area in the heat map indicates that the model believes that this area contributes more to the image quality evaluation.

[0038] In one example, performing reduction calculation on the multi-layer activation response in S3 includes: S31, perform sigmoid activation on the activation response map output by the deep neural network model, obtain the sigmoid activation value, perform full-map operation, and the value is between 0 and 1;

[0039] in, , represents the activation response map; S32: If there is an organ ultrasound segmentation image, use the organ ultrasound segmentation image as a mask, and perform probability normalization calculation on the activation response map within the mask area. softmax , get the probability heat map in the mask area; If there is no organ ultrasound segmentation image, the activation response map of the whole image is calculated by probability normalization. softmax , get the probability heat map of the whole image; S33, performing reduction calculation based on sigmoid activation value and probability heat map, and using the result as the quality evaluation estimation parameter;

[0040] in: Indicates the number of pixels, Refers to the number The sigmoid activation value of the pixel point, Refers to the number The probability of a pixel.

[0041] In this embodiment, if there is an organ ultrasound segmentation image, the calculation depends on the pre-provided segmentation image. The mask region can focus on the key regions in the image and exclude the interference of irrelevant regions, thereby improving the accuracy of evaluation. If there is no organ ultrasound segmentation image, an automatic analysis is performed based on the activation response of the entire image, and the quality evaluation can still be completed without a segmentation image.

[0042] In practical applications, the situations of having and not having a segmentation image may alternate, and the system can switch flexibly. For example, in the ultrasound image processing flow in a hospital, some images may have been segmented and labeled, while others do not have segmentation information. The system needs to be able to automatically execute according to the characteristics of the input image to ensure the consistency and accuracy of quality evaluation, greatly improving the applicability and practicality of the algorithm.

[0043] In one example, after obtaining the quality evaluation value of the organ ultrasound image, the following steps are performed: Compare the quality evaluation value with a preset threshold. If it is greater than the threshold, it is judged as a high-quality image; otherwise, it is judged as a low-quality image. Repeat the above steps for all organ ultrasound images in an examination sequence to obtain the quality evaluation value of each frame of image, and calculate the average quality score of the entire sequence as the overall quality evaluation result of this examination. Compare the quality evaluation value with a preset threshold. If the quality evaluation value is greater than the preset threshold, the quality of the organ ultrasound image is considered qualified.

[0044] In this embodiment, after obtaining the quality evaluation value of the organ ultrasound image, it is first necessary to compare this value with a preset threshold. The preset threshold is usually set according to the quality standards of medical images and actual application requirements. For example, the threshold is set to 0.7. If the quality evaluation value is greater than 0.7, the image is judged as a high-quality image; otherwise, it is judged as a low-quality image, which can quickly screen out images that meet the quality requirements for subsequent analysis and processing.

[0045] For all organ ultrasound images in an examination sequence, the above comparison steps need to be repeated to obtain the quality evaluation value of each frame of image. Calculate the average quality score of the entire sequence as the overall quality evaluation result of this examination, which helps to comprehensively evaluate the quality of the entire examination sequence, ensure the reliability and accuracy of the examination results, and provide a more accurate diagnostic basis for doctors.

[0046] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for evaluating organ ultrasound image quality based on multi-layer activation response fusion deep neural network, characterized in that: The method comprises the following steps: S1, obtaining an ultrasound image of the collected organ; S2. Inputting the organ ultrasound image into a pre-trained deep neural network model to obtain an activation response map, where the activation response map represents a fusion of activation response features of the image at different levels; S3. Performing reduction calculation on the activation response map to obtain a quality evaluation estimate of the organ ultrasound image.

2. The organ ultrasound image quality assessment method based on multi-layer activation response fusion deep neural network as claimed in claim 1 is characterized in that S1 also includes: acquiring an organ ultrasonic segmentation image corresponding to the organ ultrasonic image, wherein the organ ultrasonic segmentation image is used to provide a mask for subsequent reduction calculation.

3. The organ ultrasound image quality assessment method based on multi-layer activation response fusion deep neural network as claimed in claim 1 is characterized in that In S2, before inputting the organ ultrasound image into the pre-trained deep neural network model, the method also includes: scaling and cropping the organ ultrasound image, and the size of the processed organ ultrasound image is the same as the output size of the activation response map.

4. The organ ultrasound image quality assessment method based on multi-layer activation response fusion deep neural network as claimed in claim 1, characterized in that In S2, the deep neural network model adopts an encoder-decoder structure design, including a compression path and an expansion path; Arranging a plurality of downsampling modules in the compression path to extract multi-scale features of the organ ultrasound image; A plurality of upsampling modules and skip connections are arranged in the extension path to gradually restore the image resolution and fuse multi-scale features.

5. The organ ultrasound image quality assessment method based on multi-layer activation response fusion deep neural network as claimed in claim 4, characterized in that S2 Specifically include: Inputting the preprocessed organ ultrasound image into a pre-trained deep neural network model, and extracting multi-scale features through an encoder compression path; The extracted multi-scale features are restored to image resolution through the upsampling module and the skip connection of the extension path to obtain an activation response map output at each scale, wherein the activation response map represents the image features extracted by the model at this layer; The output activation response map is fused to obtain a final activation response region.

6. The organ ultrasound image quality assessment method based on multi-layer activation response fusion deep neural network as claimed in claim 1, characterized in that In S3, the multi-layer activation response is reduced and calculated, including: S31, performing sigmoid activation on the activation response graph output by the deep neural network model to obtain a sigmoid activation value; S32, if there is an organ ultrasound segmentation image, use the organ ultrasound segmentation image as a mask, perform probability normalization and softmax calculation on the activation response map in the mask area, and obtain a probability heat map in the mask area; If there is no organ ultrasound segmentation image, the activation response map of the whole image is probability normalized and softmax is calculated to obtain the probability heat map of the whole image; S33. Perform reduction calculation based on sigmoid activation value and probability heat map, and use the result as the quality evaluation estimation parameter.

7. The organ ultrasound image quality assessment method based on multi-layer activation response fusion deep neural network as claimed in claim 6 is characterized in that In S31, sigmoid activation uses the following formula: ； in, , Represents an activation-response graph.

8. The organ ultrasound image quality assessment method based on multi-layer activation response fusion deep neural network as claimed in claim 6, characterized in that In S33, the following formula is used for the calculation of the protocol: ； in: Indicates the number of pixels, Refers to the number The sigmoid activation value of the pixel point, Refers to the number The probability of a pixel.

9. The organ ultrasound image quality assessment method based on multi-layer activation response fusion deep neural network as claimed in claim 6, characterized in that After obtaining the quality evaluation estimate of the organ ultrasound image, the following steps are performed: Compare the quality evaluation estimate with a preset threshold, and if it is greater than the threshold, determine it as a high-quality image, otherwise determine it as a low-quality image; Repeat the above steps for all organ ultrasound images in an examination sequence to obtain a quality evaluation estimate for each frame of the image, and calculate the average quality score of the entire sequence as the overall quality evaluation result of the examination; The quality evaluation estimate is compared with a preset threshold value. If the quality evaluation estimate is greater than the preset threshold value, it is considered that the organ ultrasound image has qualified quality.

10. A system used in the organ ultrasound image quality assessment method based on multi-layer activation response fusion deep neural network as described in any one of claims 1 to 9, characterized in that: include: An image acquisition module, used for acquiring an ultrasonic image of an organ; A deep neural network processing module, used for inputting the organ ultrasound image into a pre-trained deep neural network model to obtain an activation response map, wherein the activation response map represents a fusion of activation response features of the image at different levels; The quality evaluation calculation module is used to perform a reduction calculation on the activation response map to obtain a quality evaluation estimate of the organ ultrasound image.

Images