Liver ultrasonic standard section image recognition method and device and computer equipment

Abstract

The invention provides a liver ultrasonic standard section image recognition method and device and computer equipment, and the method comprises the steps: recognizing a liver region in a liver ultrasonic section image, and carrying out the mask processing of the liver ultrasonic section image based on a recognition result, and generating a liver ultrasonic section enhanced image; inputting the liver ultrasonic section image into the global branch to generate a first feature map; inputting the liver ultrasonic section enhanced image into a diagnosis branch to generate a second feature map; after the first feature map and the second feature map are mutually fused, a judgment result of the liver ultrasonic section image is generated; according to the method, the recognition accuracy is improved through complementation of the global branch and the diagnosis branch, the attention mining technology is utilized, features related to tasks are accurately extracted, the model efficiency and accuracy are improved, and by the adoption of the recognition and judgment method, a doctor can be assisted to rapidly judge whether the image is a standard image or not, and the diagnosis efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and more specifically, to a liver ultrasound standard section image recognition method, device and computer equipment. Background Art

[0002] In the prior art, for liver ultrasound images, standard sections usually refer to clear, repeatable, and diagnostically significant images acquired at specific angles and positions according to medical guidelines and anatomical requirements. These images can display key anatomical structures of the liver, such as liver contours, blood vessels, and hepatic hilum, helping doctors accurately determine lesions or pathological conditions. Non-standard sections, on the other hand, are blurred, incomplete, or fail to display important anatomical structures well due to factors such as improper operation, poor probe angles, or patient body shape, and therefore lack diagnostic value.

[0003] Standard sections can provide doctors with the complete anatomical information they need, reducing the risk of misdiagnosis or missed diagnosis. In addition, distinguishing between standard and non-standard sections can help ensure that the images doctors rely on have sufficient diagnostic value. By automatically identifying and filtering out non-standard sections, doctors can reduce the time they spend processing invalid images and improve work efficiency. In summary, in the field of liver ultrasound technology, it is very important to identify and determine whether a liver image is a standard section.

[0004] In the field of medical imaging diagnosis, especially the analysis and diagnosis of liver ultrasound images, traditional methods rely on the personal experience and skills of doctors, which leads to individual differences in diagnostic results and lack of consistency and standardization. With the surge in the amount of medical imaging data, the workload of doctors has increased, and the efficiency and accuracy of manual interpretation are facing challenges. Although the development of computer-aided diagnosis (CAD) systems and deep learning technology has provided new possibilities for improving diagnostic efficiency and accuracy, existing technologies still have shortcomings. For example, existing CAD systems are usually based on traditional image processing algorithms such as edge detection and morphological operations, which may be computationally inefficient when processing complex and changeable ultrasound images and cannot meet the needs of real-time diagnosis. In addition, although deep learning methods, especially models based on convolutional neural networks (CNNs), perform well in image recognition tasks, they usually require a large amount of labeled data for training, which is a costly and time-consuming process in the medical field. At the same time, deep learning models have limited generalization capabilities on new or different datasets, especially when faced with challenges such as noise, shadows, and blur commonly found in ultrasound images. To overcome these challenges, the medical imaging field is actively exploring new algorithms and technologies to improve the accuracy, efficiency, and standardization of liver ultrasound image analysis. This includes developing more advanced image processing algorithms, optimizing the training process of deep learning models, and improving the model's generalization capabilities for different data sets. It is hoped that more powerful auxiliary tools will be provided to doctors to improve the overall diagnostic quality and patient care level. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a liver ultrasound standard section image recognition method, device and computer equipment to overcome the shortcomings of the prior art that the convolutional neural network (CNN) requires a large amount of labeled data for training and has high training costs.

[0006] The above technical objectives of the present invention are achieved through the following technical solutions: A liver ultrasound standard section image recognition method comprising:

[0007] Identify the liver region in the liver ultrasound section image, and perform mask processing on the liver ultrasound section image based on the identification result to generate an enhanced liver ultrasound section image;

[0008] Inputting the liver ultrasound section image into the global branch to generate a first feature map; wherein the global branch includes a first attention mining module for identifying liver feature information; inputting the liver ultrasound section enhanced image into the diagnostic branch to generate a second feature map; wherein the diagnostic branch includes a second attention mining module for identifying liver feature information;

[0009] After the first feature map and the second feature map are fused with each other, a judgment result of the liver ultrasound section image is generated; the judgment result specifically includes: the liver ultrasound section image is a liver ultrasound standard section image, or the liver ultrasound section image is a liver ultrasound non-standard section image.

[0010] In one embodiment, before identifying the liver area in the liver ultrasound section image and performing mask processing on the liver ultrasound section image based on the identification result to generate the liver ultrasound section enhanced image, it also includes: preprocessing the original liver ultrasound section image to obtain the liver ultrasound section image; preprocessing the original liver ultrasound section image, specifically including: removing operation information and patient information in the original liver ultrasound section image, and performing grayscale processing on the original liver ultrasound section image.

[0011] In one embodiment, a liver region in a liver ultrasound section image is identified, and a mask processing is performed on the liver ultrasound section image based on the identification result to generate a liver ultrasound section enhanced image, specifically comprising: using a Unet++ segmentation model to identify the liver region in the liver ultrasound section image, and generating a region mask M having the same size as the liver ultrasound section image based on the identification result. unet++ ; Using the region mask M unet++ The liver ultrasound section image is masked to generate a liver ultrasound section enhanced image.

[0012] In one embodiment, using the region mask M unet++ Masking is performed on the liver ultrasound section image to generate a liver ultrasound section enhanced image, specifically including: masking the region M unet++ The pixels of are added one by one to the pixels of the liver ultrasound section image to generate the liver ultrasound section enhanced image:

[0013] U M =U⊙M unet++ ;

[0014] Among them, U M Indicates enhanced ultrasound image of liver section; M unet++ U represents the ultrasound section image of the liver; ⊙ represents the addition of pixels one by one.

[0015] In one embodiment, the liver ultrasound section image is input into the global branch to generate a first feature map, specifically including: inputting the liver ultrasound section image into the convolution layer in the global branch for convolution to generate a first feature map F1; inputting the first feature map F1 into the global average pooling layer in the global branch for pooling to generate a first channel attention map a1; using an activation function to activate the first channel attention map a1; using the first feature map F1 to upsample the activated first channel attention map a1 to generate a first attention weight map Use the first attention mining module to mine the first attention weight map Mining is performed to generate a first feature map.

[0016] In one embodiment, the first attention weight map is mined using a first attention mining module. Mining to generate the first feature map, specifically including: judging the first attention weight map one by one Is the spatial value of each position in the first attention mining unit greater than the first threshold α1? If so, multiply the spatial value by the first hyperparameter θ1 in the first attention mining unit to generate the first eigenvalue For the first eigenvalue The first mining feature map composed of features is used for feature extraction to generate the second attention weight map Judge the second attention weight map one by one Is the spatial value of each position in the second attention mining unit greater than the second threshold α2? If so, multiply the spatial value by the second hyperparameter θ2 in the second attention mining unit to generate the second eigenvalue For the second eigenvalue The composed second mining feature graph is used to extract features and generate the first feature graph.

[0017] In one embodiment, the liver ultrasound section enhanced image is input into the diagnosis branch to generate a second feature map, specifically including: inputting the liver ultrasound section enhanced image into the convolution layer in the diagnosis branch for convolution to generate a second feature map F2; inputting the second feature map F2 into the global average pooling layer in the diagnosis branch for pooling to generate a second channel attention map a2; using an activation function to activate the second channel attention map a2; using the second feature map F2 to upsample the activated second channel attention map a2 to generate a third attention weight map Use the second attention mining module to mine the third attention weight map Mining is performed to generate a second feature map.

[0018] In one embodiment, the third attention weight map is mined using the second attention mining module. Mining generates the second feature map, specifically including: judging the third attention weight map one by one Is the spatial value of each position in the third attention mining unit greater than the third threshold α3? If so, multiply the spatial value by the third hyperparameter θ3 in the third attention mining unit to generate the third eigenvalue For the third eigenvalue The third mining feature map composed of the fourth attention weight map is used for feature extraction. Judge the fourth attention weight map one by one Is the spatial value of each position in the fourth attention mining unit greater than the fourth threshold α4? If so, multiply the spatial value by the fourth hyperparameter θ4 in the fourth attention mining unit to generate the fourth eigenvalue For the fourth eigenvalue The fourth mining feature map composed of the above-mentioned features is used to perform feature extraction and generate a second feature map.

[0019] In one embodiment, after the first feature map and the second feature map are fused with each other, a judgment result of the liver ultrasound section image is generated, which specifically includes: performing global average pooling on the first feature map to generate a first feature vector; performing global average pooling on the second feature map to generate a second feature vector; concatenating the first feature vector and the second feature vector to generate a fused feature vector; using the Softmax function to convert the fused feature vector into a probability distribution; and inputting the probability distribution into a fully connected layer to generate a judgment result.

[0020] A liver ultrasound standard section image recognition device, comprising:

[0021] an enhancement unit, used for identifying a liver region in the liver ultrasound section image, and performing mask processing on the liver ultrasound section image based on the identification result to generate an enhanced liver ultrasound section image;

[0022] An extraction unit is used to input the liver ultrasound section image into a global branch to generate a first feature map; wherein the global branch includes a first attention mining module for identifying liver feature information; input the liver ultrasound section enhanced image into a diagnosis branch to generate a second feature map; wherein the diagnosis branch includes a second attention mining module for identifying liver feature information;

[0023] The judgment unit is used to generate a judgment result of the liver ultrasound section image after fusing the first feature map and the second feature map; the judgment result specifically includes: the liver ultrasound section image is a liver ultrasound standard section image, or the liver ultrasound section image is a liver ultrasound non-standard section image.

[0024] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0025] A computer device includes a memory and a processor, wherein the memory stores a computer program and the processor implements the steps of the above method when executing the computer program.

[0026] In summary, the present invention has the following beneficial effects: a method for recognizing standard liver ultrasound section images, comprising: recognizing a liver region in a liver ultrasound section image, and performing mask processing on the liver ultrasound section image based on the recognition result to generate a liver ultrasound section enhanced image; inputting the liver ultrasound section image into a global branch to generate a first feature map; inputting the liver ultrasound section enhanced image into a diagnostic branch to generate a second feature map; after fusing the first feature map and the second feature map, generating a judgment result of the liver ultrasound section image; using the method of the present invention, overall information is extracted through a global branch, and the diagnostic branch focuses on liver region features, and the two complement each other to improve recognition accuracy, and background interference is effectively reduced through mask processing, so that the model is more focused on the liver region and the diagnostic effect is enhanced, and attention mining technology is used to dynamically adjust the weight of each channel and spatial position, accurately extract task-related features, and improve model efficiency and accuracy, thereby providing a method for recognizing and judging liver region ultrasound section images, which can assist doctors in quickly judging whether an image is a standard image. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the liver ultrasound standard section image recognition method of the present invention;

[0028] Figure 2 is a structural diagram of a liver ultrasound standard section image recognition device in an embodiment of the present invention;

[0029] Figure 3 is an internal structure diagram of a computer device in an embodiment of the present invention;

[0030] Figure 4 Schematic diagram of the liver ultrasound image model recognition model structure in an embodiment of the present invention;

[0031] In the figure, 1. enhancement unit; 2. extraction unit; 3. judgment unit. DETAILED DESCRIPTION

[0032] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. Several embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein.

[0033] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0034] In the present invention, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Moreover, the first feature being "above", "above" and "above" the second feature includes the first feature being directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature includes the first feature being directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature. The terms "vertical", "horizontal", "left", "right", "above", "below" and similar expressions are for illustrative purposes only, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0035] Embodiment 1

[0036] In order to solve the above problems, the present invention provides a method for recognizing liver ultrasound standard section images, such as Figure 1 As shown, including:

[0037] S1, identifying a liver region in a liver ultrasound section image, and performing mask processing on the liver ultrasound section image based on the identification result to generate an enhanced liver ultrasound section image;

[0038] S2, inputting the liver ultrasound section image into the global branch to generate a first feature map; wherein the global branch includes a first attention mining module for identifying liver feature information; inputting the liver ultrasound section enhanced image into the diagnosis branch to generate a second feature map; wherein the diagnosis branch includes a second attention mining module for identifying liver feature information;

[0039] S3. After fusing the first feature map and the second feature map, a judgment result of the liver ultrasound section image is generated; the judgment result specifically includes: the liver ultrasound section image is a standard liver ultrasound section image, or the liver ultrasound section image is a non-standard liver ultrasound section image.

[0040] Specifically, in the actual prediction and judgment process, it is first necessary to obtain an ultrasonic section image of the liver. In the process of obtaining the original liver ultrasound section image, different standard liver ultrasound section images will be obtained due to factors such as the doctor's operating specifications and operating habits. The standard section is obtained according to a specific angle and position, which can well display the key anatomical structure, while the non-standard section may deviate from these angles, making the image information incomplete or distorted. Standard sections usually have high image quality and contrast, and can clearly display the liver and its surrounding structures. Non-standard sections may contain noise, shadows, blur and other phenomena, which affect the doctor's interpretation. Therefore, how to quickly obtain standard liver ultrasound section images from a large number of liver ultrasound images and exclude non-standard liver ultrasound section images to reduce the time doctors spend on processing invalid images and improve work efficiency is of great significance to both doctors and patients.

[0041] In this application, it is first necessary to obtain the liver ultrasound section image to be identified and judged. The model first masks the image and then inputs the image into the global branch and the diagnostic branch respectively. The multi-level features of the image are extracted through the ResNet50 network, and the attention mechanism (LAM) is combined to enhance the model's attention to the key areas of the liver. The feature maps extracted in the global branch and the diagnostic branch are fused to form the final classification result. The model is optimized through the cross entropy loss function, and finally the prediction result for liver ultrasound section classification is obtained, that is, the input liver section image is a standard liver ultrasound section image or a non-standard liver ultrasound section image.

[0042] In the above process, masking the liver image can distinguish the liver area from other areas in the image, so that the model pays more attention to the liver part and reduces the interference of irrelevant background. The liver ultrasound section image is input into the global branch to generate the first feature map. The global branch captures the overall information of the liver ultrasound image, while the diagnostic branch focuses on guiding the focus on the key areas of the liver through the mask. By combining these two types of information, the model can improve the recognition and classification capabilities of liver ultrasound images. In addition, in the two branches, attention mining modules are set up respectively. The attention mining module can dynamically adjust the attention weight of each channel or each spatial position, so that the network can more accurately extract important features related to the task, rather than treating all input features equally. Finally, the features generated by the two branches are fused and the prediction result is output. The prediction result is to determine whether the liver ultrasound section model is a standard liver ultrasound section image or a non-standard liver ultrasound section image.

[0043] In summary, this application extracts overall information through a global branch, and the diagnostic branch focuses on the characteristics of the liver area. The two complement each other to improve recognition accuracy. Mask processing effectively reduces background interference, allowing the model to focus more on the liver area and enhance the diagnostic effect. Attention mining technology is used to dynamically adjust the weights of each channel and spatial position, accurately extract task-related features, and improve model efficiency and accuracy.

[0044] In one embodiment, before identifying the liver area in the liver ultrasound section image and performing mask processing on the liver ultrasound section image based on the identification result to generate the liver ultrasound section enhanced image, it also includes: preprocessing the original liver ultrasound section image to obtain the liver ultrasound section image; preprocessing the original liver ultrasound section image, specifically including: removing operation information and patient information in the original liver ultrasound section image, and performing grayscale processing on the original liver ultrasound section image.

[0045] The original liver ultrasound section image is preprocessed, which includes manually cropping out the operation information and patient sensitive information in the image, and graying the color ultrasound image to adapt it to the subsequent model input requirements. It can also reduce the impact of color on the model and make the model pay more attention to the shape characteristics of the image.

[0046] In one embodiment, a liver region in a liver ultrasound section image is identified, and a mask processing is performed on the liver ultrasound section image based on the identification result to generate a liver ultrasound section enhanced image, specifically comprising: using a Unet++ segmentation model to identify the liver region in the liver ultrasound section image, and generating a region mask M having the same size as the liver ultrasound section image based on the identification result. unet++ ; Using the region mask M unet++The liver ultrasound section image is masked to generate a liver ultrasound section enhanced image.

[0047] The mask is generated by the Unet++ segmentation model. The Unet++ segmentation model is a deep learning model for medical image segmentation that can accurately mark the liver area in the image. Output mask M unet++ It is a matrix of the same size as the original ultrasound image U, and each pixel value in the matrix is ​​between 0 and 1. The value of this mask matrix reflects the probability of the pixel belonging: a value close to 1 indicates that the pixel belongs to the liver area, and a value close to 0 indicates that the pixel belongs to the non-liver area. Similarly, a binary mask can be directly used to mark the pixels belonging to the liver area as 1 and the pixels belonging to the non-liver area as 0. In this way, the amount of image processing calculation can be reduced in the subsequent mask processing process. The role of the mask is to distinguish the liver area from other areas in the image, so that the model pays more attention to the liver part and reduces the interference of irrelevant background.

[0048] In one embodiment, using the region mask M unet++ Masking is performed on the liver ultrasound section image to generate a liver ultrasound section enhanced image, specifically including: masking the region M unet++ The pixels of are added one by one to the pixels of the liver ultrasound section image to generate the liver ultrasound section enhanced image:

[0049] U M =U⊙M unet++ ;

[0050] Among them, U M Indicates enhanced ultrasound image of liver section; M unet++ U represents the ultrasound section image of the liver; ⊙ represents the addition of pixels one by one.

[0051] Specifically,

[0052] U M (i,j)=U(i,j)×M unet++ (i,j);

[0053] Among them, U(i,j) represents the pixel value at position (i,j); M unet++ (i,j) represents the mask weight of the position (i,j); unet++ When (i,j)≈1, U M (i,j)≈U(i,j); in M unet++ When (i,j)≈0, U M (i,j)≈0; after pixel-by-pixel multiplication, the enhanced image U is obtained M, this image retains the pixel values ​​of the liver area while masking or weakening the pixels of the non-liver area. This allows the model to focus more on the features of the liver area during subsequent processing.

[0054] In one embodiment, the liver ultrasound section image is input into the global branch to generate a first feature map, specifically including: inputting the liver ultrasound section image into the convolution layer in the global branch for convolution to generate a first feature map F1; inputting the first feature map F1 into the global average pooling layer in the global branch for pooling to generate a first channel attention map a1; using an activation function to activate the first channel attention map a1; using the first feature map F1 to upsample the activated first channel attention map a1 to generate a first attention weight map Use the first attention mining module to mine the first attention weight map Mining is performed to generate a first feature map.

[0055] Specifically, the liver ultrasound section image is input into the convolution layer in the global branch for convolution to generate the first feature map F1; specifically, the first feature map F v The size of is H×W×C; where H is the height of the image, W is the width of the image, and C is the number of channels generated by the convolutional layer.

[0056] Then, the first feature map F1 is pooled using the global average pooling layer to compress the spatial dimensions (W dimension and H dimension), obtain the average value of each channel, and obtain the first channel attention map a1; specifically, the first channel attention map a1 includes: a1 = [a 1,1 ,a 1,2 ,a 1,3 ,…,a 1,c ];

[0057]

[0058] Among them, i represents the index in the width direction, j represents the index in the height direction, and k represents the index in the channel dimension direction; a 1,k Represents the average value of the kth channel.

[0059] The activation function S is used to activate the attention map a1 of the first channel. The specific activation function is: ReLU function, which makes the attention weight fluctuate between 0 and 1 through nonlinear transformation, reflecting the contribution of different channels to the features.

[0060] S(a1)=ReLU(a1);

[0061] Since the pooling operation reduces the resolution of the feature map, the feature map is restored to its original size through bilinear upsampling. This operation is to continue to use high-resolution features in the next layer.

[0062]

[0063] First attention weight map After upsampling calculation, the original feature map size H×W×C is returned to guide the subsequent feature mining using the attention mining module.

[0064] In one embodiment, the first attention weight map is mined using a first attention mining module. Mining to generate the first feature map, specifically including: judging the first attention weight map one by one Is the spatial value of each position in the first attention mining unit greater than the first threshold α1? If so, multiply the spatial value by the first hyperparameter θ1 in the first attention mining unit to generate the first eigenvalue For the first eigenvalue The first mining feature map composed of features is used for feature extraction to generate the second attention weight map Judge the second attention weight map one by one Is the spatial value of each position in the second attention mining unit greater than the second threshold α2? If so, multiply the spatial value by the second hyperparameter θ2 in the second attention mining unit to generate the second eigenvalue For the second eigenvalue The composed second mining feature graph is used to extract features and generate the first feature graph.

[0065] In this application, the global branch is obtained by transforming the ResNet50 network. The ResNet50 network includes 5 stages; each stage contains several residual blocks, and each residual block consists of multiple convolutional layers; among them, Stage 1: This stage includes convolutional layers and pooling layers, which are used to perform preliminary feature extraction on the input image and extract low-level features such as edges, lines and simple textures. Stage 2: From this stage, deep convolution begins to extract more abstract features, including local shapes, structures, etc. The spatial resolution of the feature map will be further reduced, and the number of channels will increase to extract more complex information. Stage 3: This stage continues to deepen the level of feature extraction and obtain higher-order features, which usually represent local area features of the target object, such as partial edges and contours of the object. This is also where the attention mining module (LAM) begins to intervene to focus on important areas. Stage 4: The feature map of this stage represents higher-level semantic information, and the input continues to be processed through the deep convolution of the residual block. The attention module (LAM) appears again here to further enhance the focus on the liver area. Stage 5: This stage is responsible for extracting the global semantic features of the image. The feature map here already represents the global information of the input image and is ready to be input into the global average pooling (GAP) layer. Two attention mining units are introduced in the global branch, which are set after Stage3 and Stage4 respectively, to ensure that the model can strengthen the liver area multiple times at multiple levels, thereby improving the recognition accuracy of the model. It also enables the model to focus on the liver area at different levels, which can enhance the perception of low-level structures (such as edges and shapes) and the understanding of high-level semantic information (such as global structures or lesion areas). If LAM is introduced in only one stage, the model may not be able to fully capture information at different levels. Through multiple attention enhancements, the model can better cope with noise, blur or other interference when facing complex ultrasound images, so that it can maintain good robustness under different conditions.

[0066] Specifically, the attention mining unit is used to identify the importance of different areas in the attention weight map by setting an attention weight threshold and comparing each spatial value with the attention weight threshold. When the total attention weight value is greater than the attention weight threshold, it means that the attention weight value needs to be paid attention to, and the attention weight value is multiplied by the preset hyperparameter θ to expand the influence of the area on the model judgment, or reduce the influence of other areas on the model judgment.

[0067] In one embodiment, the liver ultrasound section enhanced image is input into the diagnosis branch to generate a second feature map, specifically including: inputting the liver ultrasound section enhanced image into the convolution layer in the diagnosis branch for convolution to generate a second feature map F2; inputting the second feature map F2 into the global average pooling layer in the diagnosis branch for pooling to generate a second channel attention map a2; using an activation function to activate the second channel attention map a2; using the second feature map F2 to upsample the activated second channel attention map a2 to generate a third attention weight map Use the second attention mining module to mine the third attention weight map Mining is performed to generate a second feature map.

[0068] In one embodiment, the third attention weight map is mined using the second attention mining module. Mining generates the second feature map, specifically including: judging the third attention weight map one by one Is the spatial value of each position in the third attention mining unit greater than the third threshold α3? If so, multiply the spatial value by the third hyperparameter θ3 in the third attention mining unit to generate the third eigenvalue For the third eigenvalue The third mining feature map composed of the fourth attention weight map is used for feature extraction. Judge the fourth attention weight map one by one Is the spatial value of each position in the fourth attention mining unit greater than the fourth threshold α4? If so, multiply the spatial value by the fourth hyperparameter θ4 in the fourth attention mining unit to generate the fourth eigenvalue For the fourth eigenvalue The fourth mining feature map composed of the above-mentioned features is used to perform feature extraction and generate a second feature map.

[0069] Specifically, in the diagnostic branch, the image processing and recognition process is similar to that in the global branch, except that the input image of the network is an enhanced image of the liver ultrasound section after mask processing. In this application, the role of the diagnostic branch is to focus on local feature extraction and enhancement of the liver area, which complements the global branch. The diagnostic branch introduces a mask to focus on specific areas of the liver and extract local features that are highly relevant to the diagnostic task, thereby improving the accuracy and robustness of the model in liver ultrasound section recognition; the diagnostic branch and the global branch work together, the global branch is responsible for extracting the global features of the entire image, ensuring that the model can recognize the macroscopic patterns and contextual information in the ultrasound image, and the diagnostic branch focuses on the refined feature extraction of local areas, especially through the attention mechanism (LAM) to further strengthen the focus on important liver areas. The global features and diagnostic features are fused in subsequent steps to form a more comprehensive feature expression, which helps to improve the accuracy of classification.

[0070] In one embodiment, after the first feature map and the second feature map are fused with each other, a judgment result of the liver ultrasound section image is generated, which specifically includes: performing global average pooling on the first feature map to generate a first feature vector; performing global average pooling on the second feature map to generate a second feature vector; concatenating the first feature vector and the second feature vector to generate a fused feature vector; using the Softmax function to convert the fused feature vector into a probability distribution; and inputting the probability distribution into a fully connected layer to generate a judgment result.

[0071] After obtaining the two feature maps, the two feature maps are respectively subjected to global average pooling to generate corresponding feature vectors. Global average pooling can reduce the complexity of the features, reduce the number of features input to the fully connected layer, and avoid overfitting of the prediction results. The two feature vectors are spliced ​​to generate a fused feature vector. The purpose of the splicing operation is to merge the feature vectors extracted from the global branch and the diagnostic branch to form a more complete feature representation. The Softmax function is used to convert the input feature values ​​into a probability distribution and normalize them to a probability value between (0,1). In this application, the Softmax function can convert the input features into the probabilities of two classification results, that is, the probability that the liver ultrasound section image is a standard image or a non-standard image. Finally, the probability distribution generated by the Softmax function is input into the fully connected layer, which is a densely connected network layer that maps the input feature vector to the output classification result. Before the fully connected layer, the Sigmoid activation function is used. Sigmoid limits the fused features to the range of (0,1) to ensure that the value of each feature is a probability representation.

[0072] Embodiment 2

[0073] See also Figure 2 , a liver ultrasound standard section image recognition device, the liver ultrasound standard section image recognition device comprises:

[0074] Enhancement unit 1, used for identifying the liver region in the liver ultrasound section image, and performing mask processing on the liver ultrasound section image based on the identification result to generate an enhanced liver ultrasound section image;

[0075] Extraction unit 2, used for inputting the liver ultrasound section image into the global branch to generate a first feature map; wherein the global branch includes a first attention mining module for identifying liver feature information; inputting the liver ultrasound section enhanced image into the diagnosis branch to generate a second feature map; wherein the diagnosis branch includes a second attention mining module for identifying liver feature information;

[0076] The judgment unit 3 is used to generate a judgment result of the liver ultrasound section image after fusing the first feature map and the second feature map; the judgment result specifically includes: the liver ultrasound section image is a liver ultrasound standard section image, or the liver ultrasound section image is a liver ultrasound non-standard section image.

[0077] The specific definition of the liver ultrasound standard section image recognition device can be found in the definition of the liver ultrasound standard section image recognition method mentioned above, which will not be repeated here. Each module in the above-mentioned liver ultrasound standard section image recognition device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0078] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation of the present application scheme. The specific liver ultrasound standard section image recognition device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0079] Embodiment 3

[0080] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the liver ultrasound standard section image recognition method as described in Example 1.

[0081] Embodiment 4

[0082] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, a liver ultrasound standard section image recognition method is implemented.

[0083] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0084] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0085] Identify the liver region in the liver ultrasound section image, and perform mask processing on the liver ultrasound section image based on the identification result to generate an enhanced liver ultrasound section image;

[0086] Inputting the liver ultrasound section image into the global branch to generate a first feature map; wherein the global branch includes a first attention mining module for identifying liver feature information; inputting the liver ultrasound section enhanced image into the diagnostic branch to generate a second feature map; wherein the diagnostic branch includes a second attention mining module for identifying liver feature information;

[0087] After the first feature map and the second feature map are fused with each other, a judgment result of the liver ultrasound section image is generated; the judgment result specifically includes: the liver ultrasound section image is a liver ultrasound standard section image, or the liver ultrasound section image is a liver ultrasound non-standard section image.

[0088] In one embodiment, before identifying the liver region in the liver ultrasound section image and performing mask processing on the liver ultrasound section image based on the identification result to generate the liver ultrasound section enhanced image, the method further includes:

[0089] Preprocessing the original liver ultrasound section image to obtain the liver ultrasound section image;

[0090] The original liver ultrasound section image is preprocessed, specifically including: removing operation information and patient information in the original liver ultrasound section image, and performing grayscale processing on the original liver ultrasound section image.

[0091] In one embodiment, identifying a liver region in a liver ultrasound section image, and performing mask processing on the liver ultrasound section image based on the identification result to generate a liver ultrasound section enhanced image specifically includes:

[0092] The Unet++ segmentation model is used to identify the liver region in the liver ultrasound cross-section image, and a region mask M with the same size as the liver ultrasound cross-section image is generated based on the recognition result. unet++ ;

[0093] Using the region mask M unet++ The liver ultrasound section image is masked to generate a liver ultrasound section enhanced image.

[0094] In one embodiment, using the region mask M unet++ Mask processing is performed on the liver ultrasound section image to generate a liver ultrasound section enhanced image, specifically including:

[0095] The region mask M unet++ The pixels of are added one by one to the pixels of the liver ultrasound section image to generate the liver ultrasound section enhanced image:

[0096] U M =U⊙M unet++ ;

[0097] Among them, U M Indicates enhanced ultrasound image of liver section; M unet++ U represents the ultrasound section image of the liver; ⊙ represents the addition of pixels one by one.

[0098] In one embodiment, the liver ultrasound section image is input into the global branch to generate a first feature map, specifically including:

[0099] Input the liver ultrasound section image into the convolution layer in the global branch for convolution to generate the first feature map F1;

[0100] Input the first feature map F1 into the global average pooling layer in the global branch for pooling to generate the first channel attention map a1;

[0101] Use the activation function to activate the first channel attention map a1;

[0102] Use the first feature map F1 to upsample the activated first channel attention map a1 to generate the first attention weight map

[0103] Use the first attention mining module to mine the first attention weight map Mining is performed to generate a first feature map.

[0104] In one embodiment, the first attention weight map is mined using a first attention mining module. Mining is performed to generate the first feature map, specifically including:

[0105] Judge the first attention weight map one by one Is the spatial value of each position in the first attention mining unit greater than the first threshold α1? If so, multiply the spatial value by the first hyperparameter θ1 in the first attention mining unit to generate the first eigenvalue

[0106] For the first eigenvalue The first mining feature map composed of features is used for feature extraction to generate the second attention weight map

[0107] Judge the second attention weight map one by one Is the spatial value of each position in the second attention mining unit greater than the second threshold α2? If so, multiply the spatial value by the second hyperparameter θ2 in the second attention mining unit to generate the second eigenvalue

[0108] For the second eigenvalue The composed second mining feature graph is used to extract features and generate the first feature graph.

[0109] In one embodiment, after the first feature map and the second feature map are fused with each other, a judgment result of the liver ultrasound section image is generated, which specifically includes:

[0110] Performing global average pooling on the first feature map to generate a first feature vector; performing global average pooling on the second feature map to generate a second feature vector;

[0111] Concatenate the first feature vector and the second feature vector to generate a fused feature vector;

[0112] Use the Softmax function to transform the fused feature vector into a probability distribution;

[0113] The probability distribution is input into the fully connected layer to generate the judgment result.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] Embodiment 5

[0116] In this embodiment, a model training method is also provided, wherein the model includes a global branch and a diagnostic branch; wherein the global branch and the diagnostic branch are both composed of a modified ResNet50 network, and the results output by the global branch and the diagnostic branch are respectively subjected to global average pooling through a global average pooling layer, and finally are spliced ​​and input into a fully connected layer to output a judgment result.

[0117] The specific training process includes:

[0118] S1: Acquisition and annotation of ultrasound images;

[0119] S1-1: Acquisition of liver ultrasound standard section images and liver ultrasound non-standard section images for classification;

[0120] S1-2: Preprocessing the collected images to be classified. The preprocessing is to manually cut out and remove the operation information and patient sensitive information in the ultrasound image data, and grayscale the color ultrasound image.

[0121] S1-3: The standard and non-standard sections of liver ultrasound were labeled using the labeling software Labelme, which is an open source image annotation tool.

[0122] S1-4: The labeled data in S1-3 are divided into training set, validation set and test set in the ratio of 7:2:1. The training set is used to train the model, the validation set is used to evaluate the model performance during the training process, and the test set is used for the final evaluation of the trained model.

[0123] S2: diagnostic image training set of liver ultrasound cross-sectional images;

[0124] S2-1: Establish a liver diagnostic region generator (LDRG), whose process is to input the liver ultrasound cross-section image U into the Unet++ segmentation model, segment the image, and extract the liver region mask M unet++ .

[0125] S2-2: Then, the ultrasound section image U is combined with the liver region mask M unet++ Add point by point to get the liver diagnostic image U M , the purpose is to enhance the characteristic performance of the liver area. The calculation formula is as follows:

[0126] U M =U⊙M unet++ ;

[0127] Among them, U is the ultrasound cross-section of the liver, and ⊙ represents point-by-point addition.

[0128] S2-3: Liver ultrasound cross-sectional images U and liver diagnostic images U to be classified M Data enhancement is performed, including vertical flipping, random rotation, random cropping, contrast adjustment, and horizontal flipping to obtain an enhanced data set.

[0129] S3: input the data set into the global branch and the diagnostic branch;

[0130] S3-1: The ultrasound section image U obtained in S2 is used as the input of the global branch. The liver diagnostic image U obtained in S2 M As input to the diagnostic branch.

[0131] S3-2: Both the global branch and the diagnostic branch are composed of the modified ResNet50 network, which is divided into 5 stages. The modified branch adds the liver attention mining module LAM after the third and fourth stages in the network.

[0132] S3-3: Liver Attention Mining Module LAM aims to extract all feature information useful for recognition. First, construct the attention weight map Then other regions are mined by mining the mask M. The attention mining module can guide the whole network to discover complementary object regions and details sequentially.

[0133] First, the feature map F is obtained through the convolution layer Conv, and then the channel attention map a is obtained by applying the global average pooling layer to the feature map F∈H×W×C. The channel attention map a=[a1,a2,...,a n The calculation formula of ] is as follows:

[0134]

[0135] Among them, F C (i, j) is the value of the cth feature map F at position (i, j).

[0136] Then the attention weight map is obtained by the activation function S and the bilinear up-sampling function UP Weight The calculation formula is as follows:

[0137]

[0138] If the attention weight map The spatial value in is greater than the threshold α, then the mining mask multiplies the spatial value by the hyperparameter θ to obtain Otherwise, I is obtained and then input into the next layer network structure until the feature map is extracted.

[0139] S3-4: Obtain the characteristic graph U' passing through the global branch and the characteristic graph U passing through the diagnostic branch M ';

[0140] S4: Fusion and classification of global branches and diagnostic branches;

[0141] S4-1: The fusion of branches is to add a global average pooling layer and a softmax function to the global branch and the diagnostic branch respectively, and finally classify the input features through the fully connected layer FC, such as Figure 4 .

[0142] S4-2: Fusion of the global branch output U′ and the diagnostic branch output U M ′ and the calculation formula for the predicted value Output is as follows:

[0143] U cat =Cat(U′,U′ M );

[0144] Output=FC(sigmod(U cat ));

[0145] S4-3: The predicted value Output and the true value are calculated through the BCE cross entropy loss function, and the network is trained end-to-end with the NAdam optimizer. During the training process, data enhancement, learning rate adjustment and other strategies are used to optimize network performance.

[0146] S5: Evaluate the accuracy and robustness of the network on the liver ultrasound standard section recognition task, and select the best performing model as the final deployment model.

[0147] S5-1: Use the validation set divided in S1 to evaluate the accuracy of the model, which includes calculating key indicators such as classification accuracy, recall, precision, and F1 score.

[0148] S5-2: Test the robustness of the model by introducing noise and artifacts that simulate real conditions in the validation set and observing the performance of the model under different conditions.

[0149] S5-3: Finally, the model with the best performance in terms of accuracy and robustness will be selected for final deployment to ensure its efficiency and stability in practical applications.

[0150] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for recognizing liver ultrasound standard section images, characterized in that: include: Identifying a liver region in a liver ultrasound section image, and performing mask processing on the liver ultrasound section image based on the identification result to generate an enhanced liver ultrasound section image; Input the liver ultrasound section image into the global branch to generate a first feature map; wherein the global branch includes a first attention mining module for identifying liver feature information; input the liver ultrasound section enhanced image into the diagnosis branch to generate a second feature map; wherein the diagnosis branch includes a second attention mining module for identifying liver feature information; After the first feature map and the second feature map are fused with each other, a judgment result of the liver ultrasound section image is generated; the judgment result specifically includes: the liver ultrasound section image is a liver ultrasound standard section image, or the liver ultrasound section image is a liver ultrasound non-standard section image.

2. The liver ultrasound standard section image recognition method according to claim 1, characterized in that: Before the liver region in the liver ultrasound section image is identified and the liver ultrasound section image is masked based on the identification result to generate the liver ultrasound section enhanced image, the method further includes: Preprocessing the original liver ultrasound section image to obtain the liver ultrasound section image; The preprocessing of the original liver ultrasound section image specifically includes: removing the operation information and the patient information in the original liver ultrasound section image, and performing grayscale processing on the original liver ultrasound section image.

3. The liver ultrasound standard section image recognition method according to claim 2, characterized in that: The identifying of the liver region in the liver ultrasound section image and performing mask processing on the liver ultrasound section image based on the identification result to generate the liver ultrasound section enhanced image specifically includes: The Unet++ segmentation model is used to identify the liver region in the liver ultrasound section image, and a region mask M having the same size as the liver ultrasound section image is generated based on the identification result. unet++ ; Using the region mask M unet++ Mask processing is performed on the liver ultrasound section image to generate a liver ultrasound section enhanced image.

4. The liver ultrasound standard section image recognition method according to claim 3, characterized in that: The use of the region mask M unet++ The masking process is performed on the liver ultrasound section image to generate a liver ultrasound section enhanced image, specifically comprising: The region mask M unet++ The pixels of are added one by one to the pixels of the liver ultrasound section image to generate a liver ultrasound section enhanced image: IN M =U⊙M unet++ ; Among them, U M Indicates enhanced ultrasound image of liver section; M unet++ U represents the ultrasound section image of the liver; ⊙ represents the addition of pixels one by one.

5. The liver ultrasound standard section image recognition method according to claim 1, characterized in that: Inputting the liver ultrasound section image into the global branch to generate the first feature map specifically includes: Inputting the liver ultrasound section image into the convolution layer in the global branch for convolution to generate a first feature map F1; Input the first feature map F1 into the global average pooling layer in the global branch for pooling to generate a first channel attention map a1; Use the activation function to activate the first channel attention map a1; The first feature map F1 is used to upsample the activated first channel attention map a1 to generate the first attention weight map The first attention weight map is mined using the first attention mining module Mining is performed to generate a first feature map.

6. The liver ultrasound standard section image recognition method according to claim 5, characterized in that: The first attention weight map is mined using the first attention mining module Mining is performed to generate the first feature map, specifically including: Determine the first attention weight map one by one Is the spatial value of each position in the first attention mining unit greater than the first threshold α1? If so, multiply the spatial value by the first hyperparameter θ1 in the first attention mining unit to generate the first eigenvalue For the first eigenvalue The first mining feature map composed of features is used for feature extraction to generate the second attention weight map Determine the second attention weight map one by one Is the spatial value of each position in the second attention mining unit greater than the second threshold α2? If so, multiply the spatial value by the second hyperparameter θ2 in the second attention mining unit to generate the second eigenvalue For the second eigenvalue The composed second mining feature graph is used to extract features and generate the first feature graph.

7. The liver ultrasound standard section image recognition method according to claim 1, characterized in that: The step of fusing the first feature map and the second feature map to generate a judgment result of the liver ultrasound section image specifically includes: Performing global average pooling on the first feature map to generate a first feature vector; performing global average pooling on the second feature map to generate a second feature vector; Concatenating the first feature vector and the second feature vector to generate a fused feature vector; The fused feature vector is converted into a probability distribution using a Softmax function; The probability distribution is input into the fully connected layer to generate a judgment result.

8. A liver ultrasound standard section image recognition device, characterized in that: The liver ultrasound standard section image recognition device comprises: an enhancement unit, configured to identify a liver region in a liver ultrasonic section image, and perform mask processing on the liver ultrasonic section image based on the identification result to generate an enhanced liver ultrasonic section image; An extraction unit, configured to input the liver ultrasound section image into a global branch to generate a first feature map; wherein the global branch includes a first attention mining module for identifying liver feature information; and input the liver ultrasound section enhanced image into a diagnosis branch to generate a second feature map; wherein the diagnosis branch includes a second attention mining module for identifying liver feature information; A judgment unit is used to generate a judgment result of the liver ultrasound section image after fusing the first feature map and the second feature map; the judgment result specifically includes: the liver ultrasound section image is a liver ultrasound standard section image, or the liver ultrasound section image is a liver ultrasound non-standard section image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for recognizing liver ultrasound standard section images as described in any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the liver ultrasound standard section image recognition method as described in any one of claims 1 to 7 is implemented.

Images