Veterinary ultrasonic image processing method based on deep learning

Abstract

The invention provides a veterinary ultrasonic image processing method based on deep learning. The veterinary ultrasonic image processing method comprises the following steps: S1, acquiring an original ultrasonic image through an ultrasonic probe and preprocessing the original ultrasonic image; s2, constructing a deep learning model, introducing a multi-region feature fusion network and a pyramid boundary attention module, and performing multi-region feature extraction on the image; s3, training and optimizing the model; s4, segmenting the ultrasonic image obtained in the step S2 by adopting the model trained in the step S3; s5, optimizing the ultrasonic image segmented in the step S4; and S6, superposing the ultrasonic image optimized in the step S5 on the original ultrasonic image in the step S1 for visual display, obtaining a final ultrasonic image, identifying a feature image in the ultrasonic image by constructing a pyramid boundary attention model, and performing segmentation optimization on the feature image, so that the feature image is clearer and clearer, and the accuracy of the feature image is improved. And a breeder can conveniently and accurately know the health state of the cow fetus and the recovery state of the cow uterus.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for processing veterinary ultrasonic images based on deep learning. Background Art

[0002] Ultrasound imaging technology has been widely used in medicine and veterinary medicine because of its non-invasive, real-time, safe and non-invasive characteristics.

[0003] Especially in animal husbandry, B-ultrasound imaging technology has become an important means of monitoring the status of cattle during pregnancy. However, when processing complex and blurred B-ultrasound images of cattle during pregnancy, traditional ultrasound image processing methods are difficult to process because the anatomical structure of the cattle is relatively complex, especially during pregnancy, the relationship between the fetus and the mother's organs is closer, which increases the difficulty of image analysis. Therefore, ultrasound image processing methods based on deep learning have emerged to improve the processing efficiency and accuracy of B-ultrasound images of cattle during pregnancy.

[0004] Therefore, it is necessary to provide a new veterinary ultrasound image processing method based on deep learning to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a veterinary ultrasound image processing method based on deep learning.

[0006] The veterinary ultrasound image processing method based on deep learning provided by the present invention comprises the following steps:

[0007] S1. Collecting the original ultrasound image through the ultrasound probe, graying, normalizing and denoising the original ultrasound image, and marking the image according to the fetal information in the image;

[0008] S2, building a deep learning model and introducing a multi-region feature fusion network and a pyramid boundary attention module, extracting multi-region features from the denoised image in step S1 through the multi-region feature fusion network, and capturing boundary information of the extracted multi-region features through the pyramid boundary attention module, and generating a boundary segmentation map;

[0009] S3, adjusting the model constructed in step S2 according to the image features in step S2, determining the loss function, and training and optimizing the model;

[0010] S4, using the model trained in step S3 to segment the ultrasound image obtained in step S2;

[0011] S5, optimizing the ultrasound image segmented in step S4;

[0012] S6. Superimpose the ultrasound image optimized in step S5 on the original ultrasound image in step S1 for visual display, and obtain a final ultrasound image.

[0013] Preferably, the process of constructing a deep learning model and processing the image in step S2 includes the following steps:

[0014] S21. Select the Res-Unet network as the basic network structure, introduce the pyramid squeeze attention module to expand the network's receptive field and establish a global attention mechanism, introduce the contextual local attention module to achieve accurate segmentation of the fetal boundary, and introduce the boundary fusion module to improve the image segmentation accuracy, thereby constructing a pyramid boundary attention network model;

[0015] S22, using the model in step S21 to extract information from the original ultrasound image in step S1, generating multiple images of different regions, namely, an overall image, a specific organ region map, an organ surrounding region map, and a segmentation result map, and merging the overall image and the specific organ region map, and merging the organ surrounding region map and the segmentation result map;

[0016] S23, sending the feature map after preliminary feature extraction by the feature fusion module in step S23 to the pyramid boundary attention network model for further boundary information enhancement;

[0017] S24, sending the boundary feature map obtained in step S24 and processed by the pyramid boundary attention module to one or more convolutional layers to generate a higher precision boundary segmentation map.

[0018] Preferably, the loss function in step S3 is a combined mixed loss function of Focal loss and Dice loss.

[0019] Preferably, the feature fusion module in step S22 includes a basic residual block and a CBAM attention module, and the basic residual block is used again to further refine the features of the multiple images obtained in step S21, and finally residual connection is performed.

[0020] Preferably, in step S3, the Adam optimizer is used to optimize the model, and the cosine annealing algorithm is used to adjust the learning rate.

[0021] Preferably, the deep convolutional neural network in step S22 is specifically ResNet18.

[0022] Preferably, in step S5, the ultrasound image after segmentation obtained in step S4 is further optimized by performing smoothing, edge detection, morphological operations and boundary optimization on the image.

[0023] Preferably, the specific process of modifying the pyramid squeezing attention module in step S23 includes the following steps:

[0024] Using convolution kernels of various sizes in the pyramid squeeze attention module;

[0025] Add additional inter-channel interaction layers to recalibrate the importance of channels through two fully connected layers in the SEWeight module;

[0026] The global attention mechanism is strengthened to help the network recognize features in the entire image while maintaining attention to local details of the ultrasound image.

[0027] Preferably, the specific process of modifying the contextual local attention module in step S23 includes the following steps:

[0028] Choose to directly multiply the multi-scale feature map with the channel attention vector to avoid information loss caused by normalization;

[0029] All local attention feature maps of different resolutions will be upsampled to a uniform 56x56 pixels and preliminarily integrated through Block 1;

[0030] Block2 is used to further process these integrated feature maps.

[0031] Compared with the related art, the veterinary ultrasound image processing method based on deep learning provided by the present invention has the following beneficial effects:

[0032] 1. The present invention provides a veterinary ultrasound image processing method based on deep learning, which identifies feature images in ultrasound images by constructing a pyramid boundary attention model, segments the feature images, and optimizes the boundaries of the segmented feature images, so that the feature images are clearer and more specific, which is convenient for breeders to accurately know the health status of cattle fetuses and the recovery status of cow uteruses;

[0033] 2. When constructing the pyramid boundary attention model, the pyramid squeeze attention module and the context local attention module are introduced to expand the network's receptive field and establish a global attention mechanism, so that the network can capture the fetal image more accurately when recognizing features;

[0034] 3. By mixing multiple functions to form a mixed loss function, the class imbalance problem in pregnant bovine B-ultrasound images is effectively solved, and the segmentation and classification performance of the model is improved. Focal Loss is used to adjust the weights of difficult and easy samples, Dice Loss is used to improve segmentation accuracy, and the boundary loss function focuses on optimizing the segmentation effect of the boundary area, thereby improving the robustness and accuracy of ultrasound image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The overall wireframe flow chart of the deep learning-based veterinary ultrasound image processing method provided by the present invention;

[0036] Figure 2 A wireframe flow chart for constructing a pyramid boundary attention model provided by the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0038] Please refer to Figure 1 — Figure 2 ,in, Figure 1 The overall wireframe flow chart of the deep learning-based veterinary ultrasound image processing method provided by the present invention; Figure 2 A wireframe flow chart for constructing a pyramid boundary attention model provided by the present invention.

[0039] In the specific implementation process, the veterinary ultrasound image processing method based on deep learning provided by the present invention includes the following steps:

[0040] S1. Collect the original ultrasound image through the intelligent handheld B-ultrasound terminal, grayscale, normalize and denoise the original ultrasound image, and mark the fetus' head, body, limbs and other information in the image;

[0041] S2, building a deep learning model and introducing a multi-region feature fusion network and a pyramid boundary attention module, extracting multi-region features from the denoised image in step S1 through the multi-region feature fusion network, and capturing boundary information of the extracted multi-region features through the pyramid boundary attention module;

[0042] S3, adjusting the model constructed in step S2 according to the image features in step S2, determining the loss function, and training and optimizing the model;

[0043] S4, using the model trained in step S3 to segment the ultrasound image obtained in step S2;

[0044] S5, optimizing the ultrasound image segmented in step S4;

[0045] S6. Superimpose the ultrasound image optimized in step S5 on the original ultrasound image in step S1 for visual display, and obtain a final ultrasound image.

[0046] The process of building a deep learning model and processing the image in step S2 includes the following steps:

[0047] S21. Select the Res-Unet network as the basic network structure, introduce the pyramid squeeze attention module to expand the network's receptive field and establish a global attention mechanism, introduce the contextual local attention module to achieve accurate segmentation of the fetal boundary, and introduce the boundary fusion module to improve the image segmentation accuracy, thereby constructing a pyramid boundary attention network model;

[0048] S22, using the model in step S21 to extract information from the original ultrasound image in step S1, generating multiple images of different regions, namely, an overall image, a specific organ region map, an organ surrounding region map, and a segmentation result map, and merging the overall image and the specific organ region map, and merging the organ surrounding region map and the segmentation result map;

[0049] S23, sending the feature map after preliminary feature extraction in step S22 to the pyramid boundary attention network model for further boundary information enhancement;

[0050] S24, sending the boundary feature map obtained in step S23 and processed by the pyramid boundary attention module to one or more convolutional layers, and generating a higher precision boundary segmentation map;

[0051] It should be noted that the encoder of the network is built on the basis of ResNet50, and the 3×3 convolution in the Bottleneck Block of ResNet50 is replaced by the pyramid squeeze attention module to form the EPSANet Block;

[0052] The overall image refers to the entire ultrasound image, which provides a global understanding of the entire image and can capture the overall characteristics and contextual information of the fetal condition;

[0053] The specific organ region map is an image of the fetal head, torso, limbs and other organs, which focuses on the fetal organ location area, allowing the network to extract the characteristics of the fetal organs in a targeted manner;

[0054] The organ periphery map is the area surrounding a specific organ, which helps to understand the growth trend of fetal organs;

[0055] The segmentation result image is the mask image obtained by segmentation, which enables the network to accurately capture the boundaries and morphological features of the fetus, accurately predict the early pregnancy, mid-pregnancy and postpartum status of the cow, and detect the health status of the fetus and the recovery of the uterus.

[0056] The loss function in step S3 is a combined mixed loss function of Focal loss and Dice loss;

[0057] It should be noted that the formula of the Focal loss function is:

[0058] L focal (p t )=-(1-p t ) γ log(p t ),

[0059] p t : The model predicts the probability of a certain category;

[0060] γ: used to adjust the weight of difficult and easy samples, usually set to 2.

[0061] Suppose there is a pixel point, and the model predicts that the probability that it belongs to the fetal heart is 0.9 (p t ), but in fact it does belong to the heart area. In this case, since (p t ) is large, even if the prediction is correct, the value of Focal Loss will be relatively low, because the sample is easier for the model to classify. However, if (p t ) is small (such as 0.3), even if the prediction is wrong, Focal Loss will give a higher weight, prompting the model to pay more attention to these "difficult" samples;

[0062] The formula of the Dice loss function is:

[0063]

[0064] y: true label, indicating the true category of each pixel;

[0065] p(y): predicted label, indicating the probability that each pixel is predicted by the model to be a certain category;

[0066] N: total number of pixels, that is, the number of all pixels in the image.

[0067] In a pregnant cow ultrasound image, suppose we have 1,000 pixels that need to be judged whether they belong to the fetal heart region. The actual fetal heart region is marked by (y), and the model prediction result is represented by (p(y)). Dice Loss calculates the similarity between the two distributions. The closer the value is to 0, the more consistent the prediction result is with the actual label.

[0068] Then the formula of the mixing function is:

[0069] L h =L focal +αL dice ,

[0070] ɑ: The weight of the combined loss function, used to balance the contribution of Focal Loss and Dice Loss, usually set to 0.5, meaning that both are equally important.

[0071] For the boundaries of key features such as the fetal heart, the model may produce some slight deviations. The boundary loss function helps the model locate the pixels of these key boundaries more accurately, thereby improving the quality of the final segmentation. The boundary loss function is:

[0072]

[0073] Then the total loss function of the network is:

[0074] L=L h +βL bce ,

[0075] In order to ensure that the model can both perform overall classification well and accurately capture the boundaries of organs, β = 0.5 can be selected to optimize both capabilities at the same time;

[0076] Through the combination of these loss functions, the class imbalance problem in pregnant cow B-ultrasound images can be effectively solved, and the segmentation and classification performance of the model can be improved. FocalLoss is used to adjust the weights of difficult and easy samples, DiceLoss is used to improve segmentation accuracy, and the boundary loss function focuses on optimizing the segmentation effect of the boundary area, thereby improving the robustness and accuracy of ultrasound image processing.

[0077] The feature fusion module in step S22 includes a basic residual block and a CBAM attention module. The basic residual block is used again to further refine the features of the multiple images obtained in step S21, and finally a residual connection is performed.

[0078] In step S3, the Adam optimizer is used to optimize the model, and the cosine annealing algorithm is used to adjust the learning rate. The initial learning rate is 0.001. During the training process, the Mini-batch is 6 and the epoch is 180. Pytorch is used to implement the pyramid boundary attention network model.

[0079] In step S5, the segmented ultrasound image obtained in step S4 is further optimized by performing smoothing, edge detection, morphological operations and boundary optimization on the image.

[0080] The specific process of modifying the pyramid squeeze attention module in step S21 includes the following steps:

[0081] Using convolution kernels of various sizes in the pyramid squeeze attention module;

[0082] Add additional inter-channel interaction layers to recalibrate the importance of channels through two fully connected layers in SEWeightmodule;

[0083] Strengthen the global attention mechanism to help the network recognize features across the entire image while maintaining attention to local details of the ultrasound image;

[0084] It should be noted that the pyramid squeeze attention module is specifically designed to perform global average pooling on the feature map to extract the global information of the entire feature map and generate channel-level statistics. Let X∈R C×H×W is the input feature map, where H, W, and C represent its height, width, and number of channels respectively, and the global average pooling g c It can be expressed by the following formula:

[0085]

[0086] After global average pooling, the feature map is reduced in dimension using the fully connected layer W0. After the Relu activation function δ, the fully connected layer W1 is used to increase the dimension. Finally, the Sigmoid function σ is used to obtain the channel weight W. C , the formula is as follows:

[0087] w c =σ(W1δ(W0(g c ))),

[0088] Through two fully connected layers, the linear information between channels can be combined more effectively, which is conducive to the interaction of high and low channel dimensional information. The Sigmoid function can be used to assign weights to channels after channel interaction, so as to extract information more effectively.

[0089] The pyramid squeeze attention module extracts information about input features in a four-branch manner. Let the input feature be X∈R C×H×W The four branches use convolution kernels of sizes k0=3×3, k1=5×5, k2=7×7 and k3=9×9 to perform group convolution on the input features. The groups are G0=1, G1=4, G2=8 and G3=16, respectively, which can generate feature maps with different receptive fields and depths. By compressing the channel dimension of the input tensor to 1 / 4 of the original, the spatial information F0, F1, F2 and F3 with different scales on each channel-level feature map can be effectively extracted (where, ), merge the spatial information of different scales to obtain the feature map F∈R C×H×W , the formula is as follows:

[0090]

[0091] Where Conv represents the convolution operation, Cat represents the concatenation operation,

[0092] By extracting the channel attention vector information from the multi-scale feature map, the attention weight vectors of the feature maps of different scales are obtained. Mathematically, the vector of attention weights is expressed by the following formula:

[0093] S i =SEWeight(F i )i=0,1,2,3,

[0094] , SEWeight represents the squeeze and incentive weighted operation;

[0095] In order to achieve the interaction and fusion of attention information without destroying the attention vector of the original channel, the multi-scale channel attention vectors are merged. The formula is as follows:

[0096] S = Cat([F0, F1, F2, F3]),

[0097] Where S i Yes F i The attention vector S is a multi-scale attention vector. For the S multi-scale attention vector, soft attention is used to enable the network to adaptively select the weights of channels of different spatial scales. The soft attention formula is as follows:

[0098]

[0099] Softmax represents the activation function,

[0100] Then the multi-scale feature map F i Multiply it with the corresponding channel weight to get the corrected multi-scale feature map Y i , the formula is as follows:

[0101]

[0102] Compared with summation, merging channels is more effective because it can fully maintain the feature representation without destroying the information of the original feature map. The output result is shown in the formula:

[0103] Out=Cat([Y0,Y1,Y2,Y3]),

[0104] The pyramid squeezing attention module can integrate multi-scale spatial information and cross-channel attention into each feature group. Therefore, the pyramid squeezing attention module can achieve better information interaction, thereby effectively combining local and global channel attention.

[0105] The specific process of modifying the contextual local attention module in step S21 includes the following steps:

[0106] Choose to directly multiply the multi-scale feature map with the channel attention vector to avoid information loss caused by normalization;

[0107] All local attention feature maps of different resolutions will be upsampled to a uniform 56x56 pixels and preliminarily integrated through Block 1;

[0108] Block2 is used to further process these integrated feature maps.

[0109] The circuits and controls involved in the present invention are all prior art and will not be described in detail here.

[0110] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A veterinary ultrasound image processing method based on deep learning, characterized in that: The steps include: S1. Collecting the original ultrasound image through the ultrasound probe, graying, normalizing and denoising the original ultrasound image, and marking the image according to the fetal information in the image; S2, build a deep learning model and introduce a multi-region feature fusion network and a pyramid boundary attention module, extract multi-region features from the denoised image in step S1 through the multi-region feature fusion network, and capture boundary information of the extracted multi-region features through the pyramid boundary attention module and weightedly fuse them through the attention mechanism; S3, adjusting the model constructed in step S2 according to the image features in step S2, determining the loss function, and training and optimizing the model; S4, using the model trained in step S3 to segment the ultrasound image obtained in step S2; S5, optimizing the ultrasound image segmented in step S4; S6. Superimpose the ultrasound image optimized in step S5 on the original ultrasound image in step S1 for visual display, and obtain a final ultrasound image.

2. The method for processing veterinary ultrasound images based on deep learning according to claim 1, characterized in that: The process of constructing a deep learning model and processing the image in step S2 includes the following steps: S21. Select the Res-Unet network as the basic network structure, introduce the pyramid squeeze attention module to expand the network's receptive field and establish a global attention mechanism, and introduce the contextual local attention module and the boundary fusion module to construct a pyramid boundary attention network model; S22, using the model in step S21 to extract information from the original ultrasound image in step S1, generating multiple images of different regions, namely, an overall image, a specific organ region map, an organ surrounding region map, and a segmentation result map, and merging the overall image and the specific organ region map, and merging the organ surrounding region map and the segmentation result map; S23, sending the feature map after preliminary feature extraction by the feature fusion module in step S23 to the pyramid boundary attention network model for further boundary information enhancement; S24, sending the boundary feature map obtained in step S24 and processed by the pyramid boundary attention module to one or more convolutional layers to generate a higher precision boundary segmentation map.

3. The method for processing veterinary ultrasonic images based on deep learning according to claim 2, characterized in that: The loss function in step S3 is a combined mixed loss function of Focalloss and Diceloss and a boundary loss function.

4. The method for processing veterinary ultrasound images based on deep learning according to claim 3 is characterized in that: The feature fusion module in step S22 includes a basic residual block and a CBAM attention module. The basic residual block is used again to further refine the features of the multiple images obtained in step S21, and finally residual connection is performed.

5. The method for processing veterinary ultrasound images based on deep learning according to claim 4 is characterized in that: In step S3, the Adam optimizer is used to optimize the model, and the cosine annealing algorithm is used to adjust the learning rate.

6. The method for processing veterinary ultrasound images based on deep learning according to claim 5, characterized in that: The deep convolutional neural network in step S22 is specifically ResNet18.

7. The method for processing veterinary ultrasound images based on deep learning according to claim 6, characterized in that: In step S5, the segmented ultrasound image obtained in step S4 is further optimized by performing smoothing, edge detection, morphological operations and boundary optimization on the image.

8. The method for processing veterinary ultrasound images based on deep learning according to claim 7, characterized in that: The specific process of modifying the pyramid squeezing attention module in step S23 includes the following steps: Using convolution kernels of various sizes in the pyramid squeeze attention module; Add additional inter-channel interaction layers to recalibrate the importance of channels through two fully connected layers in the SEWeight module; The global attention mechanism is strengthened to help the network recognize features in the entire image while maintaining attention to local details of the ultrasound image.

9. The method for processing veterinary ultrasound images based on deep learning according to claim 8, characterized in that: The specific process of modifying the context local attention module in step S23 includes the following steps: Choose to directly multiply the multi-scale feature map with the channel attention vector to avoid information loss caused by normalization; All local attention feature maps of different resolutions will be upsampled to a uniform 56x56 pixels and preliminarily integrated through Block 1; Block2 is used to further process these integrated feature maps.

Images