Ultrasonic image processing method and training method and device based on adaptive loss function

Abstract

The invention is suitable for the technical field of ultrasound, and provides an ultrasound image processing method, and a training method and equipment based on an adaptive loss function. According to the embodiment of the invention, the model training is carried out through the adaptive loss function, and the adaptive loss function can balance the weights of a plurality of anatomical structures and a plurality of loss functions in the training process, thereby facilitating the improvement of the detection rate of small anatomical structures. Moreover, a network model of a double-end structure with an attention mechanism is designed, so that the segmentation and key point detection precision can be improved.

Description

Technical Field

[0001] This application belongs to the field of ultrasound technology, and in particular relates to an ultrasound image processing method, a training method and device based on an adaptive loss function. Background Technology

[0002] Ultrasound image segmentation and keypoint detection are two of the most fundamental and crucial tasks in modern medical image processing. In recent years, with the rapid development of deep learning technology, many studies have achieved real-time segmentation of regions of interest in ultrasound images, and have continuously improved segmentation accuracy through various methods. Among related technologies, ultrasound image processing models are mostly applied to single-task processing, with the loss function focusing on the error between the final output of the single-task processing and the actual result. When applied to multi-task processing, the multi-task processing accuracy of the model is limited. Summary of the Invention

[0003] This application provides an ultrasound image processing method, a training method and device based on an adaptive loss function, which can improve the accuracy of multi-task processing of ultrasound images.

[0004] The first aspect of this application provides an ultrasound image processing method, comprising: acquiring an ultrasound image to be processed; inputting the ultrasound image to be processed into a dual-head network model to obtain a coarse segmentation result, a fine segmentation result, and / or a key point detection result of the ultrasound image to be processed, wherein the dual-head network model includes an encoder, a coarse detection module, and a fine detection module, the dual-head network model being used to input the ultrasound image to be processed into the encoder to obtain first feature images at multiple scales from shallow to deep; inputting the first feature images at each scale into the coarse detection module to obtain global contour images at each scale and the coarse segmentation result; inputting the global contour images at each scale into the fine detection module, and obtaining the fine segmentation result and / or the key point detection result through a fine detection head with an attention mechanism.

[0005] A second aspect of this application provides a training method based on an adaptive loss function for training the dual-head network model described in the first aspect. The training method includes: acquiring sample ultrasound images; inputting the sample ultrasound images into the model to be trained for training to obtain prediction results, the prediction results including coarse image segmentation results, fine segmentation results, and key point detection results of the sample ultrasound images; calculating the loss value of an adaptive loss function based on the prediction results and the gold standard corresponding to the sample ultrasound images, the adaptive loss function being a weighted sum of a first sub-function, a second sub-function, and a third sub-function, the first sub-function being associated with the coarse image segmentation results, the second sub-function being associated with the fine segmentation results, and the third sub-function being associated with the key point detection results; performing iterative training based on the loss value, updating the weights of the adaptive loss function in each iteration until a preset number of iterations is completed, and selecting the optimal iteration result as the dual-head network model.

[0006] The third aspect of this application provides an ultrasound image processing apparatus, comprising: an image acquisition unit for acquiring an ultrasound image to be processed; and an image processing unit for inputting the ultrasound image to be processed into a dual-head network model to obtain coarse segmentation results, fine segmentation results, and / or key point detection results of the ultrasound image to be processed. The dual-head network model includes an encoder, a coarse detection module, and a fine detection module. The dual-head network model is used to input the ultrasound image to be processed into the encoder to obtain first feature images at multiple scales from shallow to deep; input the first feature images at each scale into the coarse detection module to obtain global contour images at each scale and the coarse segmentation results; and input the global contour images at each scale into the fine detection module, and obtain the fine segmentation results and / or the key point detection results through a fine detection head with an attention mechanism.

[0007] A training device based on an adaptive loss function, provided in a fourth aspect of this application, includes: a sample acquisition unit for acquiring sample ultrasound images; a model processing unit for inputting the sample ultrasound images into a model to be trained for training to obtain prediction results, the prediction results including coarse image segmentation results, fine segmentation results, and key point detection results of the sample ultrasound images; a loss calculation unit for calculating the loss value of an adaptive loss function based on the prediction results and the gold standard corresponding to the sample ultrasound images, the adaptive loss function being a weighted sum of a first sub-function, a second sub-function, and a third sub-function, the first sub-function being associated with the coarse image segmentation results, the second sub-function being associated with the fine segmentation results, and the third sub-function being associated with the key point detection results; and an iterative training unit for performing iterative training based on the loss value, updating the weights of the adaptive loss function in each iteration until a preset number of iterations is completed, and selecting the optimal iteration result as the dual-head network model.

[0008] A fifth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the training method based on an adaptive loss function as described in any of the first aspects.

[0009] A sixth aspect of this application provides an ultrasound device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the ultrasound image processing method as described in any of the first aspects.

[0010] A seventh aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the training method based on the adaptive loss function or the steps of the ultrasound image processing method described above.

[0011] The eighth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the steps of the training method based on the adaptive loss function described above, and when run on an ultrasonic device, causes the ultrasonic device to perform the steps of the ultrasonic image processing method described above.

[0012] In the embodiments of this application, the dual-head network model can process the first feature image at multiple scales through a coarse detection module and a fine detection module respectively. The coarse detection module outputs a global contour image and a coarse segmentation result, while the fine detection module outputs a fine segmentation result and / or the key point detection result. The coarse detection module can extract the features of the target part from the global information, and then the fine detection module with an attention mechanism can complete the fine segmentation of multiple anatomical structures and key point detection in the ultrasound image. Thus, the dual-head network model with an attention mechanism can output different results for multi-task processing of ultrasound images, which can improve the accuracy of segmentation and key point detection.

[0013] In the embodiments of this application, by using the adaptive loss function as a weighted sum of the first sub-function, the second sub-function, and the third sub-function, and performing iterative training based on the loss value, and updating the weights of the adaptive loss function in each iteration until a preset number of iterations is completed, and selecting the optimal iteration result as the dual-head network model, the weights of multiple anatomical structures and multiple loss functions during training can be balanced, which helps to improve the detection rate of small anatomical structures, thereby improving the accuracy of multi-task processing of ultrasound images. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram illustrating the implementation flow of the ultrasound image processing method provided in the embodiments of this application; Figure 2 This is a schematic diagram of a dual-head network model provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the implementation process of the training method based on an adaptive loss function provided in the embodiments of this application; Figure 4 This is an anatomical diagram of the long-axis section of the aortic arch in the suprasternal notch provided in the embodiments of this application; Figure 5 This is a standard cross-sectional view of the long axis of the aortic arch in the suprasternal notch provided in the embodiments of this application; Figure 6 This is a schematic diagram showing the comparison results of different image processing methods provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of an ultrasound image processing device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a training device based on an adaptive loss function provided in an embodiment of this application; Figure 9 This is a schematic diagram of the device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are protected by this application.

[0017] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0018] In the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0019] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0020] Ultrasound image segmentation and keypoint detection are two of the most fundamental and crucial tasks in modern medical image processing. In recent years, with the rapid development of deep learning technology, many studies have achieved real-time segmentation of regions of interest in ultrasound images, and have continuously improved segmentation accuracy through various methods. Among related technologies, dual-head network models are mostly used for single-task processing. Adaptive loss functions focus on the error between the final output of single-task processing and the gold standard, but their accuracy is limited when applied to multi-task processing.

[0021] Therefore, this application proposes an adaptive loss function for model training. This adaptive loss function balances the weights of multiple anatomical structures and multiple loss functions during training, helping to improve the detection rate of fine anatomical structures in the trained network model. Furthermore, a dual-head network model with an attention mechanism is designed, which can improve the accuracy of segmentation and keypoint detection. However, the multi-task processing accuracy of the model is limited when applied to multi-task processing.

[0022] To illustrate the technical solution of this application, specific embodiments are described below.

[0023] Please refer to Figure 1 , Figure 1 A schematic diagram illustrating the implementation flow of the ultrasound image processing method provided in this application embodiment is shown. This method can be applied to ultrasound equipment.

[0024] Specifically, the ultrasound image processing method described above may include the following steps S101 to S102.

[0025] Step S101: Obtain the ultrasound image to be processed.

[0026] The ultrasound image to be processed refers to the ultrasound image that requires image segmentation and / or key point detection.

[0027] Step S102: Input the ultrasound image to be processed into the dual-head network model to obtain the coarse segmentation result, fine segmentation result and / or key point detection result of the ultrasound image to be processed.

[0028] The dual-head network model can include an encoder, a coarse detection module, and a fine detection module.

[0029] Specifically, a dual-head network model can be used to input the ultrasound image to be processed into an encoder to obtain first feature images at multiple scales from shallow to deep. Subsequently, the first feature images at each scale are input into a coarse detection module to obtain global contour images and coarse image segmentation results at each scale. Next, the global contour images at each scale are input into a fine detection module, where a fine detection head with an attention mechanism is used to obtain detailed segmentation results and / or keypoint detection results.

[0030] For details, please refer to Figure 2The encoder may include a convolutional block (Conv Block) and multiple feature extraction modules. The convolutional block (Conv Block) can coarsely extract features from the input ultrasound image to be processed, reducing the size of the ultrasound image. Each feature extraction module can be a ConNextV2 block. Different ConNextV2 blocks can extract feature maps at different scales, thus obtaining first feature images at multiple scales from shallow to deep.

[0031] For example, please refer to Figure 2 The aforementioned scales, ranging from shallow to deep, can be set to B×C×W / 4×H / 4, B×2C×W / 8×H / 8, B×4C×W / 16×H / 16, and B×8C×W / 32×H / 32, where B represents the batch size, C represents the number of channels, H represents the image height, and W represents the image width. It should be noted that the number and size of these scales can be set manually or automatically according to task requirements. Each scale corresponds to a spatial resolution (determined by the four dimensions B, C, W, and H). In the following text, the scale size can be correlated one-to-one with the spatial resolution size; the scale with the smallest spatial resolution is called the minimum scale. For example, in the previous example, B×8C×W / 32×H / 32 is the minimum scale.

[0032] The coarse detection module can roughly segment the ultrasound image to be processed from the first feature images at various scales to obtain a global contour image, and then use the global contour image to generate a coarse segmentation result. The global contour image can be used to locate the region that needs fine segmentation, that is, the region where the target part is located. The coarse segmentation result can be a mask image of the target part.

[0033] The fine detection module can be used to process global contour images at various scales using a fine detection head with an attention mechanism, obtaining detailed segmentation results and / or keypoint detection results. For details, please refer to... Figure 2The fine detection module can include a Convolutional Block Attention Module (CBAM) and a Refined Head. The CBAM and Refined Head can perform more detailed segmentation of the global contour image at various scales, obtaining mask images of each anatomical region within the target area and heatmaps of key points. At this point, the mask images of each anatomical region can serve as the fine segmentation results, and the heatmaps corresponding to each key point can serve as the key point detection results. The image segmentation result of each anatomical region and the heatmap of each key point can each be considered as a fine detection sub-result for a specific category.

[0034] In the embodiments of this application, the dual-head network model can process the first feature image at multiple scales through a coarse detection module and a fine detection module respectively. The coarse detection module outputs a global contour image and a coarse image segmentation result, while the fine detection module outputs a fine segmentation result and / or a key point detection result. The coarse detection module can extract the features of the target part from the global information, and then the fine detection module with an attention mechanism can complete the fine segmentation of multiple anatomical structures and key point detection in the ultrasound image. Thus, the dual-head network model with an attention mechanism can output different results for multi-task processing of ultrasound images, which can improve the accuracy of segmentation and key point detection.

[0035] In some embodiments of this application, the connected domains of each anatomical structure within the target region segmented by the dual-headed network model are adjacent.

[0036] In some embodiments of this application, please refer to Figure 2 The coarse detection module includes the first residual module (ResNetBlock) and the first splicing module (Concat).

[0037] The process of inputting the first feature images at each scale into the coarse detection module to obtain global contour images at each scale may include: inputting the first feature image at the smallest scale into the first residual module to obtain the global contour image at the previous scale of the smallest scale; at each scale other than the smallest scale, fusing the first feature image and the global contour image at the same scale through the first stitching module, and inputting the fused image into the first residual module to obtain the corresponding global contour image at the previous scale.

[0038] The first residual module is the residual module within the coarse detection module. The residual module can extract details from the image input to the residual module through convolutional layers on the main path, and restore it to the size of the previous scale through upsampling operations.

[0039] For details, please refer to Figure 2 The first feature image at the smallest scale is input into the first residual module. The first residual module performs residual calculation on the first feature image at the smallest scale to obtain the global contour image at the next larger scale. For other scales besides the smallest scale, the first stitching module fuses the first feature image and the global contour image at the same scale across channels to obtain a fused image. This fused image is then input into the first residual module for residual calculation to obtain the corresponding global contour image at the next larger scale, thus obtaining global contour images at all scales.

[0040] In some embodiments of this application, the coarse detection module may further include a coarse detection head and a first upsampling module. Inputting the first feature images at various scales into the coarse detection module to obtain the image coarse segmentation result may include: inputting the global contour image at the largest scale into the coarse detection head and the first upsampling module for detection and upsampling to obtain the image coarse segmentation result.

[0041] Specifically, the largest-scale global contour image is input into the coarse detection head. The coarse detection head can extract the feature information of the global contour of the target area in the global contour image. Then, the scale is restored through the first upsampling module to obtain the coarse image segmentation result. The coarse image segmentation result can be an image that records the global contour of the target area, such as a mask image of the target area.

[0042] In some embodiments of this application, the fine detection module may include a second concat module, an attention mechanism module, a second residual module (ResNet Block), a fine detection head, and a second upsampling module (Upsample).

[0043] Global contour images at various scales are input into the fine detection module. Through a fine detection head with an attention mechanism, detailed segmentation results and / or key point detection results are obtained. This can include: using the global contour image of the previous scale at the smallest scale as the fine detection feature image of the previous scale at the smallest scale; at all scales other than the smallest scale, the global contour feature image and the fine detection feature image of the same scale are fused in channels, and after the attention mechanism module performs attention mechanism operations on the fused image in both spatial and channel dimensions, the residual calculation operation is performed through the second residual module to obtain the corresponding fine detection feature image of the previous scale; the fine detection feature image of the largest scale is input into the fine detection head and the second upsampling module for detection and upsampling to obtain the ultrasound image fine segmentation results and / or key point detection results.

[0044] The second residual module is the residual module of the fine detection module. Similarly, the residual module can extract details from the image input to the residual module through convolutional layers on the main path, and restore it to the size of the previous scale through upsampling operations.

[0045] For details, please refer to Figure 2 The global contour image at the previous scale of the smallest scale output by the first residual module is used as the fine detection feature image at the previous scale. At scales other than the smallest scale, the second stitching module can fuse the global contour image and the fine detection feature image at the same scale in channels. The fused feature map is then subjected to attention mechanism operations in both spatial and channel dimensions by the attention mechanism module to obtain the attention-processed feature map. The attention-processed feature map is then input into the second residual module for residual calculation to obtain the fine detection feature image at the previous scale. After obtaining the fine detection feature image at the largest scale, it is input into the fine detection head for detection and segmentation, which yields the segmentation results of several anatomical parts and the detection results of key points. Subsequently, the scale is restored by the upsampling module to obtain the detailed segmentation results and / or key point detection results.

[0046] In the embodiments of this application, a dual-head network structure is used to realize a coarse-to-fine segmentation process. The coarse detection module improves the features of the target part from the global information, and then the fine detection module is used to complete the fine segmentation of multiple anatomical structures and key point detection in the ultrasound image by combining the features of the target part. At the same time, an attention mechanism operation of channel and spatial dimension is established between the two heads to improve the network model’s attention to the region of interest.

[0047] Please refer to Figure 3 , Figure 3 This illustration shows a schematic diagram of the implementation flow of the training method based on an adaptive loss function provided in an embodiment of this application. This method can be applied to electronic devices to train the aforementioned dual-head network model. The aforementioned electronic device can be a computer, smartphone, or other intelligent device used for model training; this application does not impose any limitations on this. Considering that the dual-head network model is ultimately used in ultrasound equipment, the aforementioned electronic device can also refer to ultrasound equipment.

[0048] Specifically, the training method based on the adaptive loss function described above may include the following steps S301 to S304.

[0049] Step S301: Obtain the ultrasound image of the sample.

[0050] The sample ultrasound images are used for model training and can be obtained by performing ultrasound scans on several patients in real-world scenarios. To ensure the accuracy of model processing, multiple sample ultrasound images can be used.

[0051] Step S302: Input the sample ultrasound image into the model to be trained for training and obtain the prediction result.

[0052] Specifically, the sample ultrasound image is input into the model to be trained. The model can process the sample ultrasound image to obtain the prediction result. The prediction result can include the coarse segmentation result, fine segmentation result, and key point detection result of the sample ultrasound image.

[0053] Step S303: Calculate the loss value of the adaptive loss function based on the prediction results and the gold standard corresponding to the sample ultrasound images.

[0054] The gold standard corresponding to the ultrasound images of the samples can be obtained through automatic or manual annotation.

[0055] In the embodiments of this application, the adaptive loss function is a weighted sum of the first sub-function, the second sub-function, and the third sub-function. That is, the adaptive loss function can be expressed as: .in, and This represents the first sub-function and its weight. and This represents the second sub-function and its weight. and This represents the third subfunction and its weight.

[0056] in, A set of training hyperparameters allows for dynamic adjustment of the weights of three different sub-loss functions during training to adapt to different types of data and tasks, thereby improving the model's generalization ability.

[0057] The first sub-function is associated with the coarse image segmentation result and can characterize the error between the coarse image segmentation result and the gold standard; the second sub-function is associated with the fine segmentation result and can characterize the error between the fine segmentation result and the gold standard; the third sub-function is associated with the key point detection result and can characterize the error between the key point detection result and the gold standard.

[0058] Step S304: Perform iterative training based on the loss value, and update the weights of the adaptive loss function during each iteration until the preset number of iterations is completed, and select the best iteration result as the dual-head network model.

[0059] In the embodiments of this application, during each iteration, based on the loss value of the adaptive loss function, the model parameters within each module of the network model (such as the aforementioned encoder, coarse detection module, and fine detection module) can be adjusted to adjust the network model's prediction trend for the input image. Furthermore, the weights of the adaptive loss function can be adjusted. To balance the network model's focus on different aspects such as anatomical structures and key points, and to prevent the network model from overemphasizing large anatomical structures during training, the model parameters used in the optimal iteration are selected as the model parameters for the final dual-head network model.

[0060] In the embodiments of this application, the optimal iteration result can refer to the iteration result in which the network model has the smallest mean intersection over union (MIOU) on the validation set.

[0061] In the embodiments of this application, by using the adaptive loss function as a weighted sum of the first sub-function, the second sub-function, and the third sub-function, and performing iterative training based on the loss value, and updating the weights of the adaptive loss function in each iteration until a preset number of iterations is completed, and selecting the optimal iteration result as the dual-head network model, the weights of multiple anatomical structures and multiple loss functions during training can be balanced, which helps to improve the detection rate of small anatomical structures, thereby improving the accuracy of multi-task processing of ultrasound images.

[0062] In some embodiments of this application, the first sub-function may be represented as: .

[0063] in, This represents the coarse segmentation result of the model under test detecting the global contour of the target region in the sample ultrasound image. This represents the gold standard for coarse testing. Among them, This represents the intersection between the coarse image segmentation result and the gold standard for coarse detection. This represents the set between the coarse image segmentation results and the gold standard for coarse detection.

[0064] In some embodiments of this application, the second sub-function may be expressed as: ; in, This indicates the number of categories in the detailed segmentation sub-results within the detailed segmentation result; that is, the detailed segmentation result may include... Detailed segmentation results for each category, ≥1. Indicates the category number. Indicates the first The weights of each category, Indicates the first Loss values ​​for each category, This indicates that the training model is working on the target region in the sample ultrasound image. Detailed segmentation results obtained by detecting anatomical structures of each category. Indicates the first The gold standard for each category. It should be noted that the segmentation result of each anatomical region output by the precision detection module can be considered as a detailed sub-segmentation result for each category.

[0065] In some embodiments of this application, the weight of each category is obtained by normalizing the loss values ​​of all categories.

[0066] More specifically, the above normalization can be achieved using the Softmax function, that is, the weights of each class. .

[0067] In this way, by performing Softmax calculation on the loss values ​​of all categories, larger loss values ​​can be converted into larger weights. In ultrasound images, the smaller the anatomical structure, the more the loss value will increase when there is a slight deviation in the segmentation boundary. This allows the weights to focus on difficult categories, balance the weights of multiple anatomical structures during training, and improve the segmentation accuracy of the network model for detecting difficult anatomical structures.

[0068] In some embodiments of this application, the key point detection results include Key point detection sub-results, ≥1. In this case, the third sub-function can be expressed as: .

[0069] in, Indicates the key point number. This indicates that the model to be trained is trained on the first sample ultrasound image. The key point detection results are obtained by detecting each key point. Indicates the first The gold standard for key points.

[0070] It should be noted that the specific values ​​of N and M (i.e., the number of anatomical sites and the number of key points) can be adjusted by the number of output channels of the aforementioned fine detection head.

[0071] Taking the long-axis section of the aortic arch in the suprasternal notch as an example, this section is an important section in the standardized application of transthoracic echocardiography. Clinicians often perform quantitative analysis on this section, and when critical values ​​are found, they can provide precise, timely and effective intervention or treatment to ensure the patient's life safety. Figure 4 The diagram shows the anatomical structure of the aortic arch in the suprasternal notch along its long axis. The ascending aorta, the aortic arch, and the upper part of the descending aorta are visible in this section. From left to right, the brachiocephalic artery, the left common carotid artery, and the left subclavian artery, which originate from the aortic arch, are also visible.

[0072] like Figure 5 As shown, to achieve fully automated quantitative analysis of the standard long-axis section of the aortic arch in the suprasternal notch, it is necessary to acquire four anatomical structures: the aorta (AO), the innominate artery (IA), the left common carotid artery (LCCA), and the left subclavian artery (LSA), as well as two key points (midpoint and left subclavian opening point) from the innominate artery to the left common carotid artery. A sample set of 2931 standard long-axis section images of the aortic arch in the suprasternal notch and the gold standard image were constructed, divided in a 4:1 ratio: 2346 images for training and 585 images for testing.

[0073] Please refer to Figure 2 The standard long-axis section of the aortic arch in the suprasternal notch is preprocessed and standardized to a size of 512x512 before being fed into a dual-headed network model. In this model, an encoder extracts and encodes global features from the long-axis section of the aortic arch. A coarse detection module, combined with global features, extracts features from the target area and predicts the overall contour. Finally, a fine detection module, combined with target area features, accurately segments the four anatomical structures (AO, IA, LCCA, and LSA) and detects two key points. The aforementioned adaptive loss function... This set of weights can be initialized to {0.5, 1.0, 0.1}, which facilitates better convergence of the adaptive loss function during training.

[0074] Table 1 below and Figure 6 The results show the comparison between the ultrasound image processing method proposed in this invention and other existing models in the long axis section of the aortic arch in the suprasternal notch, and the results of various anatomical structures and key points. Figure 6 Visualization results of multiple model predictions for the long-axis section of the aortic arch in the suprasternal notch are presented. Figure 6Images (a) and (b) show two sets of comparison results, presented from left to right and top to bottom as follows: input image, gold standard image, UNet prediction result image, UNet++ prediction result image, ConNextV2+ResNet (single-head) prediction result image, and ConNextV2+ResNet (dual-head) prediction result image. ConNextV2+ResNet (dual-head) refers to the dual-head network model provided in this application. In each prediction result image, the red area represents AO, the dark blue area represents IA, the green area represents LCCA, the light blue area represents LSA, the pink dot represents the midpoint between IA and LCCA, and the yellow dot represents the LSA opening point. For each anatomical structure, the accuracy of the prediction model is evaluated using the Dice evaluation model. The output value of the Dice evaluation model is the similarity between the segmentation result of the anatomical structure output by each method and the gold standard; the higher the similarity, the higher the accuracy of the corresponding method. For each keypoint, the accuracy of the keypoint prediction model is evaluated using the Euclidean distance evaluation model. The output value of the Euclidean distance evaluation model is the distance between the keypoint detection result output by each method and the gold standard for the keypoint; the closer the distance, the higher the accuracy of the corresponding method. From Table 1 and Figure 6 It is evident that by using the method proposed in this invention, the accuracy of various anatomical structures and key points in the long-axis section of the aortic arch in the suprasternal notch is significantly improved.

[0075] Table 1 Comparison Results of Multiple Models

[0076] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders.

[0077] like Figure 7 The diagram shown is a structural schematic of an ultrasonic image processing device 700 provided in an embodiment of this application. The ultrasonic image processing device 700 is disposed on an ultrasonic device.

[0078] Specifically, the ultrasound image processing device 700 may include: Image acquisition unit 701 is used to acquire ultrasound images to be processed; Image processing unit 702 is used to input the ultrasound image to be processed into a dual-head network model to obtain coarse segmentation results, fine segmentation results, and / or key point detection results of the ultrasound image to be processed. The dual-head network model includes an encoder, a coarse detection module, and a fine detection module. The dual-head network model is used to input the ultrasound image to be processed into the encoder to obtain first feature images at multiple scales from shallow to deep; input the first feature images at each scale into the coarse detection module to obtain global contour images at each scale and the coarse segmentation results; input the global contour images at each scale into the fine detection module, and obtain the fine segmentation results and / or the key point detection results through a fine detection head with an attention mechanism.

[0079] In some embodiments of this application, the coarse detection module includes a first residual module and a first stitching module; the image processing unit 702 can be specifically used to: input the first feature image at the smallest scale into the first residual module to obtain a global contour image at the previous scale of the smallest scale; at each scale other than the smallest scale, the first feature image and the global contour image at the same scale are fused in channels through the first stitching module, and the fused image is input into the first residual module to obtain the corresponding global contour image at the previous scale.

[0080] In some embodiments of this application, the coarse detection module further includes a coarse detection head and a first upsampling module; the image processing unit 702 may be specifically used to: input the global contour image at the largest scale into the coarse detection head and the first upsampling module for detection and upsampling, and obtain the image coarse segmentation result.

[0081] In some embodiments of this application, the fine detection module includes a second stitching module, an attention mechanism module, a second residual module, a fine detection head, and a second upsampling module; the image processing unit 702 can be specifically used to: take the global contour image of the previous scale at the smallest scale as the fine detection feature image of the previous scale at the smallest scale; at each scale other than the smallest scale, fuse the global contour feature image and the fine detection feature image of the same scale in channels, and after the attention mechanism module performs attention mechanism operations on the fused image in both spatial and channel dimensions, perform residual calculation operations through the second residual module to obtain the corresponding fine detection feature image of the previous scale; input the fine detection feature image of the largest scale into the fine detection head and the second upsampling module for detection and upsampling to obtain the ultrasound image fine segmentation result and / or key point detection result.

[0082] It should be noted that, for the sake of convenience and brevity, the specific working process of the ultrasound image processing device 700 described above can be found in the following reference: Figures 1 to 2The corresponding process of the method will not be described in detail here.

[0083] like Figure 8 The diagram shown is a structural schematic of a training device 800 based on an adaptive loss function provided in an embodiment of this application. The training device 800 based on the adaptive loss function is configured on an electronic device.

[0084] Specifically, the training device 800 based on the adaptive loss function may include: The sample acquisition unit 801 is used to acquire ultrasound images of the sample. The model processing unit 802 is used to input the sample ultrasound image into the model to be trained for training and obtain the prediction result. The prediction result includes the image coarse segmentation result, fine segmentation result and key point detection result of the sample ultrasound image. The loss calculation unit 803 is used to calculate the loss value of the adaptive loss function based on the prediction result and the gold standard corresponding to the sample ultrasound image. The adaptive loss function is a weighted sum of the first sub-function, the second sub-function, and the third sub-function. The first sub-function is associated with the coarse segmentation result of the image, the second sub-function is associated with the fine segmentation result, and the third sub-function is associated with the key point detection result. The iterative training unit 804 is used to perform iterative training based on the loss value, and update the weights of the adaptive loss function in each iteration until a preset number of iterations is completed, and select the optimal iteration result as the dual-head network model.

[0085] In some embodiments of this application, the first sub-function is represented as ;in, This represents the coarse segmentation result of the image. This represents the gold standard for coarse testing.

[0086] In some embodiments of this application, the detailed segmentation results include Detailed segmentation results for each category, ≥1; The second sub-function is expressed as ;in, Indicates the category number. Indicates the first The weights of each category, Indicates the first Loss values ​​for each category, Indicates the first Detailed segmentation results for each category, Indicates the first The gold standard for precision testing in each category.

[0087] In some embodiments of this application, the weight of each category is obtained by normalizing the loss values ​​of all categories.

[0088] In some embodiments of this application, the key point detection results include Key point detection results, ≥1; the third sub-function is expressed as ;in, Indicates the key point number. Indicates the first Key point detection results, Indicates the first The gold standard for key points.

[0089] It should be noted that, for the sake of convenience and brevity, the specific working process of the training device 800 based on the adaptive loss function described above can be found in [reference needed]. Figure 1 The corresponding process of the method will not be described in detail here.

[0090] like Figure 9 The diagram shown is a schematic representation of a device 9 provided in an embodiment of this application. Specifically, device 9 may include: a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90. When device 9 is an electronic device, computer program 92 may refer to a training program based on an adaptive loss function; when device 9 is an ultrasound device, computer program 92 may refer to a training program based on an adaptive loss function and / or an ultrasound image processing program. When processor 90 executes computer program 92, it implements the steps in the various method embodiments described above. Alternatively, when processor 90 executes computer program 92, it implements the functions of each module / unit in the various device embodiments described above.

[0091] The computer program can be divided into one or more modules / units, which are stored in the memory 91 and executed by the processor 90 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the device 9.

[0092] The device 9 may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will understand that... Figure 9 This is merely an example of device 9 and does not constitute a limitation on device 9. It may include more or fewer components than shown, or combine certain components, or different components. For example, device 9 may also include input / output devices, network access devices, buses, etc.

[0093] The processor 90 referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0094] The memory 91 can be an internal storage unit of the device 9, such as a hard disk or memory of the device 9. The memory 91 can also be an external storage device of the device 9, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the device 9. Furthermore, the memory 91 can include both internal and external storage units of the device 9. The memory 91 is used to store the computer program and other programs and data required by the device 9. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0095] It should be noted that, for the sake of convenience and brevity, the structure of the above-mentioned device 9 can also be referred to the specific description of the structure in the method embodiment, which will not be repeated here.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0097] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0098] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for various specific applications, but such implementations should not be considered beyond the scope of this application.

[0099] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0102] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0103] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An ultrasound image processing method, characterized in that, include: Acquire the ultrasound image to be processed; The ultrasound image to be processed is input into a dual-head network model to obtain coarse segmentation results, fine segmentation results, and / or key point detection results. The dual-head network model includes an encoder, a coarse detection module, and a fine detection module. The dual-head network model is used to input the ultrasound image to be processed into the encoder to obtain first feature images at multiple scales from shallow to deep; input the first feature images at each scale into the coarse detection module to obtain global contour images at each scale and the coarse segmentation results; input the global contour images at each scale into the fine detection module, and obtain the fine segmentation results and / or the key point detection results through a fine detection head with an attention mechanism.

2. The ultrasound image processing method as described in claim 1, characterized in that, The coarse detection module includes a first residual module and a first splicing module; The first feature images at each scale are input into the coarse detection module to obtain global contour images at each scale, including: The first feature image at the smallest scale is input into the first residual module to obtain the global contour image at the next smaller scale. At all scales other than the minimum scale, the first feature image and the global contour image at the same scale are fused along the channel by the first stitching module, and the fused image is input into the first residual module to obtain the corresponding global contour image of the previous scale.

3. The ultrasound image processing method as described in claim 2, characterized in that, The coarse detection module also includes a coarse detection head and a first upsampling module; The first feature images at each scale are input into the coarse detection module to obtain the coarse image segmentation result, including: The global contour image at the largest scale is input into the coarse detection head and the first upsampling module for detection and upsampling to obtain the coarse segmentation result of the image.

4. The ultrasound image processing method as described in claim 2, characterized in that, The fine detection module includes a second stitching module, an attention mechanism module, a second residual module, a fine detection head, and a second upsampling module; The step of inputting global contour images at various scales into the fine detection module, and obtaining the fine segmentation results and / or the key point detection results through a fine detection head with an attention mechanism, includes: The global contour image of the previous scale of the smallest scale is used as the fine detection feature image of the previous scale of the smallest scale. At all scales other than the minimum scale, the global contour feature image and the fine detection feature image of the same scale are fused in channels. After the attention mechanism module performs attention mechanism operations on the fused image in both spatial and channel dimensions, the second residual module performs residual calculation operations to obtain the corresponding fine detection feature image of the previous scale. The fine detection feature image at the largest scale is input into the fine detection head and the second upsampling module for detection and upsampling to obtain the fine segmentation result of the ultrasound image and / or the key point detection result.

5. A training method based on an adaptive loss function, characterized in that, The training method for training the dual-head structure network model according to any one of claims 1-3 includes: Acquire ultrasound images of the sample; The sample ultrasound image is input into the training model to obtain the prediction result, which includes the coarse segmentation result, fine segmentation result, and key point detection result of the sample ultrasound image. Based on the prediction results and the gold standard corresponding to the sample ultrasound image, the loss value of the adaptive loss function is calculated. The adaptive loss function is a weighted sum of the first sub-function, the second sub-function, and the third sub-function. The first sub-function is associated with the coarse segmentation result of the image, the second sub-function is associated with the fine segmentation result, and the third sub-function is associated with the key point detection result. Iterative training is performed based on the loss value, and the weights of the adaptive loss function are updated in each iteration until the preset number of iterations is completed. The optimal iteration result is then selected as the dual-head network model.

6. The training method based on an adaptive loss function as described in claim 5, characterized in that, The first sub-function is represented as ; in, This represents the coarse segmentation result of the image. This represents the gold standard for coarse testing.

7. The training method based on an adaptive loss function as described in claim 5, characterized in that, The detailed segmentation results include Detailed segmentation results for each category, ≥1; The second sub-function is represented as ; in, Indicates the category number. Indicates the first The weights of each category, Indicates the first Loss values ​​for each category, Indicates the first Detailed segmentation results for each category, Indicates the first The gold standard for precision testing in each category.

8. The training method based on an adaptive loss function as described in claim 7, characterized in that, The weights for each category are obtained by normalizing the loss values ​​for all categories.

9. The training method based on an adaptive loss function as described in claim 5, characterized in that, The key point detection results include Key point detection results, ≥1; The third sub-function is expressed as: ; in, Indicates the key point number. Indicates the first Key point detection results, Indicates the first The gold standard for key points.

10. An ultrasonic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the ultrasound image processing method as described in any one of claims 1 to 4.

Images