Ultrasound image analysis method and device based on multi-task learning, and storage medium

Abstract

The application discloses a kind of based on multi-task learning's ultrasonic image analysis method, it is related to ultrasonic image analysis technology.Based on multi-task learning's ultrasonic image analysis method in the application, two tasks share coding layer and extract feature information;Coding layer includes spatial path and context path, and feature integration is carried out by feature fusion strategy;Decoding layer is designed according to task requirement respectively, and two independent branches are used to complete respective algorithm task;Finally, efficient transmission of feature information is realized between coding layer and decoding layer by jump connection;In the embodiment of the application, target object can be more accurately extracted from ultrasonic image data by feature point combined with segmentation graph, to improve the accuracy of ultrasonic image analysis.

Description

Technical Field

[0001] This invention relates to the field of ultrasound image analysis technology, and in particular to an ultrasound image analysis method, apparatus, and storage medium based on multi-task learning. Background Technology

[0002] Like optical images, ultrasound images contain a wealth of information. Analyzing ultrasound images to extract useful information is a crucial task of ultrasound image analysis. For example, for an ultrasound image corresponding to a target object, the image data corresponding to the target object can be determined from the ultrasound image after ultrasound image analysis.

[0003] Multi-task learning (MTL) is a machine learning approach that simultaneously processes multiple tasks, as opposed to single-task learning (STL). STL typically optimizes for a single, independent task, while MTL allows multiple tasks to influence and reinforce each other during the learning process. By integrating supervision signals from related tasks, MTL achieves joint optimization of multiple objective functions, significantly improving the model's generalization ability. In some cases, even with only a primary objective, auxiliary tasks can improve the learning performance of the primary task by providing additional information.

[0004] However, practice has shown that existing ultrasound image analysis methods based on multi-task learning are not comprehensive enough in terms of information mining from ultrasound images, and the construction of multi-task learning models is not reasonable enough, resulting in inaccurate ultrasound image analysis. Therefore, it is necessary to propose a new ultrasound image analysis method based on multi-task learning to improve the accuracy of ultrasound image analysis. Summary of the Invention

[0005] This invention provides an ultrasound image analysis method, apparatus, and storage medium based on multi-task learning, which improves the accuracy of ultrasound image analysis.

[0006] To address the aforementioned technical problems, the first aspect of this invention discloses an ultrasound image analysis method based on multi-task learning, the method comprising: Acquire ultrasound image data corresponding to the target object; in the preset multi-task learning model, the spatial path in the shared coding layer is used to generate a high-resolution feature map based on the ultrasound image data through a shallow network; the context path in the shared coding layer of the preset multi-task learning model is used to generate deep semantics based on the ultrasound image data through downsampling. The high-resolution feature map and the deep semantics are fused based on a preset feature fusion strategy to obtain fused features; The fused features are upsampled according to the deconvolution layer and convolution block in the multi-task learning model to obtain a target heatmap, which is used to represent the feature points corresponding to the target object in the ultrasound image data. The fused features are upsampled according to the feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map. The target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object. The image data corresponding to the target object is determined from the ultrasound image data based on the target heatmap and the target segmentation map.

[0007] As an optional implementation, in the first aspect of the present invention, the fusion of the high-resolution feature map and the deep semantics based on a preset feature fusion strategy to obtain fused features includes: After concatenating the high-resolution feature map and the deep semantics, the scale between the features is normalized to balance the features, thus obtaining the connection features; Pooling is performed on the connection features to generate intermediate feature vectors, and matrix multiplication is performed between the intermediate feature vectors and the connection features to obtain a weight vector. Based on the weight vector, the high-resolution feature map and the deep semantics are fused to obtain fused features.

[0008] As an optional implementation, in the first aspect of the present invention, the spatial path in the shared coding layer of the preset multi-task learning model generates a high-resolution feature map based on the ultrasound image data through a shallow network, including: In the preset multi-task learning model, the spatial path in the shared coding layer decomposes the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies through a shallow network. Each frequency subband is convolved and then concatenated with the window attention features of the shared coding layer, and a high-resolution feature map with multi-scale fusion is generated through linear transformation.

[0009] As an optional implementation, in the first aspect of the invention, the step of decomposing the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies includes: The ultrasound image data is filtered along the pixel row direction using a first filter and a second filter to obtain a high-frequency sub-band and a low-frequency sub-band, wherein the filtering frequency corresponding to the first filter is higher than the filtering frequency corresponding to the second filter. The high-frequency sub-band is filtered using a first filter and a second filter to obtain a first frequency sub-band and a second frequency sub-band. The low-frequency sub-band is filtered using a first filter and a second filter to obtain a third frequency sub-band and a fourth frequency sub-band.

[0010] As an optional implementation, in the first aspect of the present invention, the context path in the shared coding layer of the preset multi-task learning model is downsampled to generate deep semantics based on the ultrasound image data, including: The preset multi-task learning model shares the context path in the coding layer and samples the 1 / 8 scale features, 1 / 16 scale features and 1 / 32 scale features corresponding to the ultrasound image data; Global pooling is performed on the 1 / 32 scale features to obtain global features; The global features are combined with the 1 / 8 scale features and the 1 / 16 scale features to generate deep semantics.

[0011] As an optional implementation, in the first aspect of the present invention, the preset multi-task learning model includes an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heatmap. The preset multi-task learning model is obtained through the following steps: The image segmentation task in the initial multi-task learning model is trained using the first ultrasound segmentation training data to obtain the intermediate multi-task learning model. Using the second ultrasonic segmentation training data and training heatmap data, the intermediate multi-task learning model is trained based on the first training strategy or the second training strategy to obtain a trained multi-task learning model. The first training strategy includes: In each training round, the second ultrasound segmentation training data and the training heatmap data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heatmap data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. The weighted and integrated loss function is obtained according to the weights corresponding to the image segmentation task and the feature point localization task, and the training process is fed back based on the loss function. The second training strategy includes: The image segmentation task and the feature point localization task are preset with corresponding probabilities. In each training round, one of the preset probabilities of the image segmentation task and the feature point localization task is selected for training.

[0012] As an optional implementation, in the first aspect of the present invention, the ultrasound image data is ultrasound image data corresponding to the right ventricle of the heart, the target object is myocardium, the target heatmap is used to represent the feature points corresponding to myocardium in the ultrasound image data, and the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the right ventricle of the heart is identified as myocardium.

[0013] A second aspect of the present invention discloses an ultrasound image analysis device based on multi-task learning, the device comprising: The data acquisition module is used to acquire ultrasound image data corresponding to the target object; The multi-task learning module is used to generate a high-resolution feature map based on the ultrasound image data by using the spatial path in the shared coding layer of the preset multi-task learning model through a shallow network; the multi-task learning module is also used to generate deep semantics based on the ultrasound image data by using the context path in the shared coding layer of the preset multi-task learning model through downsampling. The feature fusion module is used to fuse the high-resolution feature map and the deep semantics based on a preset feature fusion strategy to obtain fused features; The first decoding module is used to upsample the fused features according to the deconvolution layer and convolution block in the multi-task learning model to obtain a target heatmap, wherein the target heatmap is used to represent the feature points corresponding to the target object in the ultrasound image data. The second decoding module is used to upsample the fused features according to the feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map. The target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object. The result determination module is used to determine the image data corresponding to the target object from the ultrasound image data based on the target heatmap and the target segmentation map.

[0014] As an optional implementation, in a second aspect of the present invention, the specific operation method by which the feature fusion module fuses the high-resolution feature map and the deep semantics based on a preset feature fusion strategy to obtain fused features includes: After concatenating the high-resolution feature map and the deep semantics, the scale between the features is normalized to balance the features, thus obtaining the connection features; Pooling is performed on the connection features to generate intermediate feature vectors, and matrix multiplication is performed between the intermediate feature vectors and the connection features to obtain a weight vector. Based on the weight vector, the high-resolution feature map and the deep semantics are fused to obtain fused features.

[0015] As an optional implementation, in a second aspect of the invention, the specific operation of the multi-task learning module generating a high-resolution feature map based on the ultrasound image data using the spatial path in the shared coding layer of a preset multi-task learning model through a shallow network includes: In the preset multi-task learning model, the spatial path in the shared coding layer decomposes the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies through a shallow network. Each frequency subband is convolved and then concatenated with the window attention features of the shared coding layer, and a high-resolution feature map with multi-scale fusion is generated through linear transformation.

[0016] As an optional implementation, in a second aspect of the invention, the specific operation method by which the multi-task learning module decomposes the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies includes: The ultrasound image data is filtered along the pixel row direction using a first filter and a second filter to obtain a high-frequency sub-band and a low-frequency sub-band, wherein the filtering frequency corresponding to the first filter is higher than the filtering frequency corresponding to the second filter. The high-frequency sub-band is filtered using a first filter and a second filter to obtain a first frequency sub-band and a second frequency sub-band. The low-frequency sub-band is filtered using a first filter and a second filter to obtain a third frequency sub-band and a fourth frequency sub-band.

[0017] As an optional implementation, in a second aspect of the present invention, the specific operation method for generating deep semantics based on the ultrasound image data by downsampling the context path in the shared coding layer of the preset multi-task learning model includes: The preset multi-task learning model shares the context path in the coding layer and samples the 1 / 8 scale features, 1 / 16 scale features and 1 / 32 scale features corresponding to the ultrasound image data; Global pooling is performed on the 1 / 32 scale features to obtain global features; The global features are combined with the 1 / 8 scale features and the 1 / 16 scale features to generate deep semantics.

[0018] As an optional implementation, in a second aspect of the present invention, the preset multi-task learning model includes an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heatmap. The preset multi-task learning model is obtained through the following steps: The image segmentation task in the initial multi-task learning model is trained using the first ultrasound segmentation training data to obtain the intermediate multi-task learning model. Using the second ultrasonic segmentation training data and training heatmap data, the intermediate multi-task learning model is trained based on the first training strategy or the second training strategy to obtain a trained multi-task learning model. The first training strategy includes: In each training round, the second ultrasound segmentation training data and the training heatmap data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heatmap data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. The weighted and integrated loss function is obtained according to the weights corresponding to the image segmentation task and the feature point localization task, and the training process is fed back based on the loss function. The second training strategy includes: The image segmentation task and the feature point localization task are preset with corresponding probabilities. In each training round, one of the preset probabilities of the image segmentation task and the feature point localization task is selected for training.

[0019] A third aspect of this invention discloses an ultrasound image analysis system based on multi-task learning, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the ultrasound image analysis method based on multi-task learning disclosed in the first aspect of the present invention.

[0020] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the ultrasound image analysis method based on multi-task learning disclosed in the first aspect of the present invention.

[0021] Compared with existing technologies, the ultrasound image analysis method based on multi-task learning in this invention shares an encoding layer to extract feature information for two tasks. The encoding layer includes spatial paths and contextual paths, and integrates features through a feature fusion strategy. The decoding layer is designed according to the task requirements, and uses two independent branches to complete their respective algorithm tasks. Finally, the encoding layer and the decoding layer achieve efficient transfer of feature information through skip connections. The embodiments of this invention, by combining feature points with segmentation maps, can more accurately extract target objects from ultrasound image data, thereby improving the accuracy of ultrasound image analysis. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating an ultrasound image analysis method based on multi-task learning disclosed in an embodiment of the present invention. Figure 2 This is a network architecture diagram of the multi-task model disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the original image (left), myocardial segmentation data (middle), and feature point localization data (right) disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an ultrasound image analysis system based on multi-task learning disclosed in an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0026] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] Example 1 Please see Figure 1, Figure 1 This is a flowchart illustrating an ultrasound image analysis method based on multi-task learning disclosed in an embodiment of the present invention. Wherein, Figure 1 The described multi-task learning-based ultrasound image analysis method can be implemented in a multi-task learning-based ultrasound image analysis device, which can be integrated into a cloud server or a local server. For example... Figure 1 As shown, this ultrasound image analysis method based on multi-task learning can include the following operations: Step 101: Obtain the ultrasound image data corresponding to the target object.

[0028] In this embodiment of the invention, the target object is the object requiring ultrasound analysis, which can be a human organ, such as the heart, or any object requiring ultrasound analysis, such as an industrial device, chemical medium, or mechanical structure. The ultrasound image data can be ultrasound video data containing multiple frames of ultrasound images, a single ultrasound image, or a collection of multiple ultrasound images used to identify the same object. For example, the ultrasound image data can be ultrasound video data of the right ventricle of the heart. In this case, the image segmentation objective of this embodiment is to accurately segment the myocardium image from the right ventricle of the heart.

[0029] Step 102: In the preset multi-task learning model, the spatial path in the shared coding layer is used to generate a high-resolution feature map based on the ultrasound image data through a shallow network; the context path in the shared coding layer of the preset multi-task learning model is used to generate deep semantics based on the ultrasound image data through downsampling.

[0030] In this embodiment of the invention, specifically, the spatial path generates high-resolution feature maps through a shallow network, preserving rich spatial information and supporting the decoding process of the feature point localization subtask. The high-resolution feature maps in the spatial path facilitate the decoding of the feature point localization subtask, avoiding information loss due to excessively deep network layers and multiple attention calculations, thus aiding in information extraction during the decoding process. The context path combines multi-stage features with a downsampling strategy to generate feature representations with the maximum receptive field and rich contextual information.

[0031] In this embodiment of the invention, both feature extraction methods are implemented based on a shared encoding layer in a pre-defined multi-task learning model, thereby achieving hard parameter sharing. Specifically, by sharing the parameters in the encoding layer, the image segmentation task provides the feature point localization task with more regional information; secondly, the design of the two sub-tasks sharing the encoder has a significant advantage in the overall number of parameters in the multi-task model, helping to reduce the total number of parameters and also reducing the total inference time of the two sub-tasks.

[0032] Step 103: Based on the preset feature fusion strategy, the high-resolution feature map and deep semantics are fused to obtain fused features.

[0033] In this embodiment of the invention, the feature representations of spatial paths and context paths have different levels and focuses. The spatial information obtained from the spatial path encodes the vast majority of detailed information, and the output feature information is relatively low-level and microscopic. In contrast, the context information obtained from the context path encodes rich semantics and global information, and the output feature information is relatively high-level and macroscopic. Based on a preset feature fusion strategy, high-resolution feature maps and deep semantics are fused to obtain fused features, thereby providing rich and efficient data for subsequent model processing.

[0034] Step 104: Upsample the fused features based on the deconvolutional layers and convolutional blocks in the multi-task learning model to obtain the target heatmap.

[0035] In this embodiment of the invention, the target heatmap is used to represent the feature points corresponding to the target object in the ultrasound image data.

[0036] This invention constructs a multi-task learning framework where the decoder is designed as an independent branch for two different sub-tasks, satisfying the different needs of the two sub-tasks and improving the model's performance and efficiency. This architectural choice is based on the differences in feature extraction and processing between the sub-tasks. Specifically, for the feature point localization task, due to its high demand for local feature recognition, the model employs deconvolutional layers combined with continuous convolutional blocks for upsampling to achieve effective feature decoding and detail capture. This design reduces computational load while enhancing detail recognition capabilities through the advantages of convolutional layers. Optionally, each convolutional block includes a convolutional layer, batch normalization, and a ReLU activation function to improve the model's non-linear expressive power.

[0037] Step 105: Upsample the fused features according to the feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain the target segmentation map.

[0038] In this embodiment of the invention, the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object. For image segmentation tasks, given the need to recognize the overall structure, this embodiment employs a feature extraction module corresponding to the shared coding layer as the core of the decoder. This structure enables it to capture long-distance dependencies, which is crucial for understanding the relative positional relationships of target objects, thereby improving the accuracy of image segmentation.

[0039] If two tasks share the same decoding branch, although they may overlap in data features, the segmentation involves a large area while the target area for feature point localization is relatively small. This makes it difficult for the network to effectively process two very different targets simultaneously, potentially leading to difficulties in training and convergence issues. Therefore, the design of independent decoding branches in this embodiment helps to update network parameters more accurately during backpropagation.

[0040] Step 106: Determine the image data corresponding to the target object from the ultrasound image data based on the target heat map and target segmentation map.

[0041] In this embodiment of the invention, the target heatmap is used to represent the feature points corresponding to the target object in the ultrasound image data; the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object. The feature points combined with the segmentation map can more accurately extract the target object from the ultrasound image data for subsequent processing.

[0042] As can be seen, in the ultrasound image analysis method based on multi-task learning in this embodiment of the invention, two tasks share the encoding layer to extract feature information; secondly, the encoding layer includes spatial path and context path, and integrates features through a feature fusion strategy; thirdly, the decoding layer is designed according to the task requirements, and uses two independent branches to complete their respective algorithm tasks; finally, the encoding layer and the decoding layer achieve efficient transfer of feature information through skip connections; this embodiment of the invention, by combining feature points with segmentation maps, can more accurately extract the target object from ultrasound image data, thereby improving the accuracy of ultrasound image analysis.

[0043] In an optional embodiment, high-resolution feature maps and deep semantics are fused based on a preset feature fusion strategy to obtain fused features, including: After concatenating high-resolution feature maps and deep semantics, the scale between features is balanced by normalization to obtain the connectivity features; Pooling is performed on the connection features to generate intermediate feature vectors. The intermediate feature vectors are then multiplied by the connection features to obtain the weight vectors. Based on the weight vector, the high-resolution feature map and deep semantics are fused to obtain the fused features.

[0044] In this optional embodiment, the effective combination of features from two paths is based on a preset feature fusion strategy. Since the feature representations of the spatial path and the context path have different levels and emphases, simple weighted fusion is ineffective; therefore, a targeted fusion strategy can be designed. To better fuse the two types of features, an attention refinement module (ARM) can be added to the context path.

[0045] ARM is used in the context path to optimize the outputs of the first two stages. ARM obtains global semantic information through global average pooling, and then performs matrix multiplication with the original output features using convolution, regularization, and sigmoid to calculate a feature vector to guide feature learning. ARM optimizes the results in the context path and integrates global features without upsampling, resulting in low overhead and negligible computational cost.

[0046] In the feature fusion strategy, the outputs of the two paths are first concatenated, and then the scale between features is balanced through normalization. A feature vector is generated by pooling the connected features, and a weight vector is obtained by matrix multiplication with the original features. This weight vector is then reweighted to complete the feature selection and fusion process. Ultimately, the feature fusion strategy combines high-level global information and low-level detailed features from the encoder, preserving rich original feature information and providing ample support for accurate decoding based on subtask requirements.

[0047] Furthermore, the increase in overall parameter count and computational cost caused by the aforementioned feature fusion strategy is relatively small. Experiments in this optional embodiment have verified that introducing the feature fusion strategy only adds 515.21K parameters and 16.47M FLOPs, a negligible increase compared to the overall parameter count and computational cost of the multi-task framework model.

[0048] In another optional embodiment, for ultrasound data, the low-frequency components often contain the basic features of the signal, while the high-frequency signals provide detailed information, such as the edge contours of image information. Therefore, to further mine the information in ultrasound data, the spatial path in the shared coding layer of the aforementioned preset multi-task learning model generates a high-resolution feature map based on the ultrasound image data through a shallow network, which may include: In the pre-defined multi-task learning model, the spatial path in the shared coding layer decomposes the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies through a shallow network. Each frequency subband is convolved and then concatenated with the window attention features of the shared coding layer. A high-resolution feature map with multi-scale fusion is generated through linear transformation.

[0049] In this optional embodiment, the spatial path generates a high-resolution feature map through a shallow network, preserving rich spatial information and supporting the decoding process of the feature point localization subtask. Optionally, the spatial path is the feature information of the first stage in the encoder network. At this time, the original tensor of the ultrasound image is segmented into multiple regions and adjusted to a 1 / 4 resolution feature vector with C channels through vector embedding. This feature vector extracts a large-scale feature map, preserving rich spatial information. The high-resolution feature map in the spatial path facilitates the decoding of the feature point localization subtask, avoiding information loss due to excessively deep network layers and multiple attention calculations, and helping the feature point subtask extract information during the decoding process.

[0050] In this optional embodiment, further optionally, decomposing the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies may include: The ultrasound image data is filtered along the pixel row direction using a first filter and a second filter to obtain a high-frequency sub-band and a low-frequency sub-band. The filtering frequency corresponding to the first filter is higher than that corresponding to the second filter. The high-frequency subband is filtered using the first and second filters to obtain the first and second frequency subbands. The low-frequency subband is filtered using the first and second filters to obtain the third and fourth frequency subbands.

[0051] Specifically, this optional embodiment proposes a method that combines frequency sub-bands of different frequencies with a window attention mechanism to effectively provide multi-scale information. Specifically, the input feature map... The image is decomposed into four wavelet sub-bands. First, it is filtered by low-pass and high-pass filters. The low-pass filter outputs the low-frequency sub-band, representing the basic information of the image; the high-pass filter outputs the high-frequency sub-band, representing the detailed information. For example, for the input feature map... Apply low-pass and high-pass filters along the path, first... Encoded as two sub-bands and Then, by continuing along these two sub-bands using the same low-pass and high-pass filters, all four sub-bands can be obtained. , , and ( ), Representing low-frequency components, it can preserve characteristics. Basic information, , and Represents high-frequency components and can preserve features The detailed features are captured by these four sub-bands, which cover every detail of the input without causing any loss of feature information. Then, these four wavelet sub-bands are connected along the channel dimension for further processing. Convolution applies spatial locality to obtain Then, The final attention result is obtained by concatenating the output of the image segmentation model's output layer with the attention result. Finally, it goes through a linear transformation layer to obtain .

[0052] This alternative embodiment allows the model to learn more information at different scales by stacking the four resulting subbands together and fusing them with window attention features. This not only enables the model to focus on attention information within the window but also to learn image features at multiple scales, including basic and detailed information.

[0053] In another optional embodiment, the context path in the shared coding layer of the preset multi-task learning model is downsampled to generate deep semantics based on ultrasound image data, which may include: The pre-defined multi-task learning model shares the context path in the coding layer and samples the 1 / 8 scale features, 1 / 16 scale features and 1 / 32 scale features corresponding to ultrasound image data. Global pooling is performed on 1 / 32 scale features to obtain global features; The global features are combined with 1 / 8 scale features and 1 / 16 scale features to generate deep semantics.

[0054] In this optional embodiment, the context path combines multi-stage features using a downsampling strategy to generate a feature representation with the maximum receptive field and rich contextual information. Optionally, the context path combines feature information from two stages, at scales of 1 / 8 and 1 / 16 respectively. Furthermore, at the final stage of the encoding layer, at scale 1 / 32, global average pooling is used to obtain global contextual information with the maximum receptive field. Finally, the global features are combined with the features from the first two stages to form a context path rich in feature information.

[0055] In another optional embodiment, the preset multi-task learning model includes an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heatmap.

[0056] The multi-task framework model designed in this embodiment of the invention has the following characteristics: First, the two tasks share an encoding layer; second, the encoding layer includes spatial paths and context paths, and integrates features through a feature fusion strategy; third, the decoding layer is designed separately according to the task requirements, using two independent branches to complete their respective algorithm tasks; finally, the encoding and decoding layers achieve efficient transfer of feature information through skip connections. These characteristics reflect the complexity of the multi-task model structure, making the training process more difficult. Therefore, it is necessary to formulate optimization strategies that can effectively improve training results. The above-mentioned preset multi-task learning model is obtained through the following steps: The image segmentation task in the initial multi-task learning model is trained using the first ultrasound segmentation training data to obtain the intermediate multi-task learning model. Using the second ultrasonic segmentation training data and training heatmap data, the intermediate multi-task learning model is trained based on the first training strategy or the second training strategy to obtain the trained multi-task learning model. The first training strategy includes: In each training round, the second ultrasound segmentation training data and the training heatmap data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heatmap data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. The weighted and integrated loss function is obtained according to the weights corresponding to the image segmentation task and the feature point localization task, and the training process is fed back based on the loss function. The second training strategy includes: Preset corresponding probabilities for image segmentation task and feature point localization task. In each training round, select one of the preset probabilities for image segmentation task and feature point localization task for training.

[0057] This optional embodiment suggests that the design should be based on the nature of the two sub-tasks and the characteristics of the dataset. The training process of the entire multi-task framework is divided into two stages. The first stage uses the first ultrasound segmentation training data (which does not include feature points) for training the segmentation task. The second stage incorporates the feature point localization task, and both branches are trained simultaneously. The idea behind this design is that when the entire multi-task framework is trained directly from the beginning, i.e., when training both task branches simultaneously, the model tends to focus on segmentation, performing poorly in feature point localization. Furthermore, the network exhibits significant oscillations and slow convergence in the early stages of training. The reasons for this optional embodiment are analyzed as follows: (1) In a multi-task learning framework, the feature point localization task is more challenging than the segmentation task. The targets in the feature point localization task are relatively small, requiring the capture of fine feature point locations. Identifying these fine target points in an image is more difficult than identifying target objects in larger areas, especially when the image quality is poor. Although the data distributions of the two tasks are similar, multi-task networks may tend to learn more salient features, i.e., the segmentation task, in the early stages of training, which may lead to the relative neglect of the feature point localization task. Furthermore, in single-task feature point localization learning, network training for the feature point localization task usually converges more slowly because the accurate localization of these small targets requires the network to learn more refined feature representations.

[0058] (2) In the early stage of network training, the encoding layer and the two task branches have the following two problems: 1) The weights of the network are in an unstable state; 2) The loss signal may deviate from the target point during the backpropagation process of the feature point localization task, resulting in instability and non-convergence during the training process. Therefore, directly adding the feature point task for training in the early stage of network training does not make good use of the similarity between the segmentation task and the feature point task to play a mutually reinforcing role.

[0059] In this optional embodiment, two training strategies are proposed for training the two branches simultaneously in the second stage: 1) overall training, which calculates a unified multi-task loss function by controlling the weights to train the networks of the two branches simultaneously; 2) selection according to probability, which randomly selects one branch for training in each round.

[0060] The core of the first training strategy is to load datasets for both tasks simultaneously in each training round and calculate their gradients separately. Then, different weights are assigned to the two sub-tasks, and their loss functions are weighted and integrated to generate a comprehensive overall loss function. This loss function optimizes the network parameters through backpropagation, thereby achieving joint training of feature point localization and segmentation tasks. This method, based on a shared model structure, coordinates the relationship between the two tasks, ensuring optimized model performance.

[0061] The second training strategy employs a probability-based selection of task branches for training, meaning only one subtask is trained per round. Since the segmentation task involves a large amount of data, and its data distribution covers the distribution area of ​​the target object's feature points, this alternative implementation will select the segmentation task branch for training with a higher probability. The gradients back from the segmentation task can enhance the generalization ability of the encoding layer in extracting features, while also providing prior information for region localization in the feature point localization task, thereby improving the training effect of the feature point localization task.

[0062] It should be noted that, in this embodiment of the invention, the ultrasound image data is the ultrasound image data corresponding to the right ventricle of the heart, the target object is the myocardium, the target heatmap is used to represent the feature points corresponding to the myocardium in the ultrasound image data, and the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the right ventricle of the heart is identified as myocardium.

[0063] Example 2 In Embodiment 1 of this invention, an ultrasound image analysis method based on multi-task learning is proposed. When the ultrasound image data is the ultrasound image data corresponding to the right ventricle of the heart, the target object is the myocardium, the target heatmap is used to represent the feature points corresponding to the myocardium in the ultrasound image data, and the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the right ventricle of the heart is identified as myocardium, Embodiment 2 of this invention proposes a feature point localization-assisted myocardial segmentation algorithm based on multi-task learning.

[0064] Specifically, this invention presents a multi-task learning framework capable of performing right ventricular myocardial segmentation and predicting the coordinates of right ventricular anatomical feature points from input apical four-chamber echocardiography images, such as... Figure 2 As shown, the multi-task framework of this invention integrates the right ventricular myocardial segmentation task and the right ventricular anatomical feature point task.

[0065] Combination Figure 2 The multi-task network framework proposed in this embodiment of the invention has the following three characteristics: (1) The idea of ​​hard parameter sharing is adopted, and the same coding block is used for feature extraction in the coding layer. This design is mainly based on two considerations: First, the right ventricular myocardial segmentation algorithm can be trained using the complete cardiac cycle of ultrasound images, but the right ventricular feature point localization algorithm is trained using only a few frames of ultrasound images of the cardiac cycle. Therefore, the right ventricular myocardial segmentation task can learn more feature information. By sharing the parameters in the coding layer, the right ventricular myocardial segmentation task provides the right ventricular feature point localization task with more myocardial region information; Second, the design of the two sub-tasks sharing the encoder forms a hard parameter sharing mechanism, which has a significant advantage in the total number of parameters in the multi-task model, helps to reduce the total number of parameters, and can also reduce the total inference time of the two sub-tasks.

[0066] (2) Based on the characteristics of both tasks, a multi-path feature fusion module was designed. In this embodiment of the invention, the encoder is divided into a spatial path and a context path, and information is fused through an attention fusion module. The spatial path retains high-resolution image information, while the context path can obtain deeper network information through downsampling, thereby obtaining rich receptive fields at different scales. Furthermore, a feature fusion module is designed to fuse the features of the two parts.

[0067] (3) The two tasks use different decoder branches based on their task characteristics. In the feature point task, convolutional blocks and deconvolution are used for decoding, and finally the heatmap of the same resolution is restored for output. In the myocardial segmentation task, the decoding blocks corresponding to the encoding layer are used symmetrically for decoding, and the predicted mask for myocardial segmentation is output.

[0068] In this embodiment of the invention, specifically, the spatial path and context path can both retain key spatial information and obtain a sufficiently large receptive field. Simultaneously, with the help of the feature fusion module, feature information from different levels is effectively integrated. For the multi-task proposed in this embodiment of the invention, it is necessary to capture detailed features while obtaining high-resolution features. This advantage applies to the two sub-tasks of this embodiment of the invention: the myocardial segmentation task and the specific point localization task.

[0069] The spatial path generates high-resolution feature maps through a shallow network, preserving rich spatial information and supporting the decoding process of the feature point localization subtask. Specifically, the spatial path represents the feature information of the first stage in the encoder network. At this stage, the original ultrasound image tensor is segmented into multiple regions and adjusted to a 1 / 4 resolution feature vector with C channels through vector embedding. This feature vector extracts a larger-scale feature map through two encoding blocks, preserving rich spatial information. The high-resolution feature map in the spatial path facilitates the decoding of the feature point localization subtask, avoiding information loss due to excessively deep network layers and multiple attention calculations, thus aiding in information extraction during the decoding process of the feature point subtask.

[0070] The context path combines multi-stage features using a downsampling strategy to generate feature representations with the largest receptive field and rich contextual information. The context path combines feature information from two stages, at scales of 1 / 8 and 1 / 16 respectively. Finally, at the end of the encoding layer, the features at scale 1 / 32 are used to obtain global contextual information with the largest receptive field through global average pooling. Finally, the global features are combined with the features from the first two stages to form a context path rich in feature information.

[0071] Ultimately, an effective combination of features from the two paths is achieved through a feature fusion module. Since the feature representations of the spatial path and the context path have different levels and focuses, simple weighting is insufficient for effective fusion; therefore, a specialized fusion strategy is required. The spatial information obtained from the spatial path encodes the vast majority of detailed information, resulting in low-level, microscopic feature information. In contrast, the context information obtained from the context path encodes rich semantic and global information, resulting in high-level, macroscopic feature information. To better fuse the two types of features, a feature fusion module is used, and an attention optimization module is added to the context path.

[0072] The attention optimization module is used in the context path to optimize the outputs of the first two stages. It obtains global semantic information through global average pooling, then performs convolution, regularization, and activation functions, followed by matrix multiplication with the original output features to calculate a feature vector to guide feature learning. This attention optimization module effectively optimizes the results in the context path and integrates global features without upsampling, resulting in low overhead and negligible computational cost.

[0073] The feature fusion module is primarily used to effectively fuse two types of features. First, it concatenates the outputs of the two paths, then normalizes and balances the scale between the features. Within the feature fusion module, pooling is performed on the connected features to generate feature vectors, which are then multiplied with the original features to obtain a weight vector. This weight vector is then reweighted to complete the feature selection and fusion process. Ultimately, the feature fusion module combines high-level global information with low-level detailed features from the encoder, preserving rich original feature information and providing ample support for accurate decoding based on subtask requirements.

[0074] Furthermore, the increase in overall parameter count and computational cost after adding the multi-path feature fusion module and attention optimization module is relatively small. In this embodiment of the invention, the introduction of these modules only adds 515.21K parameters and 16.47M FLOPs, which is negligible compared to the overall parameter count and computational cost of the multi-task framework model. In one alternative embodiment, a multi-task learning framework is used, where the decoder is designed as an independent branch for two different sub-tasks, meeting the different needs of the two sub-tasks and improving the model's performance and efficiency. This architectural choice is based on the differences in feature extraction and processing between the sub-tasks.

[0075] For feature point localization tasks, which require high recognition of local features, the model employs deconvolutional layers combined with consecutive convolutional blocks for upsampling to achieve efficient feature decoding and detail capture. This design reduces computational load while enhancing detail recognition capabilities through the advantages of convolutional layers. Specifically, each convolutional block contains a convolutional layer, batch normalization, and a ReLU activation function to improve the model's non-linear expressive power.

[0076] For the myocardial segmentation task, given its requirement for recognizing the overall structure, the model uses a feature extraction module corresponding to the shared coding layer as the core of the decoder.

[0077] Furthermore, the design of independent decoding branches helps to update network parameters more accurately during backpropagation. If two tasks share the same decoding branch, although they have some overlap in data features, the large area involved in myocardial segmentation and the relatively small target area for feature point localization make it difficult for the network to effectively handle two very different targets simultaneously, which may lead to difficulties in training and convergence problems.

[0078] In the multi-task framework, this embodiment of the invention implements a hard parameter sharing strategy, where different sub-tasks share the encoder parameters while maintaining an independent parameter set for each task. This strategy enables the model to capture general features during the encoding phase and perform detailed learning for specific tasks during the decoding phase, thereby improving the overall efficiency and accuracy of the model. The hard sharing strategy is suitable for handling highly correlated tasks, and due to parameter sharing, it reduces model complexity and the risk of overfitting.

[0079] In yet another optional embodiment, the model training process is as follows: First, starting from the dataset used by the multi-task framework model, this embodiment of the invention proposes that, in addition to training the model using the dataset for the right ventricular feature point localization task, a dataset for right ventricular myocardial segmentation training should be added. This training strategy is because both tasks can use the feature point localization dataset independently, but the feature point localization model performs poorly with limited data. This is because the segmented region of the right ventricular myocardium coincides with the region of maximum distribution of right ventricular feature points, such as... Figure 3 As shown, information about segmented regions can be learned using a myocardial segmentation dataset. However, in a multi-task framework, using only a dataset for feature point localization for both tasks leads to unsatisfactory results for both tasks. This is because the sample size for the myocardial segmentation task is relatively small, resulting in decreased performance on the test set. Therefore, it is necessary to propose using an incremental dataset for training. Using an incremental dataset allows the feature point localization branch in the multi-task framework to obtain more regional distribution information from the larger myocardial segmentation dataset.

[0080] Secondly, considering the entire network training process, this embodiment of the invention proposes that the training process of the entire multi-task framework should be designed according to the nature of the two sub-tasks and the characteristics of the dataset. The training process is divided into two stages. In the first stage, the myocardial segmentation task is trained using a myocardial segmentation dataset that does not contain feature point datasets. In the second stage, the feature point localization task is added, and the two branches are trained together. The idea behind this design is that when the entire multi-task framework is trained directly at the beginning, i.e., when training the two task branches simultaneously, the model tends to learn right ventricular myocardial segmentation, performs poorly in right ventricular feature point localization, and exhibits significant network oscillation and slow convergence speed in the early stages of training. This embodiment of the invention analyzes the reasons for this as follows: (1) In the multi-task learning framework, the feature point localization task is more challenging than the myocardial segmentation task. The targets in the feature point localization task are relatively small, requiring the capture of fine feature point locations. Identifying these fine target points in an image is more difficult than identifying larger areas of myocardium, especially when the image quality is poor. Although the data distributions of the two tasks are similar, multi-task networks may tend to learn more salient features, i.e., the myocardial segmentation task, in the early stages of training, which may lead to the relative neglect of the feature point localization task. Furthermore, in single-task feature point localization learning, network training for the feature point localization task usually converges more slowly because the accurate localization of these small targets requires the network to learn more refined feature representations.

[0081] (2) In the early stage of network training, the encoding layer and the two task branches have the following two problems: 1) The weights of the network are in an unstable state; 2) The loss signal may deviate from the target point during the backpropagation process of the feature point localization task, resulting in instability and non-convergence during the training process. Therefore, directly adding the feature point task for training in the early stage of network training does not make good use of the similarity between the myocardial segmentation task and the feature point task to play a mutually reinforcing role.

[0082] Finally, regarding the training strategy for training two branches simultaneously in the second stage, this embodiment of the invention proposes two training strategies: 1) overall training, which calculates a unified multi-task loss function by controlling the weights to train the networks of two branches simultaneously; 2) selection according to probability, which randomly selects one branch for training in each round.

[0083] The core of the first training strategy is to load datasets for both tasks simultaneously in each training round and calculate their gradients separately. Then, different weights are assigned to the two sub-tasks, and their loss functions are weighted and integrated to generate a comprehensive overall loss function. This loss function optimizes the network parameters through backpropagation, thereby achieving joint training of feature point localization and myocardial segmentation tasks. This method, while sharing the model structure, coordinates the relationship between the two tasks, ensuring optimized model performance.

[0084] The second training strategy employs a probability-based selection of task branches for training, meaning only one subtask is trained per round. Since the right ventricular myocardial segmentation task involves a large dataset, and its data distribution covers the distribution area of ​​right ventricular feature points, this embodiment of the invention will select the right ventricular myocardial segmentation task branch for training with a high probability. The gradients back from the right ventricular myocardial segmentation task can enhance the generalization ability of the encoding layer in extracting features, while also providing prior information for the right ventricular feature point localization task, thereby improving the training effect of the right ventricular feature point localization task.

[0085] In yet another alternative embodiment, the loss function is designed as follows: Since the direction and magnitude of the training gradients differ across tasks, multi-task learning requires consideration of the conflicts and balance issues between the loss functions of each task during training. If the gradient magnitudes are inconsistent, the convergence speeds of different tasks will be inconsistent; if the directions are inconsistent, conflicts will arise, and gradient cancellation may even occur due to inconsistent optimization directions, thereby affecting the training speed and quality of the model.

[0086] Currently, loss function design strategies for multi-task learning mainly focus on three aspects: magnitude, speed, and gradient. Among these, the total loss function in multi-task learning... As shown in the following formula:

[0087] in, These are the weighting coefficients. Let be the loss function for the i-th task. PcGrad starts from the direction of the gradient, considers the angle between gradient vectors, and projects them to avoid gradient pulling between different tasks. MGDA is a multi-objective optimization problem that believes that multi-objective optimization problems can use gradient descent to obtain local optima and ultimately achieve Pareto optimality. HTW is suitable for scenarios where there are main and auxiliary tasks in a multi-task environment. It learns more information from the auxiliary tasks in the early stages of training, and then focuses on learning the main task, where the auxiliary tasks are not important.

[0088] However, the multi-task framework designed in this embodiment of the invention is not applicable to all optimization strategies. For example, the gradient backpropagation strategies of GradNorm and MGDA are not suitable for the skip connection structure in the multi-task learning framework of this embodiment. The HTW strategy relies on the mechanism of first completing the learning of the auxiliary task and then focusing on learning the main task, but the two tasks in this embodiment need to produce effective outputs simultaneously, so it is not applicable. In addition, although the uncertainty weight allocation method proposed by Kendall et al. can dynamically adjust the task weights, it requires explicit mathematical derivation of the loss function for each task. However, the loss function design for the two sub-tasks in this embodiment is relatively complex and cannot provide a concise and clear expression, so this method is difficult to apply effectively in the multi-task learning framework of this embodiment.

[0089] Ultimately, this embodiment of the invention selects to adjust the weights of the loss function for different tasks based on the task difficulty and gradient magnitude. The final constructed loss function is as follows, as shown in the following equation, where... Indicates the first step in the training process. The first round The weights of the loss function for each task.

[0090]

[0091] Of particular interest is the magnitude of gradient descent during training, i.e., the speed at which the network learns. A task can be defined. In the The learning speed of the wheel is shown in the following formula:

[0092] when The smaller the value, the faster the loss decreases, and the faster the network learns. Therefore, the weight of this task needs to be set lower. Specifically, for... After normalization, the weight coefficients for different tasks are obtained, as shown in the following formula, where This is a temperature coefficient used to adjust the weighting differences between different tasks. The larger the value, the more evenly the weight distribution. When the size is large enough, the weight of each task approaches 1.

[0093]

[0094] In another strategy, KPIs are chosen as the standard for measuring weights. First, KPIs need to be selected for the feature point task and the myocardial segmentation task. In this embodiment of the invention, the KPI for the myocardial segmentation task is defined as the Dice similarity coefficient, and the KPI for the feature point localization task is the detection rate. It can be shown in the following formula, where Indicates task In the The KPIs of the round, and required in DTP. .

[0095]

[0096] Example 3 This invention discloses an ultrasound image analysis device based on multi-task learning, which may include: The data acquisition module is used to acquire ultrasound image data corresponding to the target object; The multi-task learning module is used to generate high-resolution feature maps from ultrasound image data by using the spatial paths in the shared coding layer of the preset multi-task learning model through a shallow network; the multi-task learning module is also used to generate deep semantics from ultrasound image data by using the context paths in the shared coding layer of the preset multi-task learning model through downsampling. The feature fusion module is used to fuse high-resolution feature maps and deep semantics based on a preset feature fusion strategy to obtain fused features; The first decoding module is used to upsample the fused features based on the deconvolution layer and convolution block in the multi-task learning model to obtain the target heatmap, which is used to represent the feature points corresponding to the target object in the ultrasound image data. The second decoding module is used to upsample the fused features according to the feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain the target segmentation map. The target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object. The result determination module is used to determine the image data corresponding to the target object from the ultrasound image data based on the target heatmap and target segmentation map.

[0097] In an optional embodiment, the feature fusion module fuses high-resolution feature maps and deep semantics based on a preset feature fusion strategy to obtain fused features. The specific operation may include: After concatenating high-resolution feature maps and deep semantics, the scale between features is balanced by normalization to obtain the connectivity features; Pooling is performed on the connection features to generate intermediate feature vectors. The intermediate feature vectors are then multiplied by the connection features to obtain the weight vectors. Based on the weight vector, the high-resolution feature map and deep semantics are fused to obtain the fused features.

[0098] In another optional embodiment, the specific operation of the multi-task learning module in generating high-resolution feature maps from ultrasound image data using the spatial path in the shared coding layer of a preset multi-task learning model through a shallow network may include: In the pre-defined multi-task learning model, the spatial path in the shared coding layer decomposes the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies through a shallow network. Each frequency subband is convolved and then concatenated with the window attention features of the shared coding layer. A high-resolution feature map with multi-scale fusion is generated through linear transformation.

[0099] In yet another optional embodiment, the specific operation method by which the multi-task learning module decomposes ultrasound image data into multiple frequency sub-bands corresponding to different frequencies may include: The ultrasound image data is filtered along the pixel row direction using a first filter and a second filter to obtain a high-frequency sub-band and a low-frequency sub-band. The filtering frequency corresponding to the first filter is higher than that corresponding to the second filter. The high-frequency subband is filtered using the first and second filters to obtain the first and second frequency subbands. The low-frequency subband is filtered using the first and second filters to obtain the third and fourth frequency subbands.

[0100] In another optional embodiment, the specific operation of generating deep semantics based on ultrasound image data by downsampling the context path in the shared coding layer of the preset multi-task learning model may include: The pre-defined multi-task learning model shares the context path in the coding layer and samples the 1 / 8 scale features, 1 / 16 scale features and 1 / 32 scale features corresponding to ultrasound image data. Global pooling is performed on 1 / 32 scale features to obtain global features; The global features are combined with 1 / 8 scale features and 1 / 16 scale features to generate deep semantics.

[0101] In another optional embodiment, the preset multi-task learning model may include an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heatmap. The pre-defined multi-task learning model is obtained through the following steps: The image segmentation task in the initial multi-task learning model is trained using the first ultrasound segmentation training data to obtain the intermediate multi-task learning model. Using the second ultrasonic segmentation training data and training heatmap data, the intermediate multi-task learning model is trained based on the first training strategy or the second training strategy to obtain the trained multi-task learning model. The first training strategy may include: In each training round, the second ultrasound segmentation training data and the training heatmap data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heatmap data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. The weighted and integrated loss function is obtained according to the weights corresponding to the image segmentation task and the feature point localization task, and the training process is fed back based on the loss function. The second training strategy may include: Preset corresponding probabilities for image segmentation task and feature point localization task. In each training round, select one of the preset probabilities for image segmentation task and feature point localization task for training.

[0102] In another optional embodiment, the ultrasound image data is the ultrasound image data corresponding to the right ventricle of the heart, the target object is the myocardium, the target heatmap is used to represent the feature points corresponding to the myocardium in the ultrasound image data, and the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the right ventricle of the heart is identified as myocardium.

[0103] Example 4 Please see Figure 4 , Figure 4This is a schematic diagram of the structure of an ultrasound image analysis system based on multi-task learning, as disclosed in an embodiment of the present invention. Figure 4 As shown, the ultrasound image analysis system based on multi-task learning may include: Memory 201 storing executable program code; Processor 202 coupled to memory 201; The processor 202 calls the executable program code stored in the memory 201 to execute the steps in the ultrasound image analysis method based on multi-task learning described in Embodiment 1 or Embodiment 2 of the present invention.

[0104] Example 5 This invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the steps in the ultrasound image analysis method based on multi-task learning described in Embodiment 1 or Embodiment 2 of this invention.

[0105] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

Claims

1. A method for ultrasound image analysis based on multi-task learning, characterized in that, The method includes: Acquire ultrasound image data corresponding to the target object; in the preset multi-task learning model, the spatial path in the shared coding layer is used to generate a high-resolution feature map based on the ultrasound image data through a shallow network; the context path in the shared coding layer of the preset multi-task learning model is used to generate deep semantics based on the ultrasound image data through downsampling. The high-resolution feature map and the deep semantics are fused based on a preset feature fusion strategy to obtain fused features; The fused features are upsampled according to the deconvolution layer and convolution block in the multi-task learning model to obtain a target heatmap, which is used to represent the feature points corresponding to the target object in the ultrasound image data. The fused features are upsampled according to the feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map. The target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object. The image data corresponding to the target object is determined from the ultrasound image data based on the target heatmap and the target segmentation map.

2. The ultrasound image analysis method based on multi-task learning according to claim 1, characterized in that, The preset feature fusion strategy fuses the high-resolution feature map and the deep semantics to obtain fused features, including: After concatenating the high-resolution feature map and the deep semantics, the scale between the features is normalized to balance the features, thus obtaining the connection features; Pooling is performed on the connection features to generate intermediate feature vectors, and matrix multiplication is performed between the intermediate feature vectors and the connection features to obtain a weight vector. Based on the weight vector, the high-resolution feature map and the deep semantics are fused to obtain fused features.

3. The ultrasound image analysis method based on multi-task learning according to claim 1, characterized in that, The spatial paths in the shared coding layer of the preset multi-task learning model are used to generate high-resolution feature maps based on the ultrasound image data through a shallow network, including: In the preset multi-task learning model, the spatial path in the shared coding layer decomposes the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies through a shallow network. Each frequency subband is convolved and then concatenated with the window attention features of the shared coding layer, and a high-resolution feature map with multi-scale fusion is generated through linear transformation.

4. The ultrasound image analysis method based on multi-task learning according to any one of claims 3, characterized in that, The step of decomposing the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies includes: The ultrasound image data is filtered along the pixel row direction using a first filter and a second filter to obtain a high-frequency sub-band and a low-frequency sub-band, wherein the filtering frequency corresponding to the first filter is higher than the filtering frequency corresponding to the second filter. The high-frequency sub-band is filtered using a first filter and a second filter to obtain a first frequency sub-band and a second frequency sub-band. The low-frequency sub-band is filtered using a first filter and a second filter to obtain a third frequency sub-band and a fourth frequency sub-band.

5. The ultrasound image analysis method based on multi-task learning according to any one of claims 1-4, characterized in that, In the preset multi-task learning model, the context paths in the shared coding layer are downsampled to generate deep semantics based on the ultrasound image data, including: The preset multi-task learning model shares the context path in the coding layer and samples the 1 / 8 scale features, 1 / 16 scale features and 1 / 32 scale features corresponding to the ultrasound image data; Global pooling is performed on the 1 / 32 scale features to obtain global features; The global features are combined with the 1 / 8 scale features and the 1 / 16 scale features to generate deep semantics.

6. The ultrasound image analysis method based on multi-task learning according to claim 1, characterized in that, The preset multi-task learning model includes an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heatmap. The preset multi-task learning model is obtained through the following steps: The image segmentation task in the initial multi-task learning model is trained using the first ultrasound segmentation training data to obtain the intermediate multi-task learning model. Using the second ultrasonic segmentation training data and training heatmap data, the intermediate multi-task learning model is trained based on the first training strategy or the second training strategy to obtain a trained multi-task learning model. The first training strategy includes: In each training round, the second ultrasound segmentation training data and the training heatmap data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heatmap data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. The weighted and integrated loss function is obtained according to the weights corresponding to the image segmentation task and the feature point localization task, and the training process is fed back based on the loss function. The second training strategy includes: The image segmentation task and the feature point localization task are preset with corresponding probabilities. In each training round, one of the preset probabilities of the image segmentation task and the feature point localization task is selected for training.

7. The ultrasound image analysis method based on multi-task learning according to any one of claims 1-4 and 6, characterized in that, The ultrasound image data is the ultrasound image data corresponding to the right ventricle of the heart, the target object is the myocardium, the target heatmap is used to represent the feature points corresponding to the myocardium in the ultrasound image data, and the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the right ventricle of the heart is identified as myocardium.

8. An ultrasound image analysis device based on multi-task learning, characterized in that, The device includes: The data acquisition module is used to acquire ultrasound image data corresponding to the target object; The multi-task learning module is used to generate a high-resolution feature map based on the ultrasound image data by using the spatial path in the shared coding layer of the preset multi-task learning model through a shallow network; the multi-task learning module is also used to generate deep semantics based on the ultrasound image data by using the context path in the shared coding layer of the preset multi-task learning model through downsampling. The feature fusion module is used to fuse the high-resolution feature map and the deep semantics based on a preset feature fusion strategy to obtain fused features; The first decoding module is used to upsample the fused features according to the deconvolution layer and convolution block in the multi-task learning model to obtain a target heatmap, wherein the target heatmap is used to represent the feature points corresponding to the target object in the ultrasound image data. The second decoding module is used to upsample the fused features according to the feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map. The target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object. The result determination module is used to determine the image data corresponding to the target object from the ultrasound image data based on the target heatmap and the target segmentation map.

9. An ultrasound image analysis system based on multi-task learning, characterized in that, The system includes: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the ultrasound image analysis method based on multi-task learning as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the ultrasound image analysis method based on multi-task learning as described in any one of claims 1-7.

Images