Ultrasonic image analysis method and device based on multi-task learning, and storage medium

Abstract

The invention discloses an ultrasonic image analysis method based on multi-task learning, and relates to an ultrasonic image analysis technology. According to the ultrasonic image analysis method based on multi-task learning, two tasks share a coding layer to extract feature information; the coding layer comprises a space path and a context path, and performs feature integration through a feature fusion strategy; the decoding layers are respectively designed according to task requirements, and two independent branches are adopted to complete respective algorithm tasks; and finally, realizing efficient transmission of feature information between the coding layer and the decoding layer through jump connection. According to the embodiment of the invention, the target object can be extracted from the ultrasonic image data more accurately by combining the feature points with the segmentation image, so that the accuracy of ultrasonic image analysis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of ultrasonic image analysis, and in particular to an ultrasonic image analysis method and device based on multi-task learning, and a storage medium. Background Art

[0002] Ultrasound images, like optical images, contain a wealth of information. Analyzing ultrasound images and extracting useful information from them is a crucial task in ultrasound image analysis. For example, for an ultrasound image corresponding to a target object, ultrasound image analysis can be used to determine the image data corresponding to the target object.

[0003] Multi-task learning (MTL) is a learning method in machine learning that simultaneously handles multiple tasks, in contrast to single-task learning (STL). While STL typically optimizes for a single task, MTL allows multiple tasks to influence and promote each other in the learning process. By integrating supervisory signals from related tasks, multi-task learning achieves the joint optimization of multiple objective functions, significantly improving the model's generalization ability. In some cases, even with only a single primary objective, auxiliary tasks can improve the learning of the primary task by providing additional information.

[0004] However, practice has shown that the existing ultrasound image analysis method based on multi-task learning is not comprehensive enough in information mining of ultrasound images, and the construction of the multi-task learning model is not reasonable enough, resulting in inaccurate ultrasound image analysis. Therefore, it is necessary to propose a new ultrasound image analysis method based on multi-task learning to improve the accuracy of ultrasound image analysis. Summary of the Invention

[0005] The present invention provides an ultrasonic image analysis method and device, and a storage medium based on multi-task learning, which are used to improve the accuracy of ultrasonic image analysis.

[0006] In order to solve the above technical problems, the first aspect of the present invention discloses an ultrasound image analysis method based on multi-task learning, the method comprising: Acquire ultrasound image data corresponding to the target object; generate a high-resolution feature map based on the ultrasound image data through a shallow network using a spatial path in a shared coding layer in a preset multi-task learning model; generate a deep semantic map based on the ultrasound image data through downsampling using a context path in a shared coding layer in a preset multi-task learning model; The high-resolution feature map and the deep semantics are fused based on a preset feature fusion strategy to obtain a fused feature; Upsampling the fused features according to the deconvolution layer and the convolution block in the multi-task learning model to obtain a target heat map, where the target heat map is used to represent feature points corresponding to the target object in the ultrasound image data; Upsampling the fused features according to a feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map, where the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object; Image data corresponding to the target object is determined from the ultrasound image data according to the target heat map and the target segmentation map.

[0007] As an optional implementation, in the first aspect of the present invention, the high-resolution feature map and the deep semantics are fused based on a preset feature fusion strategy to obtain a fused feature, including: After concatenating the high-resolution feature map and the deep semantics, the scales between the features are balanced by normalization to obtain a connection feature; Performing a pooling operation on the connection feature to generate an intermediate feature vector, and performing a matrix dot multiplication calculation on the intermediate feature vector and the connection feature to obtain a weight vector; According to the weight vector, the high-resolution feature map and the deep semantics are fused to obtain a fused feature.

[0008] As an optional embodiment, in the first aspect of the present invention, the spatial path in the shared encoding layer in the preset multi-task learning model generates a high-resolution feature map based on the ultrasound image data through a shallow network, including: The spatial path in the shared coding layer of the preset multi-task learning model decomposes the ultrasound image data into a plurality of frequency sub-bands corresponding to different frequencies through a shallow network; Each of the frequency subbands is convolved with the window attention features of the shared coding layer, and a multi-scale fused high-resolution feature map is generated through linear transformation.

[0009] As an optional implementation, in the first aspect of the present invention, decomposing the ultrasound image data into a plurality of frequency sub-bands corresponding to different frequencies includes: Applying a first filter and a second filter to filter the ultrasound image data along a pixel row direction to obtain a high-frequency sub-band and a low-frequency sub-band, wherein a filtering frequency corresponding to the first filter is higher than a filtering frequency corresponding to the second filter; The high frequency sub-band is filtered by applying the first filter and the second filter to obtain the first frequency sub-band and the second frequency sub-band, and the low frequency sub-band is filtered by applying the first filter and the second filter to obtain the third frequency sub-band and the fourth frequency sub-band.

[0010] As an optional embodiment, in the first aspect of the present invention, the context path in the shared coding layer in the preset multi-task learning model is down-sampled to generate deep semantics according to the ultrasound image data, including: The context path in the shared encoding layer of the preset multi-task learning model is used to sample 1 / 8 scale features, 1 / 16 scale features, and 1 / 32 scale features corresponding to the ultrasound image data; Performing global pooling on the 1 / 32 scale features to obtain global features; The global features are combined with the 1 / 8 scale features and the 1 / 16 scale features to generate deep semantics.

[0011] As an optional embodiment, in the first aspect of the present invention, the preset multi-task learning model includes an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heat map; The preset multi-task learning model is obtained by the following steps: Using the first ultrasound segmentation training data to train the image segmentation task in the initial multi-task learning model to obtain an intermediate multi-task learning model; Using the second ultrasound segmentation training data and the training heat map data, training the intermediate multi-task learning model based on the first training strategy or the second training strategy to obtain a trained multi-task learning model; The first training strategy includes: In each round of training, the second ultrasound segmentation training data and the training heat map data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heat map data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. A weighted integrated loss function is obtained according to the weights corresponding to the image segmentation task and the feature point localization task, and feedback is provided to the training process based on the loss function. The second training strategy includes: Corresponding probabilities are preset for the image segmentation task and the feature point positioning task. In each round of training, one of the image segmentation task and the feature point positioning task is selected for training based on the corresponding probabilities preset for the image segmentation task and the feature point positioning task.

[0012] As an optional embodiment, in the first aspect of the present invention, the ultrasound image data is ultrasound image data corresponding to the right ventricle of the heart, the target object is the myocardium, the target heat map is used to represent the feature points corresponding to the central myocardium of the ultrasound image data, and the target segmentation map is used to represent the probability of each pixel in the ultrasound image data corresponding to the right ventricle of the heart being identified as myocardium.

[0013] A second aspect of the present invention discloses an ultrasonic image analysis device based on multi-task learning, the device comprising: A data acquisition module, used to acquire ultrasound image data corresponding to the target object; a multi-task learning module, configured to generate a high-resolution feature map based on the ultrasound image data by using a spatial path in a shared encoding layer in a preset multi-task learning model through a shallow network; and further configured to generate a deep semantic layer based on the ultrasound image data by downsampling using a context path in a shared encoding layer in the preset multi-task learning model; A feature fusion module is used to fuse the high-resolution feature map and the deep semantics based on a preset feature fusion strategy to obtain a fused feature; a first decoding module, configured to upsample the fused features according to the deconvolution layer and the convolution block in the multi-task learning model to obtain a target heat map, wherein the target heat map is used to represent feature points corresponding to the target object in the ultrasound image data; a second decoding module, configured to upsample the fused features according to a feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map, wherein the target segmentation map is used to represent a probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object; A result determination module is used to determine the image data corresponding to the target object from the ultrasound image data according to the target heat map and the target segmentation map.

[0014] As an optional implementation, in the second aspect of the present invention, the feature fusion module fuses the high-resolution feature map and the deep semantics based on a preset feature fusion strategy to obtain a specific operation of the fusion feature, including: After concatenating the high-resolution feature map and the deep semantics, the scales between the features are balanced by normalization to obtain a connection feature; Performing a pooling operation on the connection feature to generate an intermediate feature vector, and performing a matrix dot multiplication calculation on the intermediate feature vector and the connection feature to obtain a weight vector; According to the weight vector, the high-resolution feature map and the deep semantics are fused to obtain a fused feature.

[0015] As an optional embodiment, in the second aspect of the present invention, the multi-task learning module uses the spatial path in the shared encoding layer in the preset multi-task learning model through the shallow network to generate a high-resolution feature map according to the ultrasound image data, including: The spatial path in the shared coding layer of the preset multi-task learning model decomposes the ultrasound image data into a plurality of frequency sub-bands corresponding to different frequencies through a shallow network; Each of the frequency subbands is convolved with the window attention features of the shared coding layer, and a multi-scale fused high-resolution feature map is generated through linear transformation.

[0016] As an optional implementation, in the second aspect of the present invention, the specific operation mode of the multi-task learning module for decomposing the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies includes: Applying a first filter and a second filter to filter the ultrasound image data along a pixel row direction to obtain a high-frequency sub-band and a low-frequency sub-band, wherein a filtering frequency corresponding to the first filter is higher than a filtering frequency corresponding to the second filter; The high frequency sub-band is filtered by applying the first filter and the second filter to obtain the first frequency sub-band and the second frequency sub-band, and the low frequency sub-band is filtered by applying the first filter and the second filter to obtain the third frequency sub-band and the fourth frequency sub-band.

[0017] As an optional embodiment, in the second aspect of the present invention, the specific operation mode of generating deep semantics according to the ultrasound image data by downsampling the context path in the shared coding layer in the preset multi-task learning model includes: The context path in the shared encoding layer of the preset multi-task learning model is used to sample 1 / 8 scale features, 1 / 16 scale features, and 1 / 32 scale features corresponding to the ultrasound image data; Performing global pooling on the 1 / 32 scale features to obtain global features; The global features are combined with the 1 / 8 scale features and the 1 / 16 scale features to generate deep semantics.

[0018] As an optional embodiment, in the second aspect of the present invention, the preset multi-task learning model includes an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heat map; The preset multi-task learning model is obtained by the following steps: Using the first ultrasound segmentation training data to train the image segmentation task in the initial multi-task learning model to obtain an intermediate multi-task learning model; Using the second ultrasound segmentation training data and the training heat map data, training the intermediate multi-task learning model based on the first training strategy or the second training strategy to obtain a trained multi-task learning model; The first training strategy includes: In each round of training, the second ultrasound segmentation training data and the training heat map data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heat map data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. A weighted integrated loss function is obtained according to the weights corresponding to the image segmentation task and the feature point localization task, and feedback is provided to the training process based on the loss function. The second training strategy includes: Corresponding probabilities are preset for the image segmentation task and the feature point positioning task. In each round of training, one of the image segmentation task and the feature point positioning task is selected for training based on the corresponding probabilities preset for the image segmentation task and the feature point positioning task.

[0019] A third aspect of the present invention discloses an ultrasound image analysis system based on multi-task learning, the system comprising: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the ultrasound image analysis method based on multi-task learning disclosed in the first aspect of the present invention.

[0020] A fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the ultrasound image analysis method based on multi-task learning disclosed in the first aspect of the present invention.

[0021] Compared with the existing technology, the ultrasound image analysis method based on multi-task learning in the present invention shares the encoding layer for extracting feature information between two tasks; the encoding layer includes a spatial path and a context path, and integrates features through a feature fusion strategy; the decoding layer is designed separately according to task requirements, and uses two independent branches to complete their respective algorithm tasks; finally, the encoding layer and the decoding layer are connected through jump connections to achieve efficient transmission of feature information; the embodiment of the present invention combines feature points with segmentation maps to more accurately extract target objects from ultrasound image data, thereby improving the accuracy of ultrasound image analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 1 is a flow chart of an ultrasound image analysis method based on multi-task learning disclosed in an embodiment of the present invention; Figure 2 This is a diagram of the multi-task model network architecture disclosed in an embodiment of the present invention; Figure 3 Schematic diagram of the original image (left), myocardial segmentation data (center), and feature point positioning data (right) disclosed in an embodiment of the present invention; Figure 4 This is a structural diagram of an ultrasound image analysis system based on multi-task learning disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0025] The terms "first," "second," and so on, in the description and claims of the present invention and the accompanying drawings are used to distinguish between different items, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or end comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or end.

[0026] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0027] Example 1 See also Figure 1, Figure 1 : is a flow chart of an ultrasonic image analysis method based on multi-task learning disclosed in an embodiment of the present invention. Figure 1 The ultrasound image analysis method based on multi-task learning described above can be used in an ultrasound image analysis device based on multi-task learning. The ultrasound image analysis device based on multi-task learning can be integrated in a cloud server or a local server. Figure 1 As shown, the ultrasound image analysis method based on multi-task learning may include the following operations: Step 101: Acquire ultrasound image data corresponding to a target object.

[0028] In the embodiments of the present invention, the target object is an object requiring ultrasonic analysis, and can be a human organ, such as the heart, or any other object requiring ultrasonic analysis, such as an industrial device, a chemical medium, or a mechanical structure. The ultrasonic image data can be ultrasonic video data comprising multiple frames of ultrasonic images, a single ultrasonic image, or a collection of multiple ultrasonic images used to identify the same object. For example, the ultrasonic image data can be ultrasonic video data of the right ventricle of the heart. In this case, the image segmentation in this embodiment aims to accurately segment the myocardial image from the right ventricle of the heart.

[0029] Step 102: The spatial path in the shared coding layer in the preset multi-task learning model generates a high-resolution feature map based on the ultrasound image data through a shallow network; the context path in the shared coding layer in the preset multi-task learning model generates deep semantics based on the ultrasound image data through downsampling.

[0030] In an embodiment of the present invention, the spatial path generates high-resolution feature maps through a shallow network, preserving rich spatial information and supporting the decoding process of the feature point localization subtask. The high-resolution feature maps in the spatial path facilitate the decoding of the feature point localization subtask, avoiding information loss from excessively deep network layers and multiple attention calculations, and assisting in information extraction during the decoding process of the feature point subtask. The contextual path combines multi-stage features through a downsampling strategy to generate a feature representation with the largest receptive field and rich contextual information.

[0031] In this embodiment of the present invention, both feature extraction methods are implemented using a shared encoding layer within a pre-defined multi-task learning model, thereby enabling hard parameter sharing. Specifically, by sharing parameters within the encoding layer, the image segmentation task provides more regional information to the feature point localization task. Furthermore, the shared encoder design between the two subtasks significantly reduces the overall number of parameters in the multi-task model, helping to reduce both parameters and the total inference time for both subtasks.

[0032] Step 103: Based on a preset feature fusion strategy, the high-resolution feature map and the deep semantics are fused to obtain fused features.

[0033] In this embodiment of the present invention, the feature representations of spatial paths and contextual paths have different levels and focuses. The spatial information obtained by the spatial path encodes the vast majority of detailed information, and the output feature information is relatively low-level and microscopic. In contrast, the contextual information obtained by the contextual path encodes rich semantic and global information, and the output feature information is relatively high-level and macroscopic. Based on a preset feature fusion strategy, the high-resolution feature map and deep semantics are fused to obtain fused features, thereby providing rich and efficient data for subsequent model processing.

[0034] Step 104: Upsample the fused features according to the deconvolution layer and convolution block in the multi-task learning model to obtain a target heat map.

[0035] In an embodiment of the present invention, the target heat map is used to represent feature points corresponding to the target object in the ultrasound image data.

[0036] In an embodiment of the present invention, a multi-task learning framework is constructed, in which the decoder is designed as an independent branch for two different subtasks to meet the different requirements of the two subtasks and improve the performance and efficiency of the model. This architectural choice is based on the differences in feature extraction and processing of the subtasks. Among them, for the feature point positioning task, due to its high demand for the recognition of local features, the model uses a deconvolution layer in conjunction with continuous convolution blocks for upsampling to achieve effective decoding of features and detail capture. This design reduces the computational load while enhancing the ability to recognize details through the advantages of the convolution layer. Optionally, each convolution block contains a convolution layer, batch normalization, and a ReLU activation function to enhance the nonlinear expression ability of the model.

[0037] Step 105: Upsample the fused features according to the feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map.

[0038] In this embodiment of the present invention, the target segmentation map represents the probability of each pixel in the ultrasound image data corresponding to the target object being identified as the target object. Given the image segmentation task's need to identify the overall structure, this embodiment of the present invention uses a feature extraction module corresponding to the shared coding layer as the core of the decoder. This structure enables it to capture long-range dependencies, which are crucial for understanding the relative positional relationships of the target objects, thereby improving image segmentation accuracy.

[0039] If two tasks share the same decoding branch, even though there is some overlap in data features, the larger area involved in segmentation and the smaller target area for feature point localization make it difficult for the network to effectively handle the two distinct objectives simultaneously, potentially leading to difficulties and convergence issues during training. Therefore, the design of independent decoding branches in the embodiments of the present invention facilitates more accurate updates of network parameters during backpropagation.

[0040] Step 106: Determine image data corresponding to the target object from the ultrasound image data according to the target heat map and the target segmentation map.

[0041] In an embodiment of the present invention, the target heat map is used to represent the feature points corresponding to the target object in the ultrasound image data; the target segmentation map is used to represent the probability of each pixel in the ultrasound image data corresponding to the target object being identified as the target object. The feature points combined with the segmentation map can more accurately extract the target object from the ultrasound image data for subsequent various processing.

[0042] It can be seen that in the ultrasound image analysis method based on multi-task learning in the embodiment of the present invention, the two tasks share the encoding layer to extract feature information; secondly, the encoding layer includes a spatial path and a context path, and features are integrated through a feature fusion strategy; thirdly, the decoding layer is designed separately according to task requirements, and two independent branches are used to complete their respective algorithm tasks; finally, the efficient transmission of feature information is achieved between the encoding layer and the decoding layer through jump connections; the embodiment of the present invention can more accurately extract the target object from the ultrasound image data by combining feature points with the segmentation map, so as to improve the accuracy of ultrasound image analysis.

[0043] In an optional embodiment, the high-resolution feature map and the deep semantics are fused based on a preset feature fusion strategy to obtain fused features, including: After concatenating the high-resolution feature map and deep semantics, the scales between the features are balanced by normalization to obtain the connection features; Perform pooling operation on the connection features to generate intermediate feature vectors, and perform matrix multiplication of the intermediate feature vectors and the connection features to obtain the weight vectors; According to the weight vector, the high-resolution feature map and deep semantics are fused to obtain the fused feature.

[0044] This optional embodiment effectively combines features from both paths based on a pre-defined feature fusion strategy. Because the spatial and contextual path feature representations have different hierarchies and focuses, simple weighted fusion is ineffective. Therefore, a targeted fusion strategy can be designed. To better fuse the two features, an Attention Refinement Module (ARM) can be added to the context path.

[0045] ARM is used in the context path to optimize the outputs of the first two stages. ARM obtains global semantic information through global average pooling. Then, through convolution, regularization, and sigmoid operations, it performs matrix dot multiplication with the original output features to calculate a feature vector to guide feature learning. ARM optimizes the results in the context path and integrates global features without upsampling, resulting in low overhead and almost negligible computational cost.

[0046] In the feature fusion strategy, the outputs of the two paths are first concatenated, and then normalization is performed to balance the scales of the features. A feature vector is generated by pooling the concatenated features, and a weight vector is calculated by matrix multiplication with the original features. This weight vector is then reweighted to complete the feature selection and fusion process. Ultimately, the feature fusion strategy combines high-level global information from the encoder with low-level detailed features, preserving the rich original feature information and providing sufficient support for the decoder to accurately decode according to the subtask requirements.

[0047] Furthermore, the feature fusion strategy described above results in a relatively small increase in overall parameter and computational complexity. Experimental verification of this optional embodiment showed that introducing the feature fusion strategy only resulted in an additional 515.21K parameters and 16.47M FLOPs, a negligible increase compared to the overall parameter and computational complexity of the multi-task framework model.

[0048] In another optional embodiment, for ultrasound data, its low-frequency components often contain basic signal features, while the high-frequency signals provide detailed information about the signal, such as edge contours of image information. Therefore, to further mine ultrasound data information, the spatial paths in the shared encoding layer of the aforementioned preset multi-task learning model are passed through a shallow network to generate a high-resolution feature map based on the ultrasound image data, which may include: The spatial path in the shared encoding layer of the preset multi-task learning model decomposes the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies through a shallow network; Each frequency subband is convolved with the window attention features of the shared coding layer, and a multi-scale fused high-resolution feature map is generated through linear transformation.

[0049] In this optional embodiment, the spatial path generates a high-resolution feature map through a shallow network, retains rich spatial information, and supports the decoding process of the feature point positioning subtask. Optionally, the spatial path is the feature information of the first stage in the encoder network. At this time, the original tensor of the ultrasound image is divided into multiple regions and adjusted to 1 / 4 resolution through vector embedding, and the channel is a feature vector of C. The feature vector extracts a larger-scale feature map and retains rich spatial information. The high-resolution feature map in the spatial path helps in decoding the feature point positioning subtask, avoids the loss of information encoded by too deep a network layer and multiple attention calculations, and helps the feature point subtask extract information during the decoding process.

[0050] In this optional embodiment, further optionally, decomposing the ultrasound image data into a plurality of frequency sub-bands corresponding to different frequencies may include: Applying a first filter and a second filter to filter the ultrasound image data along a pixel row direction to obtain a high-frequency sub-band and a low-frequency sub-band, wherein a filtering frequency corresponding to the first filter is higher than a filtering frequency corresponding to the second filter; The high frequency sub-band is filtered by the first filter and the second filter to obtain the first frequency sub-band and the second frequency sub-band, and the low frequency sub-band is filtered by the first filter and the second filter to obtain the third frequency sub-band and the fourth frequency sub-band.

[0051] Specifically, in order to effectively provide multi-scale information, this optional embodiment proposes a method of combining frequency sub-bands of different frequencies with a window attention mechanism. Specifically, the input feature map Decomposed into four wavelet sub-bands, first, the image is filtered by low-pass and high-pass filters. The low-pass filter outputs low-frequency sub-bands, which represent the basic information of the image; the high-pass filter outputs high-frequency sub-bands, which represent the detailed information. For example, for the input feature map Apply low-pass and high-pass filters along the line, first Encoded into two subbands and , and then continue to use the same low-pass filter and high-pass filter along these two sub-bands to obtain all four sub-bands 、 、 and ( ), Represents low-frequency components and can preserve features Basic information, 、 and Represents high-frequency components and can preserve features The four sub-bands cover every detail of the input without causing any loss of feature information. Then connect these four wavelet sub-bands along the channel dimension and further use The convolution imposes spatial locality to obtain , then, The final attention result is obtained by splicing the output result of the output layer of the image segmentation model , and finally through a linear transformation layer to obtain .

[0052] This optional embodiment stacks the four sub-bands and fuses them with the window attention feature, allowing the model to learn more information from different scales. This not only allows the model to focus on the attention information within the window, but also learns the characteristics of the image at multiple scales, including basic information and detailed information.

[0053] In yet another optional embodiment, the context path in the shared encoding layer in the preset multi-task learning model generates deep semantics based on ultrasound image data by downsampling, which may include: The context path in the shared encoding layer of the preset multi-task learning model is used to sample the 1 / 8 scale features, 1 / 16 scale features, and 1 / 32 scale features corresponding to the ultrasound image data; Perform global pooling on the 1 / 32 scale features to obtain global features; The global features are combined with the 1 / 8 scale features and the 1 / 16 scale features to generate deep semantics.

[0054] In this optional embodiment, the context path combines multi-stage features through a downsampling strategy to generate a feature representation with maximum receptive field and rich context information. Optionally, in the context path, the feature information output from two stages is combined, namely the scale of 1 / 8 and the scale of 1 / 16, and at the end of the encoding layer, that is, the features at the scale of 1 / 32, the global context information with the maximum receptive field is obtained through global average pooling. Finally, the global features are combined with the features of the first two stages to form a context path with rich feature information.

[0055] In yet another optional embodiment, the preset multi-task learning model includes an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heat map.

[0056] The multi-task framework model designed in the embodiment of the present invention has the following characteristics: first, the two tasks share the encoding layer; second, the encoding layer includes a spatial path and a context path, and features are integrated through a feature fusion strategy; third, the decoding layer is designed separately according to the task requirements, and two independent branches are used to complete their respective algorithm tasks; finally, the efficient transmission of feature information is achieved between the encoding layer and the decoding layer through jump connections. These characteristics reflect the complexity of the multi-task model structure, which makes the training process more difficult. Therefore, it is necessary to formulate an optimization strategy that can effectively improve the training effect. The above-mentioned preset multi-task learning model is obtained through the following steps: Using the first ultrasound segmentation training data to train the image segmentation task in the initial multi-task learning model to obtain an intermediate multi-task learning model; Using the second ultrasound segmentation training data and the training heat map data, training the intermediate multi-task learning model based on the first training strategy or the second training strategy to obtain a trained multi-task learning model; Among them, the first training strategy includes: In each round of training, the second ultrasound segmentation training data and training heat map data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heat map data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. A weighted integrated loss function is obtained based on the weights corresponding to the image segmentation task and the feature point localization task, and feedback is provided to the training process based on the loss function. The second training strategy includes: The corresponding probabilities are preset for the image segmentation task and the feature point localization task. In each round of training, one of the image segmentation task and the feature point localization task is selected for training based on the corresponding probabilities.

[0057] This optional embodiment is designed based on the properties of the two subtasks and the characteristics of the dataset, dividing the training process of the entire multi-task framework into two stages. In the first stage, the segmentation task is trained using the first ultrasound segmentation training data that does not contain a feature point dataset. In the second stage, the feature point localization task is added, and the two branches are trained together. The idea behind this design is that when the entire multi-task framework is trained directly at the beginning, that is, when training two task branches simultaneously, the model always tends to learn segmentation and performs poorly in feature point localization. In addition, the network oscillates significantly and converges slowly in the early stages of training. This optional embodiment analyzes the reasons for the following two points: (1) In the multi-task learning framework, the feature point localization task is more challenging than the segmentation task. The target of the feature point localization task is relatively small and requires capturing the fine feature point positions. Identifying these subtle target points in the image is more difficult than identifying target objects in larger areas, especially when the image quality is poor. Although the data distribution of the two tasks is similar, the multi-task network may tend to learn more significant features, namely the segmentation task, in the early stages of training, which may cause the learning of the feature point localization task to be relatively neglected. In addition, in the single-task learning of feature point localization, the network training of the feature point localization task usually converges slowly because the accurate positioning of these small targets requires the network to learn more refined feature representations.

[0058] (2) In the early stages of network training, the encoding layer and the two task branches have the following two problems: 1) the network weights are in an unstable state; 2) the loss signal during the backpropagation of the feature point localization task may deviate from the target point, leading to instability and non-convergence during training. Therefore, directly adding the feature point task to the training at the beginning of network training does not fully utilize the similarities between the segmentation task and the feature point task to promote each other.

[0059] In this optional embodiment, two training strategies are proposed for the training strategy of simultaneously training two branches in the second stage: 1) overall training, which trains the networks of the two branches simultaneously by controlling the weights to calculate a unified multi-task loss function; 2) selection according to probability, which randomly selects one of the branches for training in each round.

[0060] The core of the first training strategy is to load the datasets for both tasks simultaneously during each training round and calculate the gradients for both separately. Next, different weights are assigned to the two subtasks, and their loss functions are weighted and integrated to generate a comprehensive overall loss function. This loss function optimizes the network parameters through backpropagation, thereby achieving joint training of feature point localization and segmentation tasks. This approach coordinates the relationship between the two tasks based on a shared model structure, ensuring optimal model performance.

[0061] The second training strategy uses a probabilistic approach to select task branches for training, meaning only one subtask is trained per round. Because the segmentation task has a large data volume and its data distribution covers the distribution area of ​​the target object's feature points, this optional embodiment selects the segmentation task branch for training with a higher probability. The gradients returned by the segmentation task enhance the generalization capability of feature extraction in the encoding layer and provide prior information on regional positioning for the feature point localization task, thereby promoting the training effect of the feature point localization task.

[0062] It should be noted that, in an embodiment of the present invention, the ultrasound image data is ultrasound image data corresponding to the right ventricle of the heart, the target object is the myocardium, the target heat map is used to represent the feature points corresponding to the central myocardium of the ultrasound image data, and the target segmentation map is used to represent the probability of each pixel in the ultrasound image data corresponding to the right ventricle of the heart being identified as myocardium.

[0063] Example 2 In the first embodiment of the present invention, an ultrasound image analysis method based on multi-task learning is proposed. When the ultrasound image data is ultrasound image data corresponding to the right ventricle of the heart, the target object is the myocardium, the target heat map is used to represent the feature points corresponding to the central myocardium of the ultrasound image data, and the target segmentation map is used to represent the probability of each pixel in the ultrasound image data corresponding to the right ventricle of the heart being identified as the myocardium, the second embodiment of the present invention proposes a feature point positioning assisted myocardial segmentation algorithm based on multi-task learning.

[0064] Specifically, the embodiment of the present invention designs a multi-task learning framework that can perform right ventricular myocardial segmentation results and right ventricular anatomical feature point coordinate prediction on the input apical four-chamber ultrasound image. Figure 2 The multi-task framework of the embodiment of the present invention integrates the right ventricular myocardial segmentation task and the right ventricular anatomical feature point task.

[0065] Combine Figure 2 The multi-task network framework proposed in the embodiment of the present invention has the following three characteristics: (1) The idea of ​​hard parameter sharing is adopted, and the same coding block is used for feature extraction in the coding layer. This design is mainly based on two considerations: First, the right ventricular myocardium segmentation algorithm can use the complete cardiac cycle of ultrasound images for training, but the right ventricular feature point localization algorithm only uses a few frames of ultrasound images of the cardiac cycle for training, so the right ventricular myocardium segmentation task can learn more feature information. By sharing the parameters in the coding layer, the right ventricular myocardium segmentation task provides more myocardial area information to the right ventricular feature point localization task; secondly, the design of the shared encoder for the two subtasks forms a hard parameter sharing mechanism, which has obvious advantages in the overall parameter amount of the multi-task model, helps reduce the total parameter amount, and can also reduce the total inference time of the two subtasks.

[0066] (2) Based on the characteristics of both tasks, a multi-path feature fusion module is designed. This embodiment of the present invention divides the encoder into a spatial path and a context path, and performs information fusion through an attention fusion module. The spatial path retains high-resolution image information, while the context path can obtain deeper network information through down-sampled information, thereby obtaining rich receptive fields at different scales. The features of the two parts are then fused through a designed feature fusion module.

[0067] (3) The two tasks use different decoder branches based on their characteristics. In the feature point task, convolution blocks and deconvolution are used for decoding, and finally the image is restored to a heat map of the same resolution for output. In the myocardial segmentation task, the decoding blocks corresponding to the encoding layer are symmetrically used for decoding, and the predicted mask of myocardial segmentation is output.

[0068] Specifically, in this embodiment of the present invention, the spatial path and the context path are able to preserve key spatial information while achieving a sufficiently large receptive field. Furthermore, the feature fusion module effectively integrates feature information at different levels. The multi-task proposed in this embodiment requires capturing detailed features while obtaining high-resolution features. This advantage is applicable to the two subtasks of this embodiment of the present invention: myocardial segmentation and specific point localization.

[0069] The spatial path generates high-resolution feature maps through a shallow network, retaining rich spatial information and supporting the decoding process of the feature point positioning subtask. Specifically, the spatial path is the feature information of the first stage in the encoder network. At this time, the original tensor of the ultrasound image is divided into multiple regions and adjusted to 1 / 4 resolution through vector embedding, with the channel being the feature vector of C. This feature vector extracts a larger-scale feature map through two encoding blocks, retaining rich spatial information. The high-resolution feature map in the spatial path helps in decoding the feature point positioning subtask, avoiding the loss of information encoded by too deep a network layer and multiple attention calculations, and helps extract information from the feature point subtask during the decoding process.

[0070] The context path combines multi-stage features through a downsampling strategy to generate feature representations with the largest receptive field and rich context. The context path combines feature information from two stages, at the 1 / 8 scale and the 1 / 16 scale, respectively. At the end of the encoding layer, the 1 / 32 scale features are pooled using global average pooling to obtain global context with the largest receptive field. Finally, this global feature is combined with features from the first two stages to form a context path with rich feature information.

[0071] Ultimately, the feature fusion module is used to effectively combine the features of the two paths. Since the feature representations of the spatial path and the contextual path have different levels and focuses, simple weighting is difficult to achieve effective fusion, so a specially designed fusion strategy is required. The spatial information obtained by the spatial path encodes the vast majority of detailed information, and the output feature information is relatively low-level and microscopic. The contextual information obtained by the contextual path encodes rich semantics and global information, and the output feature information is relatively high-level and macroscopic. To better fuse the two features, a feature fusion module is used for fusion, and an attention optimization module is added to the contextual path.

[0072] The attention optimization module is used in the context path to optimize the outputs of the first two stages. It obtains global semantic information through global average pooling. Then, through convolution, regularization, and activation functions, it performs matrix dot multiplication with the original output features to calculate a feature vector to guide feature learning. The attention optimization module optimizes the results in the context path and integrates global features without upsampling, resulting in low overhead and negligible computational cost.

[0073] The feature fusion module is primarily used to effectively fuse two features. It first concatenates the outputs of the two paths and then balances the scales of the features through normalization. In the feature fusion module, a feature vector is generated by pooling the concatenated features. This vector is then matrix-multiplied with the original features to produce a weight vector. This weight vector is then reweighted to complete the feature selection and fusion process. Ultimately, the feature fusion module combines high-level global information from the encoder with low-level detailed features, preserving the rich original feature information and providing sufficient support for the decoding component to accurately decode according to the subtask requirements.

[0074] In addition, after adding the multi-path feature fusion module and the attention optimization module, the increase in the overall parameter volume and computational complexity is relatively small. In the embodiment of the present invention, the introduction of these modules only adds an additional 515.21K parameters and 16.47M FLOPs, which is negligible compared to the overall parameter volume and computational complexity of the multi-task framework model. In an optional embodiment, a multi-task learning framework is used, in which the decoder is designed as independent branches for two different subtasks, meeting the different requirements of the two subtasks and improving the performance and efficiency of the model. This architectural choice is based on the differences in feature extraction and processing between the subtasks.

[0075] For feature point localization tasks, which require high local feature recognition, the model employs deconvolution layers combined with continuous convolution blocks for upsampling to effectively decode features and capture detail. This design reduces computational overhead while leveraging the advantages of convolutional layers to enhance detail recognition. Specifically, each convolution block includes a convolution layer, batch normalization, and ReLU activation functions to enhance the model's nonlinear representation capabilities.

[0076] For the myocardial segmentation task, given its requirement for overall structure recognition, the model uses the feature extraction module corresponding to the shared encoding layer as the core of the decoder.

[0077] Furthermore, the design of independent decoding branches facilitates more accurate updating of network parameters during backpropagation. If two tasks share the same decoding branch, even though there is some overlap in data features, myocardial segmentation involves a larger area, while the target area for feature point localization is relatively smaller. This makes it difficult for the network to effectively handle the two distinct objectives simultaneously, potentially leading to difficulties and convergence issues during training.

[0078] In a multi-task framework, embodiments of the present invention implement a hard parameter sharing strategy, whereby different subtasks share encoder parameters while retaining an independent parameter set for each task. This strategy enables the model to capture common features during the encoding phase and learn task-specific details during the decoding phase, thereby improving the model's overall efficiency and accuracy. The hard parameter sharing strategy is suitable for processing highly correlated tasks, and due to parameter sharing, it reduces model complexity and the risk of overfitting.

[0079] In another optional embodiment, the model training process is as follows: First, starting from the dataset used by the multi-task framework model, the embodiment of the present invention believes that the dataset for right ventricular myocardial segmentation training should be added to the dataset training model for the right ventricular feature point positioning task. This training strategy is because both tasks can use the dataset for feature point positioning separately, but the feature point positioning model has poor feature point positioning effect when the amount of data is small. Because the area of ​​right ventricular myocardial segmentation coincides with the largest distribution area of ​​right ventricular feature points, such as Figure 3 As shown, the myocardial segmentation dataset can be used to learn information about segmented regions. Furthermore, in a multi-task framework, using only the feature point localization dataset for both tasks results in suboptimal results for both tasks. This is because the myocardial segmentation task has a small sample size, which ultimately reduces the performance on the test set. It is proposed that using incremental datasets for training is necessary. Using incremental datasets allows the feature point localization branch in the multi-task framework to obtain more regional distribution information from the larger myocardial segmentation dataset.

[0080] Secondly, considering the training process of the entire network, the embodiment of the present invention believes that it should be designed according to the properties of the two subtasks and the characteristics of the data set, and the training process of the entire multi-task framework is divided into two stages. In the first stage, the myocardial segmentation task is trained using a myocardial segmentation dataset that does not contain a feature point dataset. In the second stage, the feature point positioning task is added, and the two branches are trained together. The idea of ​​this design is that when the entire multi-task framework is trained directly at the beginning, that is, when the two task branches are trained at the same time, the model always tends to learn right ventricular myocardial segmentation, and performs poorly in right ventricular feature point positioning. In addition, the network oscillates significantly and converges slowly in the early stages of training. The embodiment of the present invention analyzes the reasons as follows: (1) In the multi-task learning framework, the feature point localization task is more challenging than the myocardial segmentation task. The targets of the feature point localization task are relatively small and require capturing fine feature point positions. Identifying these subtle target points in the image is more difficult than identifying larger areas of myocardium, especially when the image quality is poor. Although the data distribution of the two tasks is similar, the multi-task network may tend to learn more significant features, namely the myocardial segmentation task, in the early stages of training, which may lead to the learning of the feature point localization task being relatively neglected. In addition, in the single-task learning of feature point localization, the network training of the feature point localization task usually converges slowly because the accurate positioning of these small targets requires the network to learn more refined feature representations.

[0081] (2) In the early stages of network training, the encoding layer and the two task branches have the following two problems: 1) the network weights are in an unstable state; 2) the loss signal during the backpropagation of the feature point localization task may deviate from the target point, leading to instability and non-convergence during training. Therefore, directly adding the feature point task to the training at the beginning of network training does not fully utilize the similarity between the myocardial segmentation task and the feature point task to promote each other.

[0082] Finally, regarding the training strategy for simultaneously training two branches in the second stage, the embodiments of the present invention propose two training strategies: 1) overall training, which simultaneously trains the networks of the two branches by controlling the weights to calculate a unified multi-task loss function; 2) selection based on probability, which randomly selects one of the branches for training in each round.

[0083] The core of the first training strategy is to load the datasets for both tasks simultaneously during each training round and calculate the gradients for both separately. Next, different weights are assigned to the two subtasks, and their loss functions are weighted and integrated to generate a comprehensive overall loss function. This loss function optimizes the network parameters via backpropagation, thereby achieving joint training of feature point localization and myocardial segmentation. This approach, based on a shared model structure, coordinates the relationship between the two tasks, ensuring optimal model performance.

[0084] The second training strategy adopts the method of selecting task branches for training based on probability, that is, only one subtask is trained in each round. Since the right ventricular myocardial segmentation task has a large amount of data and its data distribution covers the distribution area of ​​the right ventricular feature points, the embodiment of the present invention will select the right ventricular myocardial segmentation task branch for training with a higher probability. The gradient returned by the right ventricular myocardial segmentation task can enhance the generalization ability of the feature extraction of the coding layer, and at the same time provide the regional positioning prior information for the right ventricular feature point positioning task, thereby promoting the training effect of the right ventricular feature point positioning task.

[0085] In yet another optional embodiment, the loss function is designed as follows: Because the directions and magnitudes of training gradients for different tasks vary, multi-task learning requires considering conflicts and balances between the loss functions of each task during training. If the gradients are inconsistent in magnitude, the convergence speeds of different tasks will be inconsistent; if the directions are inconsistent, conflicts will arise, and even gradient cancellation due to inconsistent optimization directions, which will affect the training speed and quality of the model.

[0086] At present, the loss function design strategy of multi-task mainly focuses on three aspects: magnitude, speed and gradient. Among them, the total loss function in multi-task learning is As shown in the following formula:

[0087] in, is the weight coefficient, is the loss function for the i-th task. PcGrad starts from the direction of the gradient, considers the angle between gradient vectors, and performs projection calculations to avoid gradient skewing between different tasks. MGDA is a multi-objective optimization problem that believes that multi-objective optimization problems can be solved using gradient descent to obtain local optimal solutions, ultimately achieving Pareto optimality. HTW is applicable to scenarios where multi-task scenarios are divided into primary and auxiliary tasks. By learning more information from auxiliary tasks in the early stages of training, the team can then focus on learning the primary task, while auxiliary tasks are not important.

[0088] However, the multi-task framework designed in the embodiment of the present invention is not applicable to all optimization strategies. For example, the gradient backpropagation strategy of GradNorm and MGDA is not applicable to the jump connection structure in the multi-task learning framework of the embodiment of the present invention. The HTW strategy relies on the mechanism of completing the learning of the auxiliary task first and then focusing on learning the main task, but the two tasks in the embodiment of the present invention need to produce valid outputs at the same time, so it is not applicable. In addition, although the uncertainty weight allocation method proposed by Kendall et al. can dynamically adjust the task weights, it is premised on the need for a clear mathematical derivation of the loss function of each task. However, the loss function design of the two subtasks in the embodiment of the present invention is relatively complex and cannot provide a concise and clear expression. Therefore, this method is difficult to be effectively applied in the multi-task learning framework of the embodiment of the present invention.

[0089] Finally, the embodiment of the present invention chooses to adjust the loss function weights of different tasks according to the difficulty of the task and the size of the gradient in the task. The final loss function is as follows, as shown in the following formula, where Indicates that during the training process Round The weight of the loss function for each task.

[0090]

[0091] Among them, we need to pay attention to the size of the gradient descent during training, that is, the speed of network learning. We can define the task In the The learning rate of the round is as follows:

[0092] when The smaller the value, the faster the loss decreases and the faster the network learns, so the weight of the task needs to be set lower. Normalize and get the weight coefficients of different tasks, as shown in the following formula, where is the temperature coefficient, which is used to adjust the weight differences between different tasks. The larger the value, the more evenly the weight distribution is. When is large enough, the weight of each task is close to 1.

[0093]

[0094] In another strategy, KPIs are selected as the standard for measuring weights. First, KPIs need to be selected for the feature point task and the myocardial segmentation task. In the embodiment of the present invention, the KPI of the myocardial segmentation task is defined as the Dice similarity coefficient, and the KPI of the feature point positioning task is the detection rate. It can be expressed as follows, where Indicates a task In the KPI of the round, and required in DTP .

[0095]

[0096] Example 3 An embodiment of the present invention discloses an ultrasound image analysis device based on multi-task learning, which may include: A data acquisition module, used to acquire ultrasound image data corresponding to the target object; A multi-task learning module is configured to generate a high-resolution feature map based on the ultrasound image data by using a spatial path in a shared encoding layer in a preset multi-task learning model through a shallow network; the multi-task learning module is further configured to generate a deep semantic map based on the ultrasound image data by downsampling using a context path in a shared encoding layer in a preset multi-task learning model; The feature fusion module is used to fuse high-resolution feature maps and deep semantics based on a preset feature fusion strategy to obtain fused features; A first decoding module is used to upsample the fused features according to the deconvolution layer and the convolution block in the multi-task learning model to obtain a target heat map, where the target heat map is used to represent feature points corresponding to the target object in the ultrasound image data; a second decoding module, configured to upsample the fused features according to the feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map, where the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object; The result determination module is used to determine the image data corresponding to the target object from the ultrasound image data according to the target heat map and the target segmentation map.

[0097] In an optional embodiment, the feature fusion module fuses the high-resolution feature map and the deep semantics based on a preset feature fusion strategy to obtain the fusion feature. The specific operation method may include: After concatenating the high-resolution feature map and deep semantics, the scales between the features are balanced by normalization to obtain the connection features; Perform pooling operation on the connection features to generate intermediate feature vectors, and perform matrix multiplication of the intermediate feature vectors and the connection features to obtain the weight vectors; According to the weight vector, the high-resolution feature map and deep semantics are fused to obtain the fused feature.

[0098] In yet another optional embodiment, the multi-task learning module generates a high-resolution feature map based on ultrasound image data using a spatial path in a shared encoding layer in a preset multi-task learning model through a shallow network. The specific operation may include: The spatial path in the shared encoding layer of the preset multi-task learning model decomposes the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies through a shallow network; Each frequency subband is convolved with the window attention features of the shared coding layer, and a multi-scale fused high-resolution feature map is generated through linear transformation.

[0099] In yet another optional embodiment, a specific operation of the multi-task learning module to decompose the ultrasound image data into multiple frequency sub-bands corresponding to different frequencies may include: Applying a first filter and a second filter to filter the ultrasound image data along a pixel row direction to obtain a high-frequency sub-band and a low-frequency sub-band, wherein a filtering frequency corresponding to the first filter is higher than a filtering frequency corresponding to the second filter; The high frequency sub-band is filtered by the first filter and the second filter to obtain the first frequency sub-band and the second frequency sub-band, and the low frequency sub-band is filtered by the first filter and the second filter to obtain the third frequency sub-band and the fourth frequency sub-band.

[0100] In yet another optional embodiment, the specific operation of generating deep semantics based on ultrasound image data by downsampling the context path in the shared coding layer in the preset multi-task learning model may include: The context path in the shared encoding layer of the preset multi-task learning model is used to sample the 1 / 8 scale features, 1 / 16 scale features, and 1 / 32 scale features corresponding to the ultrasound image data; Perform global pooling on the 1 / 32 scale features to obtain global features; The global features are combined with the 1 / 8 scale features and the 1 / 16 scale features to generate deep semantics.

[0101] In yet another optional embodiment, the preset multi-task learning model may include an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heat map; The preset multi-task learning model is obtained through the following steps: Using the first ultrasound segmentation training data to train the image segmentation task in the initial multi-task learning model to obtain an intermediate multi-task learning model; Using the second ultrasound segmentation training data and the training heat map data, training the intermediate multi-task learning model based on the first training strategy or the second training strategy to obtain a trained multi-task learning model; The first training strategy may include: In each round of training, the second ultrasound segmentation training data and training heat map data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heat map data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. A weighted integrated loss function is obtained based on the weights corresponding to the image segmentation task and the feature point localization task, and feedback is provided to the training process based on the loss function. Secondary training strategies may include: The corresponding probabilities are preset for the image segmentation task and the feature point localization task. In each round of training, one of the image segmentation task and the feature point localization task is selected for training based on the corresponding probabilities.

[0102] In another optional embodiment, the ultrasound image data is ultrasound image data corresponding to the right ventricle of the heart, the target object is the myocardium, the target heat map is used to represent the feature points corresponding to the central myocardium of the ultrasound image data, and the target segmentation map is used to represent the probability of each pixel in the ultrasound image data corresponding to the right ventricle of the heart being identified as myocardium.

[0103] Example 4 See also Figure 4 , Figure 4FIG is a schematic diagram of the structure of an ultrasound image analysis system based on multi-task learning disclosed in an embodiment of the present invention. Figure 4 As shown, the ultrasound image analysis system based on multi-task learning may include: A memory 201 storing executable program code; a processor 202 coupled to the memory 201; The processor 202 calls the executable program code stored in the memory 201 to execute the steps of the ultrasound image analysis method based on multi-task learning described in the first or second embodiment of the present invention.

[0104] Example 5 An embodiment of the present invention discloses a computer storage medium storing computer instructions. When the computer instructions are called, they are used to execute the steps of the ultrasound image analysis method based on multi-task learning described in Embodiment 1 or Embodiment 2 of the present invention.

[0105] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Those skilled in the art can understand and implement the present invention without inventive effort.

Claims

1. An ultrasound image analysis method based on multi-task learning, characterized in that: The method comprises: Acquire ultrasound image data corresponding to the target object; generate a high-resolution feature map based on the ultrasound image data through a shallow network using a spatial path in a shared coding layer in a preset multi-task learning model; generate a deep semantic map based on the ultrasound image data through downsampling using a context path in a shared coding layer in a preset multi-task learning model; The high-resolution feature map and the deep semantics are fused based on a preset feature fusion strategy to obtain a fused feature; Upsampling the fused features according to the deconvolution layer and the convolution block in the multi-task learning model to obtain a target heat map, where the target heat map is used to represent feature points corresponding to the target object in the ultrasound image data; Upsampling the fused features according to a feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map, where the target segmentation map is used to represent the probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object; Image data corresponding to the target object is determined from the ultrasound image data according to the target heat map and the target segmentation map.

2. The ultrasonic image analysis method based on multi-task learning according to claim 1, characterized in that: The high-resolution feature map and the deep semantics are fused based on a preset feature fusion strategy to obtain a fused feature, including: After concatenating the high-resolution feature map and the deep semantics, the scales between the features are balanced by normalization to obtain a connection feature; Performing a pooling operation on the connection feature to generate an intermediate feature vector, and performing a matrix dot multiplication calculation on the intermediate feature vector and the connection feature to obtain a weight vector; According to the weight vector, the high-resolution feature map and the deep semantics are fused to obtain a fused feature.

3. The ultrasonic image analysis method based on multi-task learning according to claim 1, characterized in that: The spatial path in the shared encoding layer in the preset multi-task learning model generates a high-resolution feature map according to the ultrasound image data through a shallow network, including: The spatial path in the shared coding layer of the preset multi-task learning model decomposes the ultrasound image data into a plurality of frequency sub-bands corresponding to different frequencies through a shallow network; Each of the frequency subbands is convolved with the window attention features of the shared coding layer, and a multi-scale fused high-resolution feature map is generated through linear transformation.

4. The ultrasonic image analysis method based on multi-task learning according to any one of claim 3, characterized in that: Decomposing the ultrasound image data into a plurality of frequency sub-bands corresponding to different frequencies includes: Applying a first filter and a second filter to filter the ultrasound image data along a pixel row direction to obtain a high-frequency sub-band and a low-frequency sub-band, wherein a filtering frequency corresponding to the first filter is higher than a filtering frequency corresponding to the second filter; The high frequency sub-band is filtered by applying the first filter and the second filter to obtain the first frequency sub-band and the second frequency sub-band, and the low frequency sub-band is filtered by applying the first filter and the second filter to obtain the third frequency sub-band and the fourth frequency sub-band.

5. The ultrasonic image analysis method based on multi-task learning according to any one of claims 1 to 4, characterized in that: The context path in the shared encoding layer in the preset multi-task learning model is down-sampled to generate deep semantics according to the ultrasound image data, including: The context path in the shared encoding layer of the preset multi-task learning model is used to sample 1 / 8 scale features, 1 / 16 scale features, and 1 / 32 scale features corresponding to the ultrasound image data; Performing global pooling on the 1 / 32 scale features to obtain global features; The global features are combined with the 1 / 8 scale features and the 1 / 16 scale features to generate deep semantics.

6. The ultrasonic image analysis method based on multi-task learning according to claim 1, characterized in that: The preset multi-task learning model includes an image segmentation task for obtaining a target segmentation map and a feature point localization task for obtaining a target heat map; The preset multi-task learning model is obtained by the following steps: Using the first ultrasound segmentation training data to train the image segmentation task in the initial multi-task learning model to obtain an intermediate multi-task learning model; Using the second ultrasound segmentation training data and the training heat map data, training the intermediate multi-task learning model based on the first training strategy or the second training strategy to obtain a trained multi-task learning model; The first training strategy includes: In each round of training, the second ultrasound segmentation training data and the training heat map data are loaded simultaneously, and the gradients of the second ultrasound segmentation training data and the training heat map data are calculated respectively. Different weights are assigned to the image segmentation task and the feature point localization task according to the gradients. A weighted integrated loss function is obtained according to the weights corresponding to the image segmentation task and the feature point localization task, and feedback is provided to the training process based on the loss function. The second training strategy includes: Corresponding probabilities are preset for the image segmentation task and the feature point positioning task. In each round of training, one of the image segmentation task and the feature point positioning task is selected for training based on the corresponding probabilities preset for the image segmentation task and the feature point positioning task.

7. The ultrasonic image analysis method based on multi-task learning according to any one of claims 1 to 4 and 6, characterized in that: The ultrasound image data is ultrasound image data corresponding to the right ventricle of the heart, the target object is the myocardium, the target heat map is used to represent the feature points corresponding to the central myocardium of the ultrasound image data, and the target segmentation map is used to represent the probability of each pixel in the ultrasound image data corresponding to the right ventricle of the heart being identified as the myocardium.

8. An ultrasonic image analysis device based on multi-task learning, characterized in that: The device comprises: A data acquisition module, used to acquire ultrasound image data corresponding to the target object; a multi-task learning module, configured to generate a high-resolution feature map based on the ultrasound image data by using a spatial path in a shared encoding layer in a preset multi-task learning model through a shallow network; and further configured to generate a deep semantic layer based on the ultrasound image data by downsampling using a context path in a shared encoding layer in the preset multi-task learning model; A feature fusion module is used to fuse the high-resolution feature map and the deep semantics based on a preset feature fusion strategy to obtain a fused feature; a first decoding module, configured to upsample the fused features according to the deconvolution layer and the convolution block in the multi-task learning model to obtain a target heat map, wherein the target heat map is used to represent feature points corresponding to the target object in the ultrasound image data; a second decoding module, configured to upsample the fused features according to a feature extraction module corresponding to the shared coding layer in the multi-task learning model to obtain a target segmentation map, wherein the target segmentation map is used to represent a probability that each pixel in the ultrasound image data corresponding to the target object is identified as the target object; A result determination module is used to determine the image data corresponding to the target object from the ultrasound image data according to the target heat map and the target segmentation map.

9. An ultrasound image analysis system based on multi-task learning, characterized in that: The system includes: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the ultrasound image analysis method based on multi-task learning as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that The computer storage medium stores computer instructions, which, when called, are used to execute the ultrasound image analysis method based on multi-task learning as described in any one of claims 1 to 7.

Images