Myocardial region data processing method and device based on joint learning framework

Abstract

This invention relates to the field of image processing technology and discloses a method and apparatus for processing myocardial region data based on a joint learning framework. By implementing this invention, a two-stage optimization method is designed, which involves myocardial segmentation pre-training, optical flow estimation pre-training, and joint training. This improves the output performance of the final unsupervised optical flow estimation model on local myocardial regions. At the same time, the semantic information brought by myocardial segmentation can provide physiological shape constraints for the optical flow estimation task and achieve optical flow refinement on local myocardial regions. Meanwhile, optical flow estimation can add temporal constraints to the segmentation task and diffuse the segmentation target frame from the ED / ES frame to the entire cardiac cycle sequence through motion consistency.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and apparatus for processing myocardial region data based on a joint learning framework. Background Technology

[0002] In modern medicine, commonly used techniques for detecting and calculating myocardial motion status and strain indicators mainly include tissue Doppler imaging (TDI) and two-dimensional speckle tracking echocardiography (2D-STE). However, in practical applications...

[0003] Doppler effect measurements have certain limitations, as the results are affected by the angle between the direction of the sound beam and the direction of tissue movement.

[0004] The measurement results of two-dimensional speckle tracking are often directly related to image quality, and there are also significant differences in the algorithm implementation and parameter meanings of different manufacturers. There is no unified calculation standard, which to some extent reduces the data analysis efficiency of professional doctors.

[0005] In recent years, with the continuous development of artificial intelligence and deep learning technologies, network models based on optical flow estimation are faster and more robust than traditional algorithms, seemingly offering a lower-cost solution to the shortcomings of traditional methods. However, since the heart is not a simple rigid body and its motion is quite complex, manually annotating dense optical flow is almost impossible, and artificially synthesizing heart motion is difficult. This has resulted in slow progress in applying deep learning technology to myocardial motion analysis, and a mature solution is lacking. Summary of the Invention

[0006] This invention provides a method and apparatus for processing myocardial region data based on a joint learning framework, which can help improve the segmentation effect of myocardial segmentation and the optical flow precision of optical flow estimation tasks.

[0007] To address the aforementioned technical problems, the first aspect of this invention discloses a method for processing myocardial region data based on a joint learning framework, the method comprising:

[0008] Acquire cardiac image data for performing framework training, the cardiac image data including multiple frame images that have undergone image annotation processing, all of which include at least ED frame images and ES frame images;

[0009] Image processing operations are performed on the cardiac image data to obtain target data, which includes at least training set data for performing the framework training; the training set data includes multiple example data, each example of the sub-data is a binary data, and each binary data consists of a previous image frame and a corresponding subsequent image frame;

[0010] The training set data is input into a pre-constructed joint learning framework, and the joint learning framework is trained according to the training set data to obtain the framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, the training task for the joint learning framework is determined to be completed.

[0011] The joint learning framework includes a segmentation branch, an optical flow branch, and a loss calculation module. The segmentation branch is used to perform myocardial region segmentation and extraction for each sub-data instance. The optical flow branch is used to determine the motion optical flow information of all the sub-data instances on the myocardium. The loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch, and a joint loss function, wherein the joint loss function is used to connect the segmentation branch and the optical flow branch.

[0012] As an optional implementation, in the first aspect of the present invention, performing image processing operations on the cardiac image data to obtain target data includes:

[0013] Multiple target frame images are selected from the cardiac image data, wherein the target frame images are either the ED frame images or the ES frame images;

[0014] For each target frame image, the target frame image is used as the first frame image, and the subsequent frame image of the target frame image is used as the second frame image to construct an image pair corresponding to the target frame image, which is denoted as a binary data; the number of subsequent frames corresponding to the subsequent frame image of the target frame image is a preset number of frames;

[0015] Perform data partitioning on all the said binary data to obtain data partitioning results corresponding to all the said binary data. The data partitioning results include validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training.

[0016] The data segmentation results are determined as the target data.

[0017] As an optional implementation, in the first aspect of the present invention, before performing framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework, the method further includes:

[0018] For each set of binary data, determine the true value of the segmentation mask corresponding to the first frame image in the binary data; add the true value of the segmentation mask corresponding to the binary data to the binary data to obtain the triple data corresponding to the binary data.

[0019] All the triplet data are determined as new training set data, and the operation of performing framework training on the joint learning framework based on the training set data is triggered to obtain the framework training result corresponding to the joint learning framework is executed.

[0020] Wherein, when performing branch training on the segmentation branch, the branch training data input to the segmentation branch is the training set data including the tuple data;

[0021] When performing branch training on the optical flow branch, the branch training data input to the optical flow branch is the new training set data that includes the triplet data.

[0022] As an optional implementation, in the first aspect of the present invention, the step of performing framework training on the joint learning framework based on the training set data to obtain a framework training result corresponding to the joint learning framework includes:

[0023] When performing branch training on the segmentation branch, for the target binary data currently input to the segmentation branch, a target segmentation processing operation is performed on the target binary data to obtain a target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on a pre-built myocardial segmentation pre-training model and segmentation loss calculation based on a preset segmentation loss function; the target segmentation processing result includes the myocardial segmentation result corresponding to the myocardial segmentation processing and the segmentation loss calculation result corresponding to the segmentation loss calculation.

[0024] When performing branch training on the optical flow branch, for the target triplet data currently input to the optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain a target estimation result corresponding to the target triplet data; the target optical flow estimation processing operation includes optical flow estimation processing based on a pre-built optical flow estimation pre-trained model and optical flow loss calculation based on a preset optical flow loss function; the target estimation result includes the optical flow estimation result corresponding to the optical flow estimation processing and the optical flow loss calculation result corresponding to the optical flow loss calculation.

[0025] Based on a pre-set joint loss function, joint loss calculation is performed on the target segmentation result and the target estimation result to obtain the joint loss calculation result corresponding to the target segmentation result and the target estimation result;

[0026] The target segmentation result, the target estimation result, and the joint loss calculation result are determined as the framework training result corresponding to the joint learning framework;

[0027] Specifically, when the segmentation loss calculation result, the optical flow loss calculation result, and the joint loss calculation result indicate that the joint learning framework has reached convergence, it is determined that the joint learning framework has completed training.

[0028] As an optional implementation, in the first aspect of the present invention, the myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image.

[0029] The segmentation loss calculation result corresponding to the target binary data is obtained by performing the segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data using the preset segmentation loss function;

[0030] The optical flow estimation result corresponding to the target triplet data includes the forward optical flow result corresponding to the target triplet data;

[0031] The optical flow loss calculation result corresponding to the target triplet data is obtained by performing the optical flow loss calculation on the forward optical flow result, the segmentation mask ground value corresponding to the target triplet data, and the first frame image in the target triplet data using the preset optical flow loss function;

[0032] Before the joint learning framework completes its training, the segmentation loss calculation results, the optical flow loss calculation results, and the joint loss calculation results are used to perform parameter optimization and updates on the branch parameters corresponding to the optical flow branch and the segmentation branch, respectively.

[0033] As an optional implementation, in the first aspect of the invention, before inputting the training set data into the pre-constructed joint learning framework, the method further includes:

[0034] Data preprocessing is performed on the training set data to obtain data preprocessing results corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing, and data augmentation, and the data augmentation includes at least random brightness transformation and / or random affine transformation;

[0035] The training set data is updated based on the data preprocessing results, and the operation of inputting the training set data into the pre-built joint learning framework is triggered.

[0036] Furthermore, the specific construction method of the pre-built joint learning framework is as follows:

[0037] A target training model that meets the training conditions is selected, including a myocardial segmentation output model and an optical flow estimation output model; all target training models have been preliminarily trained.

[0038] Determine the training parameters and their values ​​for the target training model; the training parameters include at least one of the following: total number of training iterations for the target training model, batch size, and optimizer parameters; the optimizer parameters include the learning rate and the weight decay rate.

[0039] Determine the target loss function for the target training model. The target loss function includes a segmentation loss function corresponding to the myocardial segmentation output model, an optical flow loss function corresponding to the optical flow estimation output model, and a joint loss function for connecting the myocardial segmentation output model and the optical flow estimation output model.

[0040] A joint learning framework is constructed based on the target training model, the training parameters and their values, and the target loss function.

[0041] As an optional implementation, in the first aspect of the present invention, the calculation formula corresponding to the segmentation loss function is as follows:

[0042] L seg =α*L WD +β*L CE

[0043] Where α and β are weighting coefficients; L seg Refers to the segmentation loss function; L WD Refers to the Weighted Dice Loss function; L CE Refers to the CrossEntropy Loss function;

[0044] The calculation formula for the optical flow loss function is as follows:

[0045]

[0046] Where p represents an image pixel, i and j represent pixel coordinates, d represents the partial derivatives in the x and y directions, Ψ is the robustness penalty function, q, ε, and α are the adjustable parameters of the optical flow loss function, and λ is the weighting coefficient.

[0047] The calculation formula for the joint loss function is as follows:

[0048] L=α*L flow +β*L seg +γ*Ljoint

[0049] Where α, β, and γ are all weighting coefficients; L is the final joint loss function; L joint This is the initial joint loss function;

[0050] The initial joint loss function L joint The calculation formula is as follows:

[0051]

[0052] Where, parameter l represents the subscript for different categories, i indicates the i-th pixel, and N is the total number of pixels. This indicates that the predicted segmentation mask P2 is used in the predicted optical flow F. 1→2 Mask of deformation results under action;

[0053] in:

[0054] and These represent the label value and predicted value of pixel i in category l, respectively.

[0055] A second aspect of the present invention discloses a myocardial region data processing device based on a joint learning framework, the device comprising:

[0056] The acquisition module is used to acquire cardiac image data for performing framework training. The cardiac image data includes multiple frame images that have undergone image annotation processing, and all the frame images include at least ED frame images and ES frame images.

[0057] An image processing module is used to perform image processing operations on the cardiac image data to obtain target data. The target data includes at least training set data for performing the training of the framework. The training set data includes multiple example data, each example of the sub-data is a binary data, and each binary data consists of a previous image frame and a corresponding subsequent image frame.

[0058] A data input module is used to input the training set data into a pre-built joint learning framework;

[0059] The framework training module is used to perform framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, the training task for the joint learning framework is determined to be completed.

[0060] The joint learning framework includes a segmentation branch, an optical flow branch, and a loss calculation module. The segmentation branch is used to perform myocardial region segmentation and extraction for each sub-data instance. The optical flow branch is used to determine the motion optical flow information of all the sub-data instances on the myocardium. The loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch, and a joint loss function, wherein the joint loss function is used to connect the segmentation branch and the optical flow branch.

[0061] As an optional implementation, in a second aspect of the present invention, the image processing module performs image processing operations on the cardiac image data to obtain the target data in the following specific ways:

[0062] Multiple target frame images are selected from the cardiac image data, wherein the target frame images are either the ED frame images or the ES frame images;

[0063] For each target frame image, the target frame image is used as the first frame image, and the subsequent frame image of the target frame image is used as the second frame image to construct an image pair corresponding to the target frame image, which is denoted as a binary data; the number of subsequent frames corresponding to the subsequent frame image of the target frame image is a preset number of frames;

[0064] Perform data partitioning on all the said binary data to obtain data partitioning results corresponding to all the said binary data. The data partitioning results include validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training.

[0065] The data segmentation results are determined as the target data.

[0066] As an optional implementation, in a second aspect of the invention, the apparatus further includes:

[0067] The determination module is configured to, before the framework training module performs framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework, determine the ground truth value of the segmentation mask corresponding to the first frame image in each pair of data; and add the ground truth value of the segmentation mask corresponding to the pair of data to the pair of data to obtain the triplet data corresponding to the pair of data.

[0068] The determining module is further configured to determine all the triplet data as new training set data, and trigger the framework training module to perform the operation of performing framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework.

[0069] Wherein, when performing branch training on the segmentation branch, the branch training data input to the segmentation branch is the training set data including the tuple data;

[0070] When performing branch training on the optical flow branch, the branch training data input to the optical flow branch is the new training set data that includes the triplet data.

[0071] As an optional implementation, in a second aspect of the present invention, the method by which the framework training module performs framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework specifically includes:

[0072] When performing branch training on the segmentation branch, for the target binary data currently input to the segmentation branch, a target segmentation processing operation is performed on the target binary data to obtain a target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on a pre-built myocardial segmentation pre-training model and segmentation loss calculation based on a preset segmentation loss function; the target segmentation processing result includes the myocardial segmentation result corresponding to the myocardial segmentation processing and the segmentation loss calculation result corresponding to the segmentation loss calculation.

[0073] When performing branch training on the optical flow branch, for the target triplet data currently input to the optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain a target estimation result corresponding to the target triplet data; the target optical flow estimation processing operation includes optical flow estimation processing based on a pre-built optical flow estimation pre-trained model and optical flow loss calculation based on a preset optical flow loss function; the target estimation result includes the optical flow estimation result corresponding to the optical flow estimation processing and the optical flow loss calculation result corresponding to the optical flow loss calculation.

[0074] Based on a pre-set joint loss function, joint loss calculation is performed on the target segmentation result and the target estimation result to obtain the joint loss calculation result corresponding to the target segmentation result and the target estimation result;

[0075] The target segmentation result, the target estimation result, and the joint loss calculation result are determined as the framework training result corresponding to the joint learning framework;

[0076] Specifically, when the segmentation loss calculation result, the optical flow loss calculation result, and the joint loss calculation result indicate that the joint learning framework has reached convergence, it is determined that the joint learning framework has completed training.

[0077] As an optional implementation, in a second aspect of the present invention, the myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image.

[0078] The segmentation loss calculation result corresponding to the target binary data is obtained by performing the segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data using the preset segmentation loss function;

[0079] The optical flow estimation result corresponding to the target triplet data includes the forward optical flow result corresponding to the target triplet data;

[0080] The optical flow loss calculation result corresponding to the target triplet data is obtained by performing the optical flow loss calculation on the forward optical flow result, the segmentation mask ground value corresponding to the target triplet data, and the first frame image in the target triplet data using the preset optical flow loss function;

[0081] Before the joint learning framework completes its training, the segmentation loss calculation results, the optical flow loss calculation results, and the joint loss calculation results are used to perform parameter optimization and updates on the branch parameters corresponding to the optical flow branch and the segmentation branch, respectively.

[0082] As an optional implementation, in a second aspect of the invention, the apparatus further includes:

[0083] A data preprocessing module is used to perform data preprocessing on the training set data before the data input module inputs the training set data into the pre-constructed joint learning framework, to obtain a data preprocessing result corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing, and data augmentation, and the data augmentation includes at least random brightness transformation and / or random affine transformation;

[0084] The update module is used to update the training set data according to the data preprocessing results and trigger the execution of the corresponding operation of inputting the training set data into the pre-built joint learning framework;

[0085] Furthermore, the specific construction method of the pre-built joint learning framework is as follows:

[0086] A target training model that meets the training conditions is selected, including a myocardial segmentation output model and an optical flow estimation output model; all target training models have been preliminarily trained.

[0087] Determine the training parameters and their values ​​for the target training model; the training parameters include at least one of the following: total number of training iterations for the target training model, batch size, and optimizer parameters; the optimizer parameters include the learning rate and the weight decay rate.

[0088] Determine the target loss function for the target training model. The target loss function includes a segmentation loss function corresponding to the myocardial segmentation output model, an optical flow loss function corresponding to the optical flow estimation output model, and a joint loss function for connecting the myocardial segmentation output model and the optical flow estimation output model.

[0089] A joint learning framework is constructed based on the target training model, the training parameters and their values, and the target loss function.

[0090] As an optional implementation, in the second aspect of the present invention, the calculation formula corresponding to the segmentation loss function is as follows:

[0091] L seg =α*L WD +β*L CE

[0092] Where α and β are weighting coefficients; L seg Refers to the segmentation loss function; L WD Refers to the Weighted Dice Loss function; L CE Refers to the CrossEntropy Loss function;

[0093] The calculation formula for the optical flow loss function is as follows:

[0094]

[0095] Where p represents an image pixel, i and j represent pixel coordinates, d represents the partial derivatives in the x and y directions, Ψ is the robustness penalty function, q, ε, and α are the adjustable parameters of the optical flow loss function, and λ is the weighting coefficient.

[0096] The calculation formula for the joint loss function is as follows:

[0097] L=α*L flow +β*L seg +γ*L joint

[0098] Where α, β, and γ are all weighting coefficients; L is the final joint loss function; L joint This is the initial joint loss function;

[0099] The initial joint loss function L jointThe calculation formula is as follows:

[0100]

[0101] Where, parameter l represents the subscript for different categories, i indicates the i-th pixel, and N is the total number of pixels. This indicates that the predicted segmentation mask P2 is used in the predicted optical flow F. 1→2 Mask of deformation results under action;

[0102] in:

[0103] and These represent the label value and predicted value of pixel i in category l, respectively.

[0104] A third aspect of the present invention discloses another myocardial region data processing device based on a joint learning framework, the device comprising:

[0105] Memory containing executable program code;

[0106] A processor coupled to the memory;

[0107] The processor calls the executable program code stored in the memory to execute the myocardial region data processing method based on the joint learning framework disclosed in the first aspect of the present invention.

[0108] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the myocardial region data processing method based on a joint learning framework disclosed in the first aspect of the present invention.

[0109] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0110] This invention provides a method for processing myocardial region data based on a joint learning framework. The method includes: acquiring cardiac image data for framework training, the cardiac image data comprising multiple frame images after image annotation processing, all frame images including at least ED frame images and ES frame images; performing image processing operations on the cardiac image data to obtain target data, the target data including at least training set data for framework training; the training set data comprising multiple example data, each example data being a binary tuple data, each binary tuple data consisting of a previous image frame and a corresponding subsequent image frame; and inputting the training set data into a pre-constructed joint learning framework. The training process involves training the joint learning framework using the training set data to obtain the training results corresponding to the joint learning framework. When the training results indicate that the joint learning framework has converged, the training task for the joint learning framework is considered complete. The joint learning framework includes a segmentation branch, an optical flow branch, and a loss calculation module. The segmentation branch is used to perform myocardial region segmentation and extraction for each example data. The optical flow branch is used to determine the motion optical flow information of all sub-data on the myocardium. The loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch, and a joint loss function. The joint loss function is used to connect the segmentation branch and the optical flow branch. As can be seen, by implementing this invention, preprocessing cardiac image data before inputting training set data into the joint learning framework makes the training data input into the joint learning framework more accurate, which is beneficial to improving the training efficiency of the joint learning framework. In addition, based on the constructed joint learning framework, it is possible to achieve simultaneous training and optimization of two models: a myocardial segmentation pre-training model (corresponding to the segmentation branch) and an optical flow estimation pre-training model (corresponding to the optical flow branch). Furthermore, the joint learning framework also sets a joint loss function, which enables the semantic information brought by the myocardial segmentation pre-training model to provide physiological shape constraints for the optical flow estimation task performed by the optical flow estimation pre-training model, thereby achieving optical flow refinement in local myocardial regions. This is beneficial to improving the precision of the output optical flow corresponding to the optical flow branch and reducing the graininess and fragmentation of the output optical flow. At the same time, the optical flow estimation implemented by the optical flow branch can add temporal constraints to the segmentation task of the segmentation branch, and diffuse the segmentation target frame processed by the segmentation branch to the entire cardiac cycle sequence through motion consistency. This means that the labeled data input to the segmentation branch does not need to be labeled with the entire myocardial sequence, thereby reducing the workload of myocardial labeling and image processing costs. Attached Figure Description

[0111] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0112] Figure 1 This is a flowchart illustrating a myocardial region data processing method based on a joint learning framework disclosed in an embodiment of the present invention.

[0113] Figure 2 This is a flowchart illustrating another method for processing myocardial region data based on a joint learning framework disclosed in an embodiment of the present invention.

[0114] Figure 3 This is a schematic diagram of the structure of a myocardial region data processing device based on a joint learning framework disclosed in an embodiment of the present invention;

[0115] Figure 4 This is a schematic diagram of another myocardial region data processing device based on a joint learning framework disclosed in an embodiment of the present invention;

[0116] Figure 5 This is a schematic diagram of the structure of another myocardial region data processing device based on a joint learning framework disclosed in an embodiment of the present invention;

[0117] Figure 6 This is a schematic diagram of a collaborative learning framework disclosed in an embodiment of the present invention. Detailed Implementation

[0118] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0119] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0120] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0121] This invention discloses a method and apparatus for myocardial region data processing based on a joint learning framework. Preprocessing of cardiac image data before inputting training set data into the joint learning framework makes the training data more accurate, improving the training efficiency of the framework. Furthermore, based on the constructed joint learning framework, simultaneous training and optimization of two models—a myocardial segmentation pre-training model (corresponding to the segmentation branch) and an optical flow estimation pre-training model (corresponding to the optical flow branch)—can be achieved. Further, the joint learning framework sets a joint loss function, enabling the semantic information from the myocardial segmentation pre-training model to provide physiological shape constraints for the optical flow estimation task performed by the optical flow estimation pre-training model. This achieves optical flow refinement in local myocardial regions, improving the precision of the output optical flow of the optical flow branch and reducing the graininess and fragmentation of the output optical flow. Simultaneously, the optical flow estimation achieved by the optical flow branch can add temporal constraints to the segmentation task of the segmentation branch and, through motion consistency, diffuse the segmentation target frame processed by the segmentation branch to the entire cardiac cycle sequence. This eliminates the need for full-sequence myocardial annotation of the input data to the segmentation branch, reducing the workload of myocardial annotation and image processing costs. The following sections will provide detailed explanations.

[0122] Example 1

[0123] Please see Figure 1 , Figure 1 This is a flowchart illustrating a myocardial region data processing method based on a joint learning framework disclosed in an embodiment of the present invention. Figure 1 The described myocardial region data processing method based on a joint learning framework can be applied to myocardial region data processing devices based on a joint learning framework, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, this myocardial region data processing method based on a joint learning framework may include the following operations:

[0124] 101. Obtain cardiac image data for performing framework training. The cardiac image data includes multiple frame images that have undergone image annotation processing. All frame images include at least ED frame images and ES frame images.

[0125] In this embodiment of the invention, the cardiac image data comes from the publicly available CAMUS dataset and a clinically collected dataset from a collaborating hospital. The CAMUS dataset contains 2D apical four-chamber (4CH) and two-chamber (2CH) view sequences obtained from 500 patients. Approximately 19% of these patients had poor image quality, and half of them had a left ventricular ejection fraction (LVEF) below 45%, indicating a pathological risk. Each patient's sample data consists of a complete 2CH / 4CH cardiac cycle sequence, ultrasound images from both the ED and ES phases along with their cardiac structural annotations, and the patient's acquisition information. The original input images are provided in raw / mhd file format. The cardiac structural annotation data was manually obtained by professional physicians using a semi-automated annotation tool at the ED and ES phases. In the mask classification, 0, 1, 2, and 3 represent pixels located in the background, left ventricular cavity, left ventricular myocardium, and left atrial cavity, respectively.

[0126] 102. Perform image processing operations on the cardiac image data to obtain target data, which includes at least the training set data used to perform framework training.

[0127] In this embodiment of the invention, the training set data includes multiple example data, each example data is a binary data, and each binary data consists of a previous image frame and the corresponding subsequent image frame.

[0128] 103. Input the training set data into the pre-built joint learning framework, and perform framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework.

[0129] 104. When the training results indicate that the joint learning framework has converged, the training task for the joint learning framework is considered complete.

[0130] In this embodiment of the invention, the joint learning framework includes a segmentation branch, an optical flow branch, and a loss calculation module; the segmentation branch is used to perform myocardial region segmentation and extraction for each example data; the optical flow branch is used to determine the motion optical flow information of all sub-data on the myocardium; the loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch, and a joint loss function, the joint loss function being used to connect the segmentation branch and the optical flow branch.

[0131] It is evident that implementation Figure 1The described myocardial region data processing method based on a joint learning framework preprocesses cardiac image data before inputting the training set data into the framework, making the training data more accurate and improving the training efficiency of the framework. Furthermore, based on the constructed joint learning framework, it enables simultaneous training and optimization of two models: a myocardial segmentation pre-training model (corresponding to the segmentation branch) and an optical flow estimation pre-training model (corresponding to the optical flow branch). Further, the joint learning framework sets a joint loss function, allowing the semantic information from the myocardial segmentation pre-training model to provide physiological shape constraints for the optical flow estimation task performed by the optical flow estimation pre-training model. This achieves optical flow refinement in local myocardial regions, improving the precision of the output optical flow of the optical flow branch and reducing the graininess and fragmentation of the output optical flow. Simultaneously, the optical flow estimation performed by the optical flow branch adds temporal constraints to the segmentation task of the segmentation branch and diffuses the segmentation target frames processed by the segmentation branch to the entire cardiac cycle sequence through motion consistency. This eliminates the need for full-sequence myocardial annotation of the input data for the segmentation branch, thereby reducing the workload of myocardial annotation and image processing costs.

[0132] In an optional embodiment, before step 103 above, which inputs the training set data into the pre-built joint learning framework, the method further includes:

[0133] Perform data preprocessing on the training set data to obtain the data preprocessing results corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing, and data augmentation, and the data augmentation includes at least random brightness transformation and / or random affine transformation;

[0134] Update the training set data based on the data preprocessing results, and trigger the corresponding operation of inputting the training set data into the pre-built joint learning framework as described above.

[0135] In this optional embodiment, the preset image size is typically set to 256*256 pixels; the normalization process is used to normalize the image to a value between 0 and 1; and the data augmentation operation is used to improve the model's generalization ability. Furthermore, the pre-built joint learning framework is constructed as follows:

[0136] Target training models that meet the training conditions were selected, including myocardial segmentation output models and optical flow estimation output models; all target training models have been preliminarily trained.

[0137] Determine the training parameters and their values ​​for the target training model; the training parameters include at least one of the following: total number of training iterations for the target training model, batch size, and optimizer parameters; the optimizer parameters include the learning rate and weight decay rate.

[0138] Determine the target loss function for the target training model. The target loss function includes the segmentation loss function corresponding to the myocardial segmentation output model, the optical flow loss function corresponding to the optical flow estimation output model, and the joint loss function used to connect the myocardial segmentation output model and the optical flow estimation output model.

[0139] A joint learning framework is constructed based on the target training model, training parameters and their values, and the target loss function.

[0140] In this optional embodiment, during the joint training phase, the joint learning framework simultaneously optimizes the optical flow model (optical flow estimation pre-trained model) and the segmentation model (myocardial segmentation pre-trained model) in one iteration. The total number of training iterations (epochs) is set to 50, and the batch size is set to 10. In actual experiments, the Adam optimizer can be selected to perform gradient optimization and backpropagation on the two branch networks, with its learning rate fixed at 0.0001 and weight decay rate fixed at 0.0001. Other network configurations remain the same as those of each branch during its pre-training phase.

[0141] In this optional embodiment, the calculation formula for the segmentation loss function is as follows:

[0142] L seg =α*L WD +β*L CE

[0143] Where α and β are weighting coefficients, in actual experiments, α = 1.0 and β = 0.1 can be set to maintain L WD The dominant role of functions in model optimization, where L seg Refers to the segmentation loss function; L WD Refers to the Weighted Dice Loss function; L CE Refers to the CrossEntropy Loss function;

[0144] Furthermore, the calculation formula for the Weighted Dice Loss function is as follows:

[0145]

[0146] Wherein, parameter l represents the subscript for different categories, and Let N represent the label value and predicted value of pixel i in category l, respectively, and N be the total number of pixels.

[0147]

[0148] The calculation formula for the CrossEntropy Loss function is as follows:

[0149]

[0150] In this optional embodiment, the calculation formula for the optical flow loss function is as follows:

[0151]

[0152] Where p represents an image pixel, i and j represent pixel coordinates, d represents the partial derivatives in the x and y directions, Ψ is the robustness penalty function, q, ε, and α are the adjustable parameters of the optical flow loss function, and in practical applications, q, ε, and α can be set to 0.4, 0.01, and 10, respectively; λ is the weighting coefficient, which can be set to 0.1.

[0153] In this optional embodiment, the joint loss L joint Based on motion consistency constraints, the predicted segmentation mask P2 for unlabeled image frame I2 is used to compute the forward predicted optical flow F. 1→2 The difference between the deformed result mask and the labeled mask M1 is used to connect the optical flow branch and the segmentation branch and exchange complementary information, optimizing both tasks simultaneously during training and spreading the segmented target frame to the entire cardiac cycle sequence. The difference measure function for the joint loss adopts the Weighted Dice Loss proposed in the myocardial segmentation pre-training part. The final calculation formula for the joint loss function is as follows:

[0154] L=α*L flow +β*L seg +γ*L joint

[0155] Where α, β, and γ are weighting coefficients, and in the actual experiment, α, β, and γ are set to 1.0, 1.0, and 0.1, respectively; L is the final joint loss function; L joint This is the initial joint loss function;

[0156] The initial joint loss function L joint The calculation formula is as follows:

[0157]

[0158] Where, parameter l represents the subscript for different categories, i indicates the i-th pixel, and N is the total number of pixels. This indicates that the predicted segmentation mask P2 is used in the predicted optical flow F. 1→2 Mask of deformation results under action;

[0159] in:

[0160] and These represent the label value and predicted value of pixel i in category l, respectively.

[0161] In this optional embodiment,

[0162] As can be seen, in this optional embodiment, a data preprocessing procedure for the training set data is set before inputting the training set data into the joint learning framework. Through image scaling and normalization, the consistency of the training set data is improved, which is conducive to improving the training efficiency of subsequent framework training (including model training). This data augmentation is conducive to improving the generalization of the model. At the same time, for the joint learning framework, the parameter values ​​of relevant training parameters can be accurately set according to the actual training requirements of the framework, which improves the accuracy of the framework construction of the joint learning framework.

[0163] Example 2

[0164] Please see Figure 2 , Figure 2 This is a flowchart illustrating another method for myocardial region data processing based on a joint learning framework disclosed in an embodiment of the present invention. Figure 2 The described myocardial region data processing method based on a joint learning framework can be applied to myocardial region data processing devices based on a joint learning framework, and the embodiments of the present invention are not limited thereto. Figure 2 As shown, this myocardial region data processing method based on a joint learning framework may include the following operations:

[0165] 201. Obtain cardiac image data for performing framework training. The cardiac image data includes multiple frame images that have undergone image annotation processing. All frame images include at least ED frame images and ES frame images.

[0166] 202. Select multiple target frame images from the cardiac image data. The target frame images are ED frame images or ES frame images.

[0167] 203. For each target frame image, take the target frame image as the first frame image and the subsequent frame image of the target frame image as the second frame image to construct an image pair corresponding to the target frame image, which is denoted as a binary data.

[0168] In this embodiment of the invention, the number of subsequent frames corresponding to the target frame image is a preset number of frames; the preset number of frames can be 1 frame, 2 frames, ..., n frames; further, when actually acquiring multiple target frame images, the data can be arranged and stored according to the cardiac cycle as a basis. For example, the multiple target frame images include multiple sets of frame images, each set of frame images corresponding to one cardiac cycle. Further, the selected target frame image is an ED frame image or an ES frame image, serving as the first frame image, and the subsequent frame images corresponding to the target frame image are frame images belonging to the same cardiac cycle as the target frame image.

[0169] 204. Perform data partitioning on all pairs of data to obtain the data partitioning results corresponding to all pairs of data.

[0170] In this embodiment of the invention, the data partitioning result includes validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training.

[0171] In this embodiment of the invention, when multiple sets of binary data are actually determined, the total number of binary data sets can be 17739, of which 9223 sets of binary data sets have a good image quality rating, 6448 sets of binary data sets have a medium image quality rating, and 2068 sets of binary data sets have a poor image quality rating. In order to maintain consistent performance on the validation set and the test set, the experiments in this section fix both the validation set and the test set at 1500 sets, and set the ratio of good quality: medium quality: poor quality in the data composition to 8:5:2. The remaining data sets are used as the training set for training.

[0172] 205. The data partitioning results shall be determined as the target data, which shall include at least the training set data used to perform framework training.

[0173] 206. Input the training set data into the pre-built joint learning framework, and perform framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework.

[0174] 207. When the training results indicate that the joint learning framework has converged, the training task for the joint learning framework is considered complete.

[0175] For further descriptions of steps 201-202 and 206-207 in this embodiment of the invention, please refer to the other specific descriptions of steps 101-102 and 103-104 in Embodiment 1. These descriptions will not be repeated in this embodiment of the invention.

[0176] It is evident that implementation Figure 2The described myocardial region data processing method based on a joint learning framework, when preparing training set data, uses ED frame images / ES frame images as the first frame image, and the subsequent frames of the ED frame images / ES frame images as the second frame image, and combines and constructs image pairs to obtain training set data according to the set data partitioning requirements, thereby improving the reliability and accuracy of training set data preparation.

[0177] In an optional embodiment, before performing framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework, the method further includes:

[0178] For each pair of data, determine the true value of the segmentation mask corresponding to the first frame image in the pair of data; add the true value of the segmentation mask corresponding to the pair of data to the pair of data to obtain the triplet data corresponding to the pair of data.

[0179] All triplet data are identified as new training set data, and the above-mentioned operation of training the joint learning framework based on the training set data is triggered to obtain the corresponding framework training result of the joint learning framework is executed.

[0180] When performing branch training on the split branch, the branch training data input to the split branch is the training set data including binary data;

[0181] When performing branch training on the optical flow branch, the branch training data input to the optical flow branch is a new training set data that includes triplet data.

[0182] As can be seen, in this optional embodiment, the input training data can be adjusted as needed for different training branches and different training tasks. For the segmentation branch, pre-prepared training set data including binary data is sufficient, while for the optical flow branch, training set data including triple data is required. The triple data includes additional segmentation mask ground truth values ​​to constrain unsupervised optical flow estimation, which helps improve the optical flow estimation effect of this branch. Specifically, it improves the precision of optical flow, eliminates the optical flow fragmentation generated by the pre-trained model during the pre-training stage, and improves the continuity of motion tracking. This precise input training set data also helps improve the training accuracy of the joint learning framework.

[0183] In another optional embodiment, the method of performing framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework specifically includes:

[0184] When performing branch training on the segmentation branch, for the target binary data of the current input segmentation branch, the target segmentation processing operation is performed on the target binary data to obtain the target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on the pre-built myocardial segmentation pre-training model and segmentation loss calculation based on the preset segmentation loss function; the target segmentation processing result includes the myocardial segmentation result corresponding to the myocardial segmentation processing and the segmentation loss calculation result corresponding to the segmentation loss calculation.

[0185] When performing branch training on the optical flow branch, for the target triplet data of the current input optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain the target estimation result corresponding to the target triplet data. The target optical flow estimation processing operation includes optical flow estimation processing based on a pre-built optical flow estimation pre-trained model and optical flow loss calculation based on a preset optical flow loss function. The target estimation result includes the optical flow estimation result corresponding to the optical flow estimation processing and the optical flow loss calculation result corresponding to the optical flow loss calculation.

[0186] Based on the pre-set joint loss function, the joint loss calculation is performed on the target segmentation result and the target estimation result to obtain the joint loss calculation result corresponding to the target segmentation result and the target estimation result;

[0187] The target segmentation results, target estimation results, and joint loss calculation results are determined as the framework training results corresponding to the joint learning framework;

[0188] Specifically, when the calculation results of segmentation loss, optical flow loss, and joint loss indicate that the joint learning framework has converged, it is determined that the joint learning framework has completed training.

[0189] In this optional embodiment, the myocardial segmentation pre-training model is specifically a model improved and adjusted based on the traditional U-Net network; the improvements of this myocardial segmentation pre-training model are as follows:

[0190] During downsampling, the encoder extracts features from the input data through four downsampling passes, doubling the number of channels in the feature map with each downsampling pass. Except for the last layer, each downsampling block contains two 3x3 convolutional layers and a 2x2 max pooling layer. Each convolutional layer is batch normalized and feature activated by adding InstanceNorm and LeakyReLU. The initial number of channels is set to 48, which makes our network wider, allowing it to learn more details.

[0191] During the upsampling process, the decoder recovers the feature map size through four upsampling operations. Each subsequent upsampling operation reduces the number of feature channels to half of the original number, and then concatenates it with the feature map of the shrunk path along the channel dimension at the corresponding layer to fuse contextual information. In the final layer of the network, a convolutional layer of size 1 maps the output to the number of categories to be predicted. In this paper, the number of predicted categories is set to 2, representing background pixels and myocardial pixels, respectively.

[0192] In this optional embodiment, the optical flow estimation pre-trained model can specifically be the EchoFlowNet pattern. It should be noted that both the myocardial segmentation pre-trained model and the optical flow estimation pre-trained model used in this joint learning framework are pre-trained models, and the joint learning framework further optimizes these two pre-trained models.

[0193] In this optional embodiment, it should be noted that the myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image.

[0194] The segmentation loss calculation result corresponding to the target binary data is obtained by performing segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data using a preset segmentation loss function;

[0195] The optical flow estimation results corresponding to the target triplet data include the forward optical flow results corresponding to the target triplet data;

[0196] The optical flow loss calculation result corresponding to the target triplet data is obtained by performing optical flow loss calculation on the forward optical flow result, the true value of the segmentation mask corresponding to the target triplet data, and the first frame image in the target triplet data through a preset optical flow loss function.

[0197] Before the joint learning framework is determined and training is completed, the segmentation loss calculation results, optical flow loss calculation results, and joint loss calculation results are used to perform parameter optimization and update on the branch parameters corresponding to the optical flow branch and the segmentation branch, respectively.

[0198] In this optional embodiment, please refer to Figure 6 , Figure 6 This is a schematic diagram of a joint learning framework disclosed in an embodiment of the present invention. Specifically, during the joint training process, two branch networks are optimized simultaneously, corresponding to the segmentation network and the optical flow network; and, for each iteration of training of the joint learning framework, such as... Figure 6As shown, the training set data input to the current frame training is denoted as the previous frame image (first frame image) I1 and the next frame image (second frame image) I2; the binary data is (I1, I2); the triple data also includes the ground truth value of the segmentation mask corresponding to the previous frame image I1, denoted as M1, and the corresponding triple data is (I1, I2, M1). Correspondingly, the first preset segmentation mask corresponding to the previous frame image I1 is denoted as P1, and the first preset segmentation mask corresponding to the next frame image I2 is denoted as P2. The forward optical flow result corresponding to the target triple data is denoted as F. 1→2 .

[0199] As can be seen, in this optional embodiment, during the training phase of the joint learning framework, the input training data, in addition to the image pairs / tuples I1 and I2 composed of the previous and next frame images, also includes the ground truth value M1 of the segmentation mask corresponding to the previous frame image I1, which is used for auxiliary calculation of the loss modules (optical flow loss, joint loss, and segmentation loss). Specifically, in the optical flow branch, thanks to the constraint of the ground truth value M1 of the segmentation mask, the unsupervised optical flow loss L... flow The optimization target shifted from the global to the local myocardial region.

[0200] In this optional embodiment, the unsupervised optical flow loss L flow It mainly consists of two parts: photometric loss and smoothing loss. The traditional photometric loss, combined with the mask M1, calculates the results only on the myocardial region to ensure the accuracy of the local output optical flow. At the same time, a first-order smoothing loss is added on this basis to achieve smoothing and refinement of local myocardial motion.

[0201] In this optional embodiment, the segmentation branch is built based on a pre-trained myocardial segmentation model, and outputs predicted segmentation masks P1 and P2 on images I1 and I2 respectively, with a segmentation loss L. seg Supervised computation is then performed based on the predicted mask P1 and the corresponding ground truth mask M1 to ensure the effectiveness of the segmentation branch network's output. This joint learning framework not only improves the smoothness and continuity of motion but also further reduces losses in local myocardial regions. P2, as the predicted segmentation mask for non-ED / ES sequence frames, does not have a corresponding ground truth label during training. However, due to the motion consistency between the image and the mask, P2 contributes to the prediction of optical flow F. 1→2 The deformation result mask under the action should be consistent with M1; based on this constraint, a joint loss is proposed to connect the optical flow branch and the segmentation branch, and the segmentation target frame is spread to the entire cardiac cycle sequence by exchanging additional complementary information during training.

[0202] Example 3

[0203] Please see Figure 3 , Figure 3This is a schematic diagram of a myocardial region data processing device based on a federated learning framework, as disclosed in an embodiment of the present invention. The myocardial region data processing device based on the federated learning framework can be a myocardial region data processing terminal, device, system, or server based on a federated learning framework. The server can be a local server, a remote server, or a cloud server (also known as a cloud server). When the server is not a cloud server, it can communicate with the cloud server; this embodiment of the present invention does not impose any limitations. Figure 3 As shown, the myocardial region data processing device based on the joint learning framework may include an acquisition module 301, an image processing module 302, a data input module 303, and a framework training module 304, wherein:

[0204] The acquisition module 301 is used to acquire cardiac image data for performing framework training. The cardiac image data includes multiple frame images that have undergone image annotation processing. All frame images include at least ED frame images and ES frame images.

[0205] Image processing module 302 is used to perform image processing operations on cardiac image data to obtain target data. The target data includes at least training set data for performing framework training. The training set data includes multiple example data, each example data is a pair of data, and each pair of data consists of a previous image frame and the corresponding subsequent image frame.

[0206] The data input module 303 is used to input training set data into a pre-built joint learning framework;

[0207] The framework training module 304 is used to perform framework training on the joint learning framework based on the training set data, and obtain the framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, the training task for the joint learning framework is determined to be completed.

[0208] The joint learning framework includes a segmentation branch, an optical flow branch, and a loss calculation module. The segmentation branch is used to perform myocardial region segmentation and extraction for each example data. The optical flow branch is used to determine the motion optical flow information of all sub-data on the myocardium. The loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch, and a joint loss function. The joint loss function is used to connect the segmentation branch and the optical flow branch.

[0209] It is evident that implementation Figure 3The described myocardial region data processing device based on a joint learning framework preprocesses cardiac image data before inputting training data into the framework, making the training data more accurate and improving the training efficiency of the framework. Furthermore, based on the constructed joint learning framework, it enables simultaneous training and optimization of two models: a myocardial segmentation pre-training model (corresponding to the segmentation branch) and an optical flow estimation pre-training model (corresponding to the optical flow branch). Further, the joint learning framework sets a joint loss function, allowing the semantic information from the myocardial segmentation pre-training model to provide physiological shape constraints for the optical flow estimation task performed by the optical flow estimation pre-training model. This achieves optical flow refinement in local myocardial regions, improving the precision of the output optical flow of the optical flow branch and reducing the graininess and fragmentation of the output optical flow. Simultaneously, the optical flow estimation performed by the optical flow branch adds temporal constraints to the segmentation task of the segmentation branch and diffuses the segmentation target frames processed by the segmentation branch to the entire cardiac cycle sequence through motion consistency. This eliminates the need for full-sequence myocardial annotation of the input data for the segmentation branch, reducing the workload of myocardial annotation and image processing costs.

[0210] In an optional embodiment, the image processing module 302 performs image processing operations on the cardiac image data to obtain the target data in the following specific ways:

[0211] Multiple target frame images are selected from cardiac image data. The target frame images are either ED frame images or ES frame images.

[0212] For each target frame image, the target frame image is used as the first frame image, and the subsequent frame image of the target frame image is used as the second frame image to construct an image pair corresponding to the target frame image, which is denoted as a tuple data; the number of subsequent frames corresponding to the subsequent frame image of the target frame image is a preset number of frames;

[0213] Perform data partitioning on all binary data sets to obtain the data partitioning results corresponding to all binary data sets. The data partitioning results include validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training.

[0214] The data partitioning results are used as the target data.

[0215] As can be seen, in this optional embodiment, when preparing training set data, the ED frame image / ES frame image is used as the first frame image, and the subsequent frame image of the ED frame image / ES frame image is used as the second frame image. The image pairs are combined and constructed, so as to obtain the training set data according to the set data partitioning requirements, thereby improving the reliability and accuracy of the training set data preparation.

[0216] In another alternative embodiment, such as Figure 4 As shown, the device also includes a determining module 305, wherein

[0217] The determination module 305 is used to determine the ground truth value of the segmentation mask corresponding to the first frame image in each pair of data before the framework training module 304 performs framework training on the joint learning framework based on the training set data and obtains the framework training result corresponding to the joint learning framework; and to add the ground truth value of the segmentation mask corresponding to the pair of data to the pair of data to obtain the triplet data corresponding to the pair of data.

[0218] The determination module 305 is also used to determine all triplet data as new training set data and trigger the framework training module 304 to perform the above-mentioned frame training on the joint learning framework based on the training set data, and obtain the operation corresponding to the framework training result corresponding to the joint learning framework.

[0219] When performing branch training on the split branch, the branch training data input to the split branch is the training set data including binary data;

[0220] When performing branch training on the optical flow branch, the branch training data input to the optical flow branch is a new training set data that includes triplet data.

[0221] As can be seen, in this optional embodiment, the input training data can be adjusted as needed for different training branches and different training tasks. For the segmentation branch, pre-prepared training set data including binary data is sufficient, while for the optical flow branch, training set data including triple data is required. The triple data includes additional segmentation mask ground truth values ​​to constrain unsupervised optical flow estimation, which helps improve the optical flow estimation effect of this branch. Specifically, it improves the precision of optical flow, eliminates the optical flow fragmentation generated by the pre-trained model during the pre-training stage, and improves the continuity of motion tracking. This precise input training set data also helps improve the training accuracy of the joint learning framework.

[0222] In another optional embodiment, the framework training module 304 performs framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework in the following specific ways:

[0223] When performing branch training on the segmentation branch, for the target binary data of the current input segmentation branch, the target segmentation processing operation is performed on the target binary data to obtain the target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on the pre-built myocardial segmentation pre-training model and segmentation loss calculation based on the preset segmentation loss function; the target segmentation processing result includes the myocardial segmentation result corresponding to the myocardial segmentation processing and the segmentation loss calculation result corresponding to the segmentation loss calculation.

[0224] When performing branch training on the optical flow branch, for the target triplet data of the current input optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain the target estimation result corresponding to the target triplet data. The target optical flow estimation processing operation includes optical flow estimation processing based on a pre-built optical flow estimation pre-trained model and optical flow loss calculation based on a preset optical flow loss function. The target estimation result includes the optical flow estimation result corresponding to the optical flow estimation processing and the optical flow loss calculation result corresponding to the optical flow loss calculation.

[0225] Based on the pre-set joint loss function, the joint loss calculation is performed on the target segmentation result and the target estimation result to obtain the joint loss calculation result corresponding to the target segmentation result and the target estimation result;

[0226] The target segmentation results, target estimation results, and joint loss calculation results are determined as the framework training results corresponding to the joint learning framework;

[0227] Specifically, when the calculation results of segmentation loss, optical flow loss, and joint loss indicate that the joint learning framework has converged, it is determined that the joint learning framework has completed training.

[0228] In this optional embodiment, the myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image.

[0229] The segmentation loss calculation result corresponding to the target binary data is obtained by performing segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data using a preset segmentation loss function;

[0230] The optical flow estimation results corresponding to the target triplet data include the forward optical flow results corresponding to the target triplet data;

[0231] The optical flow loss calculation result corresponding to the target triplet data is obtained by performing optical flow loss calculation on the forward optical flow result, the true value of the segmentation mask corresponding to the target triplet data, and the first frame image in the target triplet data through a preset optical flow loss function.

[0232] Before the joint learning framework is determined and training is completed, the segmentation loss calculation results, optical flow loss calculation results, and joint loss calculation results are used to perform parameter optimization and update on the branch parameters corresponding to the optical flow branch and the segmentation branch, respectively.

[0233] As can be seen, in this optional embodiment, during the training phase of the joint learning framework, the input training data, in addition to the image pairs / tuples I1 and I2 composed of the previous and next frame images, also includes the ground truth value M1 of the segmentation mask corresponding to the previous frame image I1, which is used for auxiliary calculation of the loss modules (optical flow loss, joint loss, and segmentation loss). Specifically, in the optical flow branch, thanks to the constraint of the ground truth value M1 of the segmentation mask, the unsupervised optical flow loss L... flow The optimization target shifted from the global to the local myocardial region.

[0234] In another alternative embodiment, such as Figure 4 As shown, the device also includes a data preprocessing module 306 and an update module 307, wherein:

[0235] The data preprocessing module 306 is used to perform data preprocessing on the training set data before the data input module 303 inputs the training set data into the pre-constructed joint learning framework, so as to obtain the data preprocessing result corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing and data augmentation, and the data augmentation includes at least random brightness transformation and / or random affine transformation.

[0236] The update module 307 is used to update the training set data according to the data preprocessing results and trigger the execution of the above-mentioned operation of inputting the training set data into the pre-built joint learning framework.

[0237] Furthermore, the specific construction method of the pre-built joint learning framework is as follows:

[0238] Target training models that meet the training conditions were selected, including myocardial segmentation output models and optical flow estimation output models; all target training models have been preliminarily trained.

[0239] Determine the training parameters and their values ​​for the target training model; the training parameters include at least one of the following: total number of training iterations for the target training model, batch size, and optimizer parameters; the optimizer parameters include the learning rate and weight decay rate.

[0240] Determine the target loss function for the target training model. The target loss function includes the segmentation loss function corresponding to the myocardial segmentation output model, the optical flow loss function corresponding to the optical flow estimation output model, and the joint loss function used to connect the myocardial segmentation output model and the optical flow estimation output model.

[0241] A joint learning framework is constructed based on the target training model, training parameters and their values, and the target loss function.

[0242] In this optional embodiment, the calculation formula corresponding to the above segmentation loss function is as follows:

[0243] L seg =α*L WD +β*L CE

[0244] Where α and β are weighting coefficients; L seg Refers to the segmentation loss function; L WD Refers to the Weighted Dice Loss function; L CE Refers to the CrossEntropy Loss function;

[0245] The formula for calculating the optical flow loss function is as follows:

[0246]

[0247] Where p represents an image pixel, i and j represent pixel coordinates, d represents the partial derivatives in the x and y directions, Ψ is the robustness penalty function, q, ε, and α are the adjustable parameters of the optical flow loss function, and λ is the weighting coefficient.

[0248] The formula for calculating the joint loss function is as follows:

[0249] L=α*L flow +β*L seg +γ*L joint

[0250] Where α, β, and γ are all weighting coefficients; L is the final joint loss function; L joint This is the initial joint loss function;

[0251] The initial joint loss function L joint The calculation formula is as follows:

[0252]

[0253] Where, parameter l represents the subscript for different categories, i indicates the i-th pixel, and N is the total number of pixels. This indicates that the predicted segmentation mask P2 is used in the predicted optical flow F. 1→2 Mask of deformation results under action;

[0254] in:

[0255] and These represent the label value and predicted value of pixel i in category l, respectively.

[0256] As can be seen, in this optional embodiment, a data preprocessing procedure for the training set data is set before inputting the training set data into the joint learning framework. Through image scaling and normalization, the consistency of the training set data is improved, which is conducive to improving the training efficiency of subsequent framework training (including model training). This data augmentation is conducive to improving the generalization of the model. At the same time, for the joint learning framework, the parameter values ​​of relevant training parameters can be accurately set according to the actual training requirements of the framework, which improves the accuracy of the framework construction of the joint learning framework.

[0257] Example 4

[0258] Please see Figure 5 , Figure 5 This is a schematic diagram of another myocardial region data processing device based on a joint learning framework disclosed in an embodiment of the present invention. Figure 5 As shown, the myocardial region data processing device based on the joint learning framework may include:

[0259] Memory 401 storing executable program code;

[0260] Processor 402 coupled to memory 401;

[0261] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the myocardial region data processing method based on the joint learning framework described in Embodiment 1 or Embodiment 2 of the present invention.

[0262] Example 5

[0263] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the myocardial region data processing method based on a joint learning framework described in Embodiment 1 or Embodiment 2 of this invention.

[0264] Example 6

[0265] This invention discloses a computer program product, which includes a non-transitory computer storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the myocardial region data processing method based on the joint learning framework described in Embodiment 1 or Embodiment 2.

[0266] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0267] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0268] Finally, it should be noted that the myocardial region data processing method and apparatus based on a joint learning framework disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing myocardial region data based on a joint learning framework, characterized in that, The method includes: Acquire cardiac image data for performing framework training, the cardiac image data including multiple frame images that have undergone image annotation processing, all of which include at least ED frame images and ES frame images; Image processing operations are performed on the cardiac image data to obtain target data, which includes at least training set data for performing the framework training; multiple target frame images are selected from the cardiac image data, which are either the ED frame image or the ES frame image; for each target frame image, an image pair corresponding to the target frame image is constructed by taking the target frame image as the first frame image and the subsequent frame image of the target frame image as the second frame image, and denoted as a binary data set; the training set data includes multiple example data, each example of the sub-data is a binary data set, and each binary data set consists of a previous image frame and the corresponding subsequent image frame; For each pair of data, determine the ground truth value of the segmentation mask corresponding to the first frame image in the pair of data; add the ground truth value of the segmentation mask corresponding to the pair of data to the pair of data to obtain the triplet data corresponding to the pair of data; determine all the triplet data as new training set data, and trigger the operation of performing framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework; wherein, when performing branch training on the segmentation branch, the input branch training data of the segmentation branch is the training set data including the pair of data; when performing branch training on the optical flow branch, the input branch training data of the optical flow branch is the new training set data including the triplet data. The training set data is input into a pre-constructed joint learning framework, and the joint learning framework is trained according to the training set data to obtain the framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, the training task for the joint learning framework is determined to be completed. The joint learning framework includes a segmentation branch, an optical flow branch, and a loss calculation module. The segmentation branch is used to perform myocardial region segmentation and extraction for each sub-data instance. The optical flow branch is used to determine the motion optical flow information of all the sub-data instances on the myocardium. The loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch, and a joint loss function, wherein the joint loss function is used to connect the segmentation branch and the optical flow branch.

2. The myocardial region data processing method based on a joint learning framework according to claim 1, characterized in that, The step of performing image processing operations on the cardiac image data to obtain target data includes: The number of subsequent frames corresponding to the target frame image is the preset number of frames; Perform data partitioning on all the said binary data to obtain data partitioning results corresponding to all the said binary data. The data partitioning results include validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training. The data segmentation results are determined as the target data.

3. The myocardial region data processing method based on a joint learning framework according to claim 2, characterized in that, The step of performing framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework includes: When performing branch training on the segmentation branch, for the target binary data currently input to the segmentation branch, a target segmentation processing operation is performed on the target binary data to obtain a target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on a pre-built myocardial segmentation pre-training model and segmentation loss calculation based on a preset segmentation loss function; the target segmentation processing result includes the myocardial segmentation result corresponding to the myocardial segmentation processing and the segmentation loss calculation result corresponding to the segmentation loss calculation. When performing branch training on the optical flow branch, for the target triplet data currently input to the optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain a target estimation result corresponding to the target triplet data; the target optical flow estimation processing operation includes optical flow estimation processing based on a pre-built optical flow estimation pre-trained model and optical flow loss calculation based on a preset optical flow loss function; the target estimation result includes the optical flow estimation result corresponding to the optical flow estimation processing and the optical flow loss calculation result corresponding to the optical flow loss calculation. Based on a pre-set joint loss function, joint loss calculation is performed on the target segmentation result and the target estimation result to obtain the joint loss calculation result corresponding to the target segmentation result and the target estimation result; The target segmentation result, the target estimation result, and the joint loss calculation result are determined as the framework training result corresponding to the joint learning framework; Specifically, when the segmentation loss calculation result, the optical flow loss calculation result, and the joint loss calculation result indicate that the joint learning framework has reached convergence, it is determined that the joint learning framework has completed training.

4. The myocardial region data processing method based on a joint learning framework according to claim 3, characterized in that, The myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image. The segmentation loss calculation result corresponding to the target binary data is obtained by performing the segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data using the preset segmentation loss function; The optical flow estimation result corresponding to the target triplet data includes the forward optical flow result corresponding to the target triplet data; The optical flow loss calculation result corresponding to the target triplet data is obtained by performing the optical flow loss calculation on the forward optical flow result, the segmentation mask ground value corresponding to the target triplet data, and the first frame image in the target triplet data using the preset optical flow loss function; Before the joint learning framework completes its training, the segmentation loss calculation results, the optical flow loss calculation results, and the joint loss calculation results are used to perform parameter optimization and updates on the branch parameters corresponding to the optical flow branch and the segmentation branch, respectively.

5. The method for processing myocardial region data based on a joint learning framework according to any one of claims 1-4, characterized in that, Before inputting the training set data into the pre-built joint learning framework, the method further includes: Data preprocessing is performed on the training set data to obtain data preprocessing results corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing, and data augmentation, and the data augmentation includes at least random brightness transformation and / or random affine transformation; The training set data is updated based on the data preprocessing results, and the operation of inputting the training set data into the pre-built joint learning framework is triggered. Furthermore, the specific construction method of the pre-built joint learning framework is as follows: A target training model that meets the training conditions is selected, including a myocardial segmentation output model and an optical flow estimation output model; all target training models have been preliminarily trained. Determine the training parameters and their values ​​for the target training model; the training parameters include at least one of the following: total number of training iterations for the target training model, batch size, and optimizer parameters; the optimizer parameters include the learning rate and the weight decay rate. Determine the target loss function for the target training model. The target loss function includes a segmentation loss function corresponding to the myocardial segmentation output model, an optical flow loss function corresponding to the optical flow estimation output model, and a joint loss function for connecting the myocardial segmentation output model and the optical flow estimation output model. A joint learning framework is constructed based on the target training model, the training parameters and their values, and the target loss function.

6. The method for processing myocardial region data based on a joint learning framework according to claim 5, characterized in that, The calculation formula for the segmentation loss function is as follows: in, α , β These are the weighting coefficients; L seg Refers to the segmentation loss function; L WD Refers to the Weighted Dice Loss function; L CE Refers to the CrossEntropy Loss function; The calculation formula for the optical flow loss function is as follows: in, p Represents image pixels, i , j Represents pixel coordinates, d express x and y The partial derivative in the direction, Ψ is the robustness penalty function, q , ε , α These are the adjustable parameters of the optical flow loss function; λ Weighting coefficients: The calculation formula for the joint loss function is as follows: in, α , β , γ All are weighting coefficients; L This is the final joint loss function; L joint This is the initial joint loss function; The initial joint loss function L joint The calculation formula is as follows: Among them, parameters l Indicates different category subscripts, i Used to indicate the i Pixels N Total number of pixels Represents the predicted segmentation mask Predicting optical flow Mask of deformation results under action; in: , ; and Representing pixels i In category l The label value and the predicted value at that location.

7. A myocardial region data processing apparatus based on a joint learning framework for implementing the method as described in any one of claims 1-6, characterized in that, The device includes: The acquisition module is used to acquire cardiac image data for performing framework training. The cardiac image data includes multiple frame images that have undergone image annotation processing, and all the frame images include at least ED frame images and ES frame images. An image processing module is used to perform image processing operations on the cardiac image data to obtain target data. The target data includes at least training set data for performing the training of the framework. The training set data includes multiple example data, each example of the sub-data is a binary data, and each binary data consists of a previous image frame and a corresponding subsequent image frame. A data input module is used to input the training set data into a pre-built joint learning framework; The framework training module is used to perform framework training on the joint learning framework based on the training set data to obtain the framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, the training task for the joint learning framework is determined to be completed. The joint learning framework includes a segmentation branch, an optical flow branch, and a loss calculation module. The segmentation branch is used to perform myocardial region segmentation and extraction for each sub-data instance. The optical flow branch is used to determine the motion optical flow information of all the sub-data instances on the myocardium. The loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch, and a joint loss function, wherein the joint loss function is used to connect the segmentation branch and the optical flow branch.

8. A myocardial region data processing device based on a joint learning framework, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the myocardial region data processing method based on the joint learning framework as described in any one of claims 1-6.

9. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the myocardial region data processing method based on the joint learning framework as described in any one of claims 1-6.

Images