Myocardial region data processing method and device based on joint learning framework

Abstract

The invention relates to the technical field of image processing, discloses a myocardial region data processing method and device based on a joint learning framework, and designs a two-stage optimization method of myocardial segmentation pre-training, optical flow estimation pre-training and joint training. The output performance of a final unsupervised optical flow estimation model on a local myocardial region is improved, and semantic information brought by myocardial segmentation can provide physiological shape constraint for an optical flow estimation task and realize optical flow refinement on the local myocardial region; the optical flow estimation can add time sequence constraint for the segmentation task, and the segmentation target frame is diffused to the whole cardiac cycle sequence from the ED / ES frame through motion consistency.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for processing myocardial region data based on a joint learning framework. Background Art

[0002] In modern medicine, the commonly used technologies for detecting and calculating myocardial motion state and strain indexes mainly include tissue Doppler imaging (TDI) technology and 2D-speckle tracking echocardiography (2D-STE) technology. However, in practical applications,

[0003] Doppler effect measurement has certain limitations, and its measurement results are affected by the angle between the direction of the sound beam and the direction of tissue movement;

[0004] The measurement effect of two-dimensional speckle tracking is often directly related to the image quality, and there are large differences in the algorithm implementation and parameter meanings of different manufacturers. There is no unified calculation standard, which to a certain extent reduces the data analysis efficiency of professional doctors.

[0005] In recent years, with the continuous development of artificial intelligence and deep learning technology, the network model based on optical flow estimation has faster calculation speed and higher robustness than traditional algorithms, which seems to bring a lower-cost solution to the defects in traditional methods. However, since the heart is not a simple rigid body movement, its movement form is relatively complex, it is almost impossible to manually annotate dense optical flow, and it is difficult to artificially synthesize heart movement, resulting in slow progress in the application of deep learning technology to myocardial motion analysis, and a lack of a mature solution. Summary of the invention

[0006] The present invention provides a myocardial region data processing method and device based on a joint learning framework, which can be beneficial to improving the segmentation effect of myocardial segmentation and improving the optical flow precision of optical flow estimation tasks.

[0007] In order to solve the above technical problems, the first aspect of the present invention discloses a method for processing myocardial region data based on a joint learning framework, the method comprising:

[0008] Acquire cardiac image data for performing framework training, wherein the cardiac image data includes a plurality of frame images that have been processed by image annotation, and all the frame images include at least ED frame images and ES frame images;

[0009] Performing an image processing operation on the cardiac image data to obtain target data, wherein the target data at least includes training set data for performing the framework training; the training set data includes multiple examples of data, each example of sub-data is a two-tuple data, and each of the two-tuple data consists of a front image frame and a rear image frame corresponding to the front image frame;

[0010] Inputting the training set data into a pre-constructed joint learning framework, and performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, determining that the training task for the joint learning framework is completed;

[0011] Among them, the joint learning framework includes a segmentation branch, an optical flow branch and a loss calculation module; the segmentation branch is used to perform myocardial area segmentation and extraction for each example of the sub-data; the optical flow branch is used to determine the motion optical flow information of all the sub-data on the myocardium; the loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch and a joint loss function, and the joint loss function is used to connect the segmentation branch and the optical flow branch.

[0012] As an optional implementation, in the first aspect of the present invention, performing an image processing operation on the cardiac image data to obtain target data includes:

[0013] Selecting a plurality of target frame images from the cardiac image data, wherein the target frame images are the ED frame images or the ES frame images;

[0014] For each target frame image, the target frame image is taken as the first frame image, and the subsequent frame image of the target frame image is taken as the second frame image, to construct an image pair corresponding to the target frame image, which is recorded as a binary data; the subsequent frame number corresponding to the subsequent frame image of the target frame image is a preset frame number;

[0015] Performing data partitioning on all the two-tuple data to obtain data partitioning results corresponding to all the two-tuple data, wherein the data partitioning results include validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training;

[0016] The data division result is determined as target data.

[0017] As an optional implementation, in the first aspect of the present invention, before performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework, the method further includes:

[0018] For each of the two-tuple data, determine the segmentation mask true value corresponding to the first frame image in the two-tuple data; add the segmentation mask true value corresponding to the two-tuple data to the two-tuple data to obtain the three-tuple data corresponding to the two-tuple data;

[0019] Determine all the triplet data as new training set data, and trigger the execution of the operation of performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework;

[0020] Wherein, when performing branch training on the segmentation branch, the branch training data input into the segmentation branch is the training set data including the binary data;

[0021] When performing branch training on the optical flow branch, the branch training data input to the optical flow branch is the new training set data including the triplet data.

[0022] As an optional implementation, in the first aspect of the present invention, performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework includes:

[0023] When branch training is performed on the segmentation branch, for the target binary data currently input into the segmentation branch, a target segmentation processing operation is performed on the target binary data to obtain a target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on a pre-constructed myocardial segmentation pre-training model and segmentation loss calculation based on a preset segmentation loss function; the target segmentation processing result includes a myocardial segmentation result corresponding to the myocardial segmentation processing and a segmentation loss calculation result corresponding to the segmentation loss calculation;

[0024] When branch training is performed on the optical flow branch, for the target triplet data currently input into the optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain a target estimation result corresponding to the target triplet data; the target optical flow estimation processing operation includes an optical flow estimation processing based on a pre-built optical flow estimation pre-training model and an optical flow loss calculation based on a preset optical flow loss function; the target estimation result includes an optical flow estimation result corresponding to the optical flow estimation processing and an optical flow loss calculation result corresponding to the optical flow loss calculation;

[0025] According to a preset joint loss function, performing a joint loss calculation on the target segmentation result and the target estimation result to obtain a joint loss calculation result corresponding to the target segmentation result and the target estimation result;

[0026] Determine the target segmentation result, the target estimation result, and the joint loss calculation result as a framework training result corresponding to the joint learning framework;

[0027] When the segmentation loss calculation result, the optical flow loss calculation result, and the joint loss calculation result indicate that the joint learning framework has been trained to convergence, it is determined that the joint learning framework has completed training.

[0028] As an optional implementation, in the first aspect of the present invention, the myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image;

[0029] The segmentation loss calculation result corresponding to the target binary data is obtained by performing the segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data through the preset segmentation loss function;

[0030] The optical flow estimation result corresponding to the target triplet data includes a forward optical flow result corresponding to the target triplet data;

[0031] The optical flow loss calculation result corresponding to the target triplet data is obtained by performing the optical flow loss calculation on the forward optical flow result, the segmentation mask true value corresponding to the target triplet data, and the first frame image in the target triplet data through the preset optical flow loss function;

[0032] Before determining that the joint learning framework has completed framework training, the segmentation loss calculation result, the optical flow loss calculation result and the joint loss calculation result are used to perform parameter optimization and update on branch parameters corresponding to the optical flow branch and the segmentation branch.

[0033] As an optional implementation, in the first aspect of the present invention, before inputting the training set data into the pre-built joint learning framework, the method further includes:

[0034] Performing data preprocessing on the training set data to obtain data preprocessing results corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing, and data enhancement, and the data enhancement includes at least random brightness transformation and / or random affine transformation;

[0035] The training set data is updated according to the data preprocessing result, and the corresponding operation of inputting the training set data into the pre-built joint learning framework is triggered to be executed;

[0036] And, the pre-built federated learning framework is built as follows:

[0037] Selecting target training models that meet the training conditions, the target training models include a myocardial segmentation output model and an optical flow estimation output model; the target training models have all preliminarily passed model training;

[0038] Determine the training parameters and parameter values ​​for the target training model; the training parameters include at least one of the total number of training iterations, batch size, and optimizer parameters for the target training model; the optimizer parameters include a learning rate and a weight decay rate;

[0039] Determine a target loss function for the target training model, wherein the target loss function includes a segmentation loss function corresponding to the myocardial segmentation output model, an optical flow loss function corresponding to the optical flow estimation output model, and a joint loss function for connecting the myocardial segmentation output model and the optical flow estimation output model;

[0040] A joint learning framework is constructed based on the target training model, the training parameters and their parameter values, and the target loss function.

[0041] As an optional implementation, in the first aspect of the present invention, the calculation formula corresponding to the segmentation loss function is as follows:

[0042] L seg =α*L WD +β*L CE

[0043] Among them, α and β are weight coefficients; L seg Refers to the segmentation loss function; L WD Refers to the Weighted Dice Loss function; L CE Refers to the CrossEntropy Loss function;

[0044] The calculation formula corresponding to the optical flow loss function is as follows:

[0045]

[0046] Among them, p represents the image pixel, i and j represent the pixel coordinates, d represents the partial derivatives in the x and y directions, Ψ is the robustness penalty function, q, ε, and α are the adjustable parameters of the optical flow loss function; λ is the weight coefficient:

[0047] The calculation formula corresponding to the joint loss function is as follows:

[0048] L=α*L flow +β*L seg +γ*Ljoint

[0049] Among them, α, β, and γ are weight coefficients; L is the final joint loss function; L joint is the initial joint loss function;

[0050] The initial joint loss function L joint The calculation formula is as follows:

[0051]

[0052] Among them, parameter l represents the subscript of different categories, i is used to indicate the i-th pixel, N is the total number of pixels, Indicates that the predicted segmentation mask P2 is used to predict the optical flow F 1→2 The deformation result mask under the action;

[0053] in:

[0054] and Represent the label value and predicted value of pixel i in category l respectively.

[0055] The second aspect of the present invention discloses a myocardial region data processing device based on a joint learning framework, the device comprising:

[0056] An acquisition module, used for acquiring cardiac image data for executing framework training, wherein the cardiac image data includes a plurality of frame images that have been processed by image annotation, and all the frame images include at least ED frame images and ES frame images;

[0057] An image processing module, configured to perform image processing operations on the cardiac image data to obtain target data, wherein the target data at least includes training set data for performing the framework training; the training set data includes multiple examples of data, each example of sub-data is a binary data, and each binary data is composed of a front image frame and a rear image frame corresponding to the front image frame;

[0058] A data input module, used for inputting the training set data into a pre-built joint learning framework;

[0059] A framework training module, used to perform framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, determine that the training task for the joint learning framework is completed;

[0060] Among them, the joint learning framework includes a segmentation branch, an optical flow branch and a loss calculation module; the segmentation branch is used to perform myocardial area segmentation and extraction for each example of the sub-data; the optical flow branch is used to determine the motion optical flow information of all the sub-data on the myocardium; the loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch and a joint loss function, and the joint loss function is used to connect the segmentation branch and the optical flow branch.

[0061] As an optional implementation, in the second aspect of the present invention, the image processing module performs an image processing operation on the cardiac image data to obtain the target data in a manner that specifically includes:

[0062] Selecting a plurality of target frame images from the cardiac image data, wherein the target frame images are the ED frame images or the ES frame images;

[0063] For each target frame image, the target frame image is taken as the first frame image, and the subsequent frame image of the target frame image is taken as the second frame image, to construct an image pair corresponding to the target frame image, which is recorded as a binary data; the subsequent frame number corresponding to the subsequent frame image of the target frame image is a preset frame number;

[0064] Performing data partitioning on all the two-tuple data to obtain data partitioning results corresponding to all the two-tuple data, wherein the data partitioning results include validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training;

[0065] The data division result is determined as target data.

[0066] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0067] a determination module, configured to determine, for each of the two-tuple data, a true value of a segmentation mask corresponding to the first frame image in the two-tuple data before the framework training module performs framework training on the joint learning framework according to the training set data and obtains a framework training result corresponding to the joint learning framework; and add the true value of the segmentation mask corresponding to the two-tuple data to the two-tuple data to obtain a triplet of data corresponding to the two-tuple data;

[0068] The determination module is further used to determine all the triple data as new training set data, and trigger the framework training module to perform the framework training on the joint learning framework according to the training set data to obtain an operation corresponding to the framework training result corresponding to the joint learning framework;

[0069] Wherein, when performing branch training on the segmentation branch, the branch training data input into the segmentation branch is the training set data including the binary data;

[0070] When performing branch training on the optical flow branch, the branch training data input to the optical flow branch is the new training set data including the triplet data.

[0071] As an optional implementation, in the second aspect of the present invention, the framework training module performs framework training on the joint learning framework according to the training set data, and a method of obtaining a framework training result corresponding to the joint learning framework specifically includes:

[0072] When branch training is performed on the segmentation branch, for the target binary data currently input into the segmentation branch, a target segmentation processing operation is performed on the target binary data to obtain a target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on a pre-constructed myocardial segmentation pre-training model and segmentation loss calculation based on a preset segmentation loss function; the target segmentation processing result includes a myocardial segmentation result corresponding to the myocardial segmentation processing and a segmentation loss calculation result corresponding to the segmentation loss calculation;

[0073] When branch training is performed on the optical flow branch, for the target triplet data currently input into the optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain a target estimation result corresponding to the target triplet data; the target optical flow estimation processing operation includes an optical flow estimation processing based on a pre-built optical flow estimation pre-training model and an optical flow loss calculation based on a preset optical flow loss function; the target estimation result includes an optical flow estimation result corresponding to the optical flow estimation processing and an optical flow loss calculation result corresponding to the optical flow loss calculation;

[0074] According to a preset joint loss function, performing a joint loss calculation on the target segmentation result and the target estimation result to obtain a joint loss calculation result corresponding to the target segmentation result and the target estimation result;

[0075] Determine the target segmentation result, the target estimation result, and the joint loss calculation result as a framework training result corresponding to the joint learning framework;

[0076] When the segmentation loss calculation result, the optical flow loss calculation result, and the joint loss calculation result indicate that the joint learning framework has been trained to convergence, it is determined that the joint learning framework has completed training.

[0077] As an optional implementation, in the second aspect of the present invention, the myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image;

[0078] The segmentation loss calculation result corresponding to the target binary data is obtained by performing the segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data through the preset segmentation loss function;

[0079] The optical flow estimation result corresponding to the target triplet data includes a forward optical flow result corresponding to the target triplet data;

[0080] The optical flow loss calculation result corresponding to the target triplet data is obtained by performing the optical flow loss calculation on the forward optical flow result, the segmentation mask true value corresponding to the target triplet data, and the first frame image in the target triplet data through the preset optical flow loss function;

[0081] Before determining that the joint learning framework has completed framework training, the segmentation loss calculation result, the optical flow loss calculation result and the joint loss calculation result are used to perform parameter optimization and update on branch parameters corresponding to the optical flow branch and the segmentation branch.

[0082] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0083] A data preprocessing module, configured to perform data preprocessing on the training set data before the data input module inputs the training set data into a pre-built joint learning framework, so as to obtain a data preprocessing result corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing, and data enhancement, and the data enhancement includes at least random brightness transformation and / or random affine transformation;

[0084] An updating module, used for updating the training set data according to the data preprocessing result, and triggering the execution of the corresponding operation of inputting the training set data into the pre-built joint learning framework;

[0085] And, the pre-built federated learning framework is built as follows:

[0086] Selecting target training models that meet the training conditions, the target training models include a myocardial segmentation output model and an optical flow estimation output model; the target training models have all preliminarily passed model training;

[0087] Determine the training parameters and parameter values ​​for the target training model; the training parameters include at least one of the total number of training iterations, batch size, and optimizer parameters for the target training model; the optimizer parameters include a learning rate and a weight decay rate;

[0088] Determine a target loss function for the target training model, wherein the target loss function includes a segmentation loss function corresponding to the myocardial segmentation output model, an optical flow loss function corresponding to the optical flow estimation output model, and a joint loss function for connecting the myocardial segmentation output model and the optical flow estimation output model;

[0089] A joint learning framework is constructed based on the target training model, the training parameters and their parameter values, and the target loss function.

[0090] As an optional implementation, in the second aspect of the present invention, the calculation formula corresponding to the segmentation loss function is as follows:

[0091] L seg =α*L WD +β*L CE

[0092] Among them, α and β are weight coefficients; L seg Refers to the segmentation loss function; L WD Refers to the Weighted Dice Loss function; L CE Refers to the CrossEntropy Loss function;

[0093] The calculation formula corresponding to the optical flow loss function is as follows:

[0094]

[0095] Among them, p represents the image pixel, i and j represent the pixel coordinates, d represents the partial derivatives in the x and y directions, Ψ is the robustness penalty function, q, ε, and α are the adjustable parameters of the optical flow loss function; λ is the weight coefficient:

[0096] The calculation formula corresponding to the joint loss function is as follows:

[0097] L=α*L flow +β*L seg +γ*L joint

[0098] Among them, α, β, and γ are weight coefficients; L is the final joint loss function; L joint is the initial joint loss function;

[0099] The initial joint loss function L jointThe calculation formula is as follows:

[0100]

[0101] Among them, parameter l represents the subscript of different categories, i is used to indicate the i-th pixel, N is the total number of pixels, Indicates that the predicted segmentation mask P2 is used to predict the optical flow F 1→2 The deformation result mask under the action;

[0102] in:

[0103] and Represent the label value and predicted value of pixel i in category l respectively.

[0104] The third aspect of the present invention discloses another myocardial region data processing device based on a joint learning framework, the device comprising:

[0105] A memory storing executable program code;

[0106] a processor coupled to the memory;

[0107] The processor calls the executable program code stored in the memory to execute the myocardial region data processing method based on the joint learning framework disclosed in the first aspect of the present invention.

[0108] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the myocardial region data processing method based on the joint learning framework disclosed in the first aspect of the present invention.

[0109] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0110] In an embodiment of the present invention, a method for processing myocardial region data based on a joint learning framework is provided, the method comprising: acquiring cardiac image data for executing framework training, the cardiac image data comprising a plurality of frame images that have been processed by image annotation, all frame images comprising at least ED frame images and ES frame images; performing image processing operations on the cardiac image data to obtain target data, the target data comprising at least training set data for executing framework training; the training set data comprising a plurality of example data, each example data being a binary data, each binary data consisting of a front image frame and a rear image frame corresponding to the front image frame; inputting the training set data into a pre-constructed joint learning framework , and perform framework training on the joint learning framework according to the training set data to obtain the framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, it is determined that the training task for the joint learning framework is completed; wherein the joint learning framework includes a segmentation branch, an optical flow branch and a loss calculation module; the segmentation branch is used to perform myocardial area segmentation and extraction on each example data; the optical flow branch is used to determine the motion optical flow information of all sub-data on the myocardium; the loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch and a joint loss function, and the joint loss function is used to connect the segmentation branch and the optical flow branch. It can be seen that the implementation of the present invention implements the preprocessing of cardiac image data before inputting the training set data into the joint learning framework, so that the training data input into the joint learning framework is more accurate, which is beneficial to improving the training efficiency of the joint learning framework; in addition, based on the constructed joint learning framework, the dual-model synchronous training and optimization of the myocardial segmentation pre-training model (corresponding to the segmentation branch) and the optical flow estimation pre-training model (corresponding to the optical flow branch) can be realized; further, the joint learning framework also sets a joint loss function, so that the semantic information brought by the myocardial segmentation pre-training model can provide physiological shape constraints for the optical flow estimation task performed by the optical flow estimation pre-training model, thereby realizing the optical flow refinement on the local myocardial area, which is beneficial to improve the fineness of the output optical flow corresponding to the optical flow branch, and reduce the granularity and sense of fragmentation corresponding to the optical flow output by the optical flow branch; at the same time, the optical flow estimation realized by the optical flow branch can add timing constraints to the segmentation task of the segmentation branch, and diffuse the segmentation target frame processed by the segmentation branch to the entire cardiac cycle sequence through motion consistency, so that the annotation data input to the segmentation branch does not need to be fully labeled for the myocardial sequence, thereby reducing the workload of myocardial annotation and the image processing cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0112] Figure 1 It is a flow chart of a method for processing myocardial region data based on a joint learning framework disclosed in an embodiment of the present invention;

[0113] Figure 2 It is a flow chart of another method for processing myocardial region data based on a joint learning framework disclosed in an embodiment of the present invention;

[0114] Figure 3 It is a structural schematic diagram of a myocardial region data processing device based on a joint learning framework disclosed in an embodiment of the present invention;

[0115] Figure 4 is a schematic structural diagram of another myocardial region data processing device based on a joint learning framework disclosed in an embodiment of the present invention;

[0116] Figure 5 is a structural schematic diagram of another myocardial region data processing device based on a joint learning framework disclosed in an embodiment of the present invention;

[0117] Figure 6 It is a framework diagram of a joint learning framework disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0118] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0119] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0120] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0121] The present invention discloses a myocardial region data processing method and device based on a joint learning framework, which implements preprocessing of cardiac image data before inputting training set data into the joint learning framework, so that the training data input into the joint learning framework is more accurate, which is beneficial to improving the training efficiency of the joint learning framework; in addition, based on the constructed joint learning framework, it is possible to implement dual-model synchronous training and optimization of a myocardial segmentation pre-training model (corresponding to a segmentation branch) and an optical flow estimation pre-training model (corresponding to an optical flow branch); further, the joint learning framework is also provided with a joint loss function, so that the semantic information brought by the myocardial segmentation pre-training model can provide physiological shape constraints for an optical flow estimation task performed by the optical flow estimation pre-training model, thereby realizing optical flow refinement on a local myocardial region, which is beneficial to improving the fineness of the output optical flow corresponding to the optical flow branch, and reducing the granularity and sense of fragmentation corresponding to the optical flow output by the optical flow branch; at the same time, the optical flow estimation implemented by the optical flow branch can add timing constraints to the segmentation task of the segmentation branch, and diffuse the segmentation target frame processed by the segmentation branch to the entire cardiac cycle sequence through motion consistency, so that the annotation data input into the segmentation branch does not need to be fully myocardially annotated, thereby reducing the workload of myocardial annotation and the image processing cost. The following are detailed descriptions of each.

[0122] Embodiment 1

[0123] See also Figure 1 , Figure 1 is a flow chart of a method for processing myocardial region data based on a joint learning framework disclosed in an embodiment of the present invention. Figure 1 The described myocardial region data processing method based on the joint learning framework can be applied to a myocardial region data processing device based on the joint learning framework, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the myocardial region data processing method based on the joint learning framework may include the following operations:

[0124] 101. Acquire cardiac image data for executing framework training, where the cardiac image data includes a plurality of frame images that have been processed by image annotation, and all frame images include at least ED frame images and ES frame images.

[0125] In the embodiment of the present invention, the cardiac image data comes from the public dataset CAMUS dataset and the clinical self-collected dataset of the cooperative hospital. The CAMUS dataset contains 2D apical four-chamber (4CH) and two-chamber (2CH) view sequences obtained from 500 patients, of which about 19% of the patients have poor image quality, and half of them have a left ventricular ejection fraction (LVEF) lower than 45% and are considered to have pathological risks. The sample data of each patient consists of a complete set of 2CH / 4CH cardiac cycle sequences, ultrasound images of the ED / ES period and their cardiac structure annotations, patient acquisition information and other basic parts, and the original input image is provided in raw / mhd file format. Among them, the cardiac structure annotation data is obtained by manual annotation at ED and ES by professional doctors through semi-automatic annotation tools. In the mask classification, 0, 1, 2 and 3 represent pixels located in the background, left ventricular cavity, left ventricular myocardium and left atrial cavity, respectively.

[0126] 102. Perform an image processing operation on the cardiac image data to obtain target data, where the target data at least includes training set data for performing framework training.

[0127] In the embodiment of the present invention, the training set data includes multiple example data, each example data is a two-tuple data, and each two-tuple data consists of a front image frame and a rear image frame corresponding to the front image frame.

[0128] 103. Input the training set data into a pre-built joint learning framework, and perform framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework.

[0129] 104. When the framework training result indicates that the joint learning framework training has reached convergence, it is determined that the training task for the joint learning framework is completed.

[0130] In an embodiment of the present invention, a joint learning framework includes a segmentation branch, an optical flow branch and a loss calculation module; the segmentation branch is used to perform myocardial region segmentation extraction on each example data; the optical flow branch is used to determine the motion optical flow information of all sub-data on the myocardium; the loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch and a joint loss function, and the joint loss function is used to connect the segmentation branch and the optical flow branch.

[0131] It can be seen that the implementation Figure 1The described method for processing myocardial region data based on a joint learning framework implements preprocessing of cardiac image data before inputting the training set data into the joint learning framework, so that the training data input into the joint learning framework is more accurate, which is beneficial to improving the training efficiency of the joint learning framework; in addition, based on the constructed joint learning framework, it is possible to realize the synchronous training and optimization of the dual models of the myocardial segmentation pre-training model (corresponding to the segmentation branch) and the optical flow estimation pre-training model (corresponding to the optical flow branch); further, the joint learning framework also sets a joint loss function, so that the semantic information brought by the myocardial segmentation pre-training model can provide physiological shape constraints for the optical flow estimation task performed by the optical flow estimation pre-training model, thereby realizing the optical flow refinement on the local myocardial region, which is beneficial to improving the fineness of the output optical flow corresponding to the optical flow branch, and reducing the granularity and sense of fragmentation corresponding to the optical flow output by the optical flow branch; at the same time, the optical flow estimation realized by the optical flow branch can add timing constraints to the segmentation task of the segmentation branch, and diffuse the segmentation target frame processed by the segmentation branch to the entire cardiac cycle sequence through motion consistency, so that the labeled data input to the segmentation branch does not need to be labeled for the entire sequence of myocardium, thereby reducing the workload of myocardial labeling and the image processing cost.

[0132] In an optional embodiment, before the above step 103 inputs the training set data into the pre-built joint learning framework, the method further includes:

[0133] Performing data preprocessing on the training set data to obtain data preprocessing results corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing, and data enhancement, and the data enhancement includes at least random brightness transformation and / or random affine transformation;

[0134] Update the training set data according to the data preprocessing results, and trigger the execution of the above-mentioned operation of inputting the training set data into the pre-built joint learning framework;

[0135] In this optional embodiment, the preset image size is conventionally set to: 256*256 size; the normalization process is used to standardize the image to between 0 and 1; the data enhancement operation is used to improve the generalization of the model. And, the construction method of the pre-built joint learning framework is specifically as follows:

[0136] Select target training models that meet the training conditions. The target training models include myocardial segmentation output model and optical flow estimation output model. The target training models have all passed the preliminary model training.

[0137] Determine training parameters and parameter values ​​for the target training model; the training parameters include at least one of the total number of training iterations, batch size, and optimizer parameters for the target training model; the optimizer parameters include a learning rate and a weight decay rate;

[0138] Determine a target loss function for the target training model, the target loss function including a segmentation loss function corresponding to the myocardial segmentation output model, an optical flow loss function corresponding to the optical flow estimation output model, and a joint loss function for connecting the myocardial segmentation output model and the optical flow estimation output model;

[0139] A joint learning framework is constructed based on the target training model, training parameters and their parameter values, and the target loss function.

[0140] In this optional embodiment, in the joint training stage, the joint learning framework simultaneously optimizes the optical flow model (optical flow estimation pre-training model) and the segmentation model (myocardial segmentation pre-training model) in one iteration, the total number of training iterations epochs is set to 50, and the batch size is set to 10. In actual experiments, the Adam optimizer can be selected to perform gradient optimization and back propagation on the two branch networks, with the learning rate fixed to 0.0001 and the weight decay rate fixed to 0.0001. Other network configurations remain the same as the settings of each branch in its pre-training stage.

[0141] In this optional embodiment, the calculation formula corresponding to the segmentation loss function is as follows:

[0142] L seg =α*L WD +β*L CE

[0143] Among them, α and β are weight coefficients. In actual experiments, α can be set to 1.0 and β to 0.1 to maintain L WD The dominant role of function in model optimization, where L seg Refers to the segmentation loss function; L WD Refers to the Weighted Dice Loss function; L CE Refers to the CrossEntropy Loss function;

[0144] Furthermore, the calculation formula corresponding to the Weighted Dice Loss function is as follows:

[0145]

[0146] Among them, the parameter l represents the subscript of different categories, and Respectively represent the label value and predicted value of pixel i in category l, N is the total number of pixels, where:

[0147]

[0148] The calculation formula corresponding to the CrossEntropy Loss function is as follows:

[0149]

[0150] In this optional embodiment, the calculation formula corresponding to the optical flow loss function is as follows:

[0151]

[0152] Among them, p represents the image pixel point, i and j represent the pixel coordinates, d represents the partial derivatives in the x and y directions, Ψ is the robustness penalty function, q, ε, and α are the adjustable parameters of the optical flow loss function, which can be set to 0.4, 0.01, and 10 respectively in practical applications; λ is the weight coefficient, which can be set to 0.1.

[0153] In this alternative embodiment, the combined loss L joint Based on the motion consistency constraint, the predicted segmentation mask P2 for calculating the unannotated image frame I2 is used to predict the optical flow F in the forward direction. 1→2 The difference between the deformation result mask under the action and the annotation mask M1 is used to connect the optical flow branch and the segmentation branch and exchange complementary information. The two tasks are optimized simultaneously during training, and the segmentation target frame is spread to the entire cardiac cycle sequence. The difference measurement function of the joint loss adopts the Weighted Dice Loss proposed in the myocardial segmentation pre-training part. The final calculation formula corresponding to the joint loss function is as follows:

[0154] L=α*L flow +β*L seg +γ*L joint

[0155] Among them, α, β, and γ are weight coefficients. In actual experiments, α, β, and γ are set to 1.0, 1.0, and 0.1 respectively; L is the final joint loss function; L joint is the initial joint loss function;

[0156] The initial joint loss function L joint The calculation formula is as follows:

[0157]

[0158] Among them, parameter l represents the subscript of different categories, i is used to indicate the i-th pixel, N is the total number of pixels, Indicates that the predicted segmentation mask P2 is used to predict the optical flow F 1→2 The deformation result mask under the action;

[0159] in:

[0160] and Represent the label value and predicted value of pixel i in category l respectively.

[0161] In this alternative embodiment,

[0162] It can be seen that in this optional embodiment, before the training set data is input into the joint learning framework, a data preprocessing process is set up for the training set data. Through image scaling and normalization processing, the consistency of the training set data is improved, which is beneficial to improving the training efficiency of subsequent execution framework training (including model training); this data enhancement is beneficial to improving the generalization of the model; at the same time, for the joint learning framework, the parameter values ​​of the relevant training parameters can be accurately set according to the training requirements of the actual framework, thereby improving the framework construction accuracy of the joint learning framework.

[0163] Embodiment 2

[0164] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of another method for processing myocardial region data based on a joint learning framework disclosed in an embodiment of the present invention. Figure 2 The described myocardial region data processing method based on the joint learning framework can be applied to a myocardial region data processing device based on the joint learning framework, and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the myocardial region data processing method based on the joint learning framework may include the following operations:

[0165] 201. Acquire cardiac image data for executing framework training, where the cardiac image data includes a plurality of frame images that have been processed by image annotation, and all frame images include at least ED frame images and ES frame images.

[0166] 202. Select multiple target frame images from the cardiac image data, where the target frame images are ED frame images or ES frame images.

[0167] 203. For each target frame image, take the target frame image as the first frame image and the subsequent frame image of the target frame image as the second frame image, and construct an image pair corresponding to the target frame image, which is recorded as binary data.

[0168] In an embodiment of the present invention, the number of subsequent frames corresponding to the subsequent frame image of the target frame image is a preset number of frames; the preset number of frames may be 1 frame, 2 frames, ..., n frames; further, when multiple copies of target frame images are actually acquired, the data may be arranged and combined and stored based on the cardiac cycle, for example, the multiple target frame images include multiple groups of frame images, each group of frame images corresponds to a cardiac cycle. Further, the selected target frame image is an ED frame image or an ES frame image, as the first frame image, and the subsequent frame image corresponding to the target frame image is a frame image belonging to the same cardiac cycle as the target frame image.

[0169] 204. Perform data partitioning on all the two-tuple data to obtain data partitioning results corresponding to all the two-tuple data.

[0170] In an embodiment of the present invention, the data division result includes validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training.

[0171] In an embodiment of the present invention, when multiple sets of binary data are actually determined, the number of all binary data may be 17739, of which 9223 sets of binary data have good image quality, 6448 sets of binary data have medium image quality, and 2068 sets of binary data have poor image quality. In order to maintain consistent performance on the validation set and the test set, the experiment in this section fixes both the validation set and the test set to 1500 examples, and sets the ratio of good quality: medium quality: poor quality to 8:5:2 in the data composition, and the remaining data are all used as training sets for training.

[0172] 205. Determine the data division result as target data, where the target data at least includes training set data for performing framework training.

[0173] 206. Input the training set data into a pre-built joint learning framework, and perform framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework.

[0174] 207. When the framework training result indicates that the joint learning framework training has reached convergence, it is determined that the training task for the joint learning framework is completed.

[0175] In the embodiment of the present invention, for other descriptions of steps 201-202 and steps 206-207, please refer to other specific descriptions of steps 101-102 and steps 103-104 in the first embodiment, which will not be repeated in the embodiment of the present invention.

[0176] It can be seen that the implementation Figure 2The described myocardial area data processing method based on a joint learning framework, when preparing training set data, uses the ED frame image / ES frame image as the first frame image, and the subsequent frame image of the ED frame image / ES frame image as the second frame image, to combine and construct image pairs, so as to divide the training set data according to the set data division requirements, thereby improving the reliability and accuracy of the preparation of the training set data.

[0177] In an optional embodiment, before performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework, the method further includes:

[0178] For each binary data, determine the segmentation mask true value corresponding to the first frame image in the binary data; add the segmentation mask true value corresponding to the binary data to the binary data to obtain the triplet data corresponding to the binary data;

[0179] Determine all triplet data as new training set data, and trigger the execution of the above-mentioned frame training of the joint learning framework according to the training set data to obtain the operation corresponding to the frame training result corresponding to the joint learning framework;

[0180] Wherein, when performing branch training on the segmentation branch, the branch training data input to the segmentation branch is the training set data including the binary data;

[0181] When performing branch training on the optical flow branch, the branch training data input to the optical flow branch is new training set data including triple data.

[0182] It can be seen that in this optional embodiment, for different training branches and different training tasks, the input training data can be adjusted as needed, wherein the pre-prepared training set data including binary data can be input for the segmentation branch, and the training set data including triplet data can be input for the optical flow branch; the triplet data adds an additional segmentation mask true value for constraining the unsupervised optical flow estimation, which is beneficial to improving the optical flow estimation effect of the optical flow branch, specifically improving the fineness of the optical flow, eliminating the sense of optical flow fragmentation generated by the optical flow estimation pre-training model in the pre-training stage, and improving the continuity of motion tracking; the precise training set data input is also beneficial to improving the framework training accuracy of the joint learning framework.

[0183] In another optional embodiment, the method of performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework specifically includes:

[0184] When performing branch training on the segmentation branch, for the target binary data currently input into the segmentation branch, a target segmentation processing operation is performed on the target binary data to obtain a target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on a pre-constructed myocardial segmentation pre-training model and segmentation loss calculation based on a preset segmentation loss function; the target segmentation processing result includes a myocardial segmentation result corresponding to the myocardial segmentation processing and a segmentation loss calculation result corresponding to the segmentation loss calculation;

[0185] When branch training is performed on the optical flow branch, for the target triplet data of the current input optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain a target estimation result corresponding to the target triplet data; the target optical flow estimation processing operation includes an optical flow estimation processing based on a pre-built optical flow estimation pre-training model and an optical flow loss calculation based on a preset optical flow loss function; the target estimation result includes an optical flow estimation result corresponding to the optical flow estimation processing and an optical flow loss calculation result corresponding to the optical flow loss calculation;

[0186] According to a preset joint loss function, a joint loss calculation is performed on the target segmentation result and the target estimation result to obtain a joint loss calculation result corresponding to the target segmentation result and the target estimation result;

[0187] Determine the target segmentation result, the target estimation result, and the joint loss calculation result as the framework training result corresponding to the joint learning framework;

[0188] Among them, when the segmentation loss calculation result, the optical flow loss calculation result and the joint loss calculation result indicate that the joint learning framework is trained to convergence, it is determined that the joint learning framework has completed training.

[0189] In this optional embodiment, the myocardial segmentation pre-training model is specifically a model that is improved and adjusted based on the traditional U-Net network; the improvements corresponding to the myocardial segmentation pre-training model are as follows: During the downsampling process, the encoder extracts features from the input data by downsampling four times. The number of feature map channels doubles each time the sample is downsampled. Except for the last layer, each downsampling block contains two 3*3 convolutional layers and a 2*2 maximum pooling layer. Each convolutional layer performs batch normalization and feature activation by adding InstanceNorm and LeakyRelu. The initial number of channels is set to 48, which will make our network wider and learn more details.

[0191] During the upsampling process, the decoder restores the size of the feature map through four upsampling operations, and then each upsampling operation reduces the number of feature channels to half of the original number, and concatenates it with the feature map of the contraction path in the channel dimension at the corresponding level to fuse the context information. In the last layer of the network, the output is mapped to the number of categories to be predicted through a convolutional layer of size . In this paper, the number of output prediction categories is set to 2, representing background pixels and myocardial pixels, respectively.

[0192] In this optional embodiment, the optical flow estimation pre-training model can specifically be an EchoFlowNet model. It should be noted that the myocardial segmentation pre-training model and the optical flow estimation pre-training model used in the joint learning framework are both models pre-trained by the model, and the two pre-trained models are further optimized by the joint learning framework.

[0193] In this optional embodiment, it should be noted that the myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image;

[0194] The segmentation loss calculation result corresponding to the target binary data is obtained by performing segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data through a preset segmentation loss function;

[0195] The optical flow estimation result corresponding to the target triplet data includes a forward optical flow result corresponding to the target triplet data;

[0196] The optical flow loss calculation result corresponding to the target triplet data is obtained by performing optical flow loss calculation on the forward optical flow result, the segmentation mask true value corresponding to the target triplet data, and the first frame image in the target triplet data through a preset optical flow loss function;

[0197] Before determining that the joint learning framework has completed the framework training, the segmentation loss calculation results, the optical flow loss calculation results, and the joint loss calculation results are used to perform parameter optimization and update on the branch parameters corresponding to the optical flow branch and the segmentation branch.

[0198] In this alternative embodiment, see Figure 6 , Figure 6 is a schematic diagram of a joint learning framework disclosed in an embodiment of the present invention. Specifically, during the joint training process, two branch networks are optimized simultaneously, corresponding to the segmentation network and the optical flow network. At the same time, for each iterative training of the joint learning framework, Figure 6As shown, the training set data of the current frame training input is recorded as the previous frame image (first frame image) I1 and the next frame image (second frame image) I2; the binary data is (I1, I2); the ternary data also includes the segmentation mask true value corresponding to the previous frame image I1, recorded as M1, and the corresponding ternary data is (I1, I2, M1). Correspondingly, the first preset segmentation mask corresponding to the previous frame image I1 is recorded as P1, and the first preset segmentation mask corresponding to the next frame image I2 is recorded as P2. The forward optical flow result corresponding to the target ternary data is recorded as F 1→2 .

[0199] It can be seen that in this optional embodiment, in the training stage of the joint learning framework, the input training data includes not only the image pair / binary data I1 and I2 composed of the previous and next frame images, but also the segmentation mask true value M1 corresponding to the previous frame image I1, which is used for the auxiliary calculation of the loss module (optical flow loss, joint loss, segmentation loss). In the optical flow branch, thanks to the constraint of the segmentation mask true value M1, the unsupervised optical flow loss L flow The optimization goal is shifted from global to local myocardial area.

[0200] In this optional embodiment, the unsupervised optical flow loss L flow It mainly includes two parts: photometric loss and smoothing loss. Among them, the traditional photometric loss combined with mask M1 only calculates the result on the myocardial area to ensure the accuracy of the local output optical flow. At the same time, a first-order smoothing loss is added on this basis to achieve local myocardial motion smoothing and refinement.

[0201] In this optional embodiment, the segmentation branch is built based on the myocardial segmentation pre-training model, and the predicted segmentation masks P1 and P2 on the images I1 and I2 are output respectively, and the segmentation loss L seg A supervised calculation is performed based on the predicted mask P1 and the corresponding true mask M1 to ensure the output validity of the segmentation branch network. Through the setting of this joint learning framework, not only the smooth continuity of the motion is improved, but also the loss in the local myocardial area is further reduced. P2 is the predicted segmentation mask of the non-ED / ES sequence frame image. There is no corresponding true value annotation in the training. However, due to the consistency of the motion of the image and the mask, P2 is effective in predicting the optical flow F. 1→2 The deformed result mask under the action should be consistent with M1; based on this constraint, a joint loss is proposed to connect the optical flow branch and the segmentation branch, and the segmentation target frame is spread to the entire cardiac cycle sequence by exchanging additional complementary information during training.

[0202] Embodiment 3

[0203] See also Figure 3 , Figure 3: is a schematic diagram of the structure of a myocardial region data processing device based on a joint learning framework disclosed in an embodiment of the present invention. The myocardial region data processing device based on a joint learning framework can be a myocardial region data processing terminal, device, system or server based on a joint learning framework. The server can be a local server, a remote server, or a cloud server (also known as a cloud server). When the server is a non-cloud server, the non-cloud server can communicate with the cloud server, which is not limited in the embodiment of the present invention. Figure 3 As shown, the myocardial region data processing device based on the joint learning framework may include an acquisition module 301, an image processing module 302, a data input module 303, and a framework training module 304, wherein:

[0204] An acquisition module 301 is used to acquire cardiac image data for performing framework training, where the cardiac image data includes a plurality of frame images that have been processed by image annotation, and all frame images include at least ED frame images and ES frame images;

[0205] The image processing module 302 is used to perform image processing operations on the cardiac image data to obtain target data, wherein the target data at least includes training set data for performing framework training; the training set data includes multiple example data, each example data is a two-tuple data, and each two-tuple data consists of a front image frame and a rear image frame corresponding to the front image frame;

[0206] A data input module 303 is used to input the training set data into the pre-built joint learning framework;

[0207] The framework training module 304 is used to perform framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework training has converged, it is determined that the training task for the joint learning framework is completed;

[0208] Among them, the joint learning framework includes a segmentation branch, an optical flow branch and a loss calculation module; the segmentation branch is used to perform myocardial area segmentation extraction for each example data; the optical flow branch is used to determine the motion optical flow information of all sub-data on the myocardium; the loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch and a joint loss function, and the joint loss function is used to connect the segmentation branch and the optical flow branch.

[0209] It can be seen that the implementation Figure 3The described myocardial region data processing device based on the joint learning framework implements preprocessing of cardiac image data before inputting the training set data into the joint learning framework, so that the training data input into the joint learning framework is more accurate, which is beneficial to improving the training efficiency of the joint learning framework; in addition, based on the constructed joint learning framework, it is possible to realize the synchronous training and optimization of the dual models of the myocardial segmentation pre-training model (corresponding to the segmentation branch) and the optical flow estimation pre-training model (corresponding to the optical flow branch); further, the joint learning framework also sets a joint loss function, so that the semantic information brought by the myocardial segmentation pre-training model can provide physiological shape constraints for the optical flow estimation task performed by the optical flow estimation pre-training model, thereby realizing the optical flow refinement on the local myocardial region, which is beneficial to improving the fineness of the output optical flow corresponding to the optical flow branch, and reducing the granularity and sense of fragmentation corresponding to the optical flow output by the optical flow branch; at the same time, the optical flow estimation realized by the optical flow branch can add timing constraints to the segmentation task of the segmentation branch, and diffuse the segmentation target frame processed by the segmentation branch to the entire cardiac cycle sequence through motion consistency, so that the annotation data input to the segmentation branch does not need to be fully myocardially annotated, thereby reducing the workload of myocardial annotation and image processing costs.

[0210] In an optional embodiment, the image processing module 302 performs image processing operations on the cardiac image data to obtain target data in a manner specifically including:

[0211] Selecting a plurality of target frame images from the cardiac image data, wherein the target frame images are ED frame images or ES frame images;

[0212] For each target frame image, the target frame image is taken as the first frame image, and the subsequent frame image of the target frame image is taken as the second frame image, and an image pair corresponding to the target frame image is constructed and recorded as a two-tuple data; the subsequent frame number corresponding to the subsequent frame image of the target frame image is a preset frame number;

[0213] Perform data partitioning on all the two-tuple data to obtain data partitioning results corresponding to all the two-tuple data, wherein the data partitioning results include validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training;

[0214] The data division result is determined as the target data.

[0215] It can be seen that in this optional embodiment, when preparing training set data, the ED frame image / ES frame image is used as the first frame image, and the subsequent frame image of the ED frame image / ES frame image is used as the second frame image, and image pairs are combined and constructed, so that the training set data is divided according to the set data division requirements, thereby improving the reliability and accuracy of the preparation of the training set data.

[0216] In another optional embodiment, Figure 4 As shown, the device also includes a determination module 305, wherein

[0217] The determination module 305 is used for determining, for each pair of binary data, a true value of a segmentation mask corresponding to the first frame image in the binary data before the framework training module 304 performs framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework; and adding the true value of the segmentation mask corresponding to the binary data to the binary data to obtain a triplet of data corresponding to the binary data;

[0218] The determination module 305 is further used to determine all triplet data as new training set data, and trigger the framework training module 304 to perform the above-mentioned framework training on the joint learning framework according to the training set data to obtain the operation corresponding to the framework training result corresponding to the joint learning framework;

[0219] Wherein, when performing branch training on the segmentation branch, the branch training data input to the segmentation branch is the training set data including the binary data;

[0220] When performing branch training on the optical flow branch, the branch training data input to the optical flow branch is new training set data including triple data.

[0221] It can be seen that in this optional embodiment, for different training branches and different training tasks, the input training data can be adjusted as needed, wherein the pre-prepared training set data including binary data can be input for the segmentation branch, and the training set data including triplet data can be input for the optical flow branch; the triplet data adds an additional segmentation mask true value for constraining the unsupervised optical flow estimation, which is beneficial to improving the optical flow estimation effect of the optical flow branch, specifically improving the fineness of the optical flow, eliminating the sense of optical flow fragmentation generated by the optical flow estimation pre-training model in the pre-training stage, and improving the continuity of motion tracking; the precise training set data input is also beneficial to improving the framework training accuracy of the joint learning framework.

[0222] In another optional embodiment, the framework training module 304 performs framework training on the joint learning framework according to the training set data, and the method of obtaining the framework training result corresponding to the joint learning framework specifically includes:

[0223] When performing branch training on the segmentation branch, for the target binary data currently input into the segmentation branch, a target segmentation processing operation is performed on the target binary data to obtain a target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on a pre-constructed myocardial segmentation pre-training model and segmentation loss calculation based on a preset segmentation loss function; the target segmentation processing result includes a myocardial segmentation result corresponding to the myocardial segmentation processing and a segmentation loss calculation result corresponding to the segmentation loss calculation;

[0224] When branch training is performed on the optical flow branch, for the target triplet data of the current input optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain a target estimation result corresponding to the target triplet data; the target optical flow estimation processing operation includes an optical flow estimation processing based on a pre-built optical flow estimation pre-training model and an optical flow loss calculation based on a preset optical flow loss function; the target estimation result includes an optical flow estimation result corresponding to the optical flow estimation processing and an optical flow loss calculation result corresponding to the optical flow loss calculation;

[0225] According to a preset joint loss function, a joint loss calculation is performed on the target segmentation result and the target estimation result to obtain a joint loss calculation result corresponding to the target segmentation result and the target estimation result;

[0226] Determine the target segmentation result, the target estimation result, and the joint loss calculation result as the framework training result corresponding to the joint learning framework;

[0227] Among them, when the segmentation loss calculation result, the optical flow loss calculation result and the joint loss calculation result indicate that the joint learning framework is trained to convergence, it is determined that the joint learning framework has completed training.

[0228] In this optional embodiment, the myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image;

[0229] The segmentation loss calculation result corresponding to the target binary data is obtained by performing segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data through a preset segmentation loss function;

[0230] The optical flow estimation result corresponding to the target triplet data includes a forward optical flow result corresponding to the target triplet data;

[0231] The optical flow loss calculation result corresponding to the target triplet data is obtained by performing optical flow loss calculation on the forward optical flow result, the segmentation mask true value corresponding to the target triplet data, and the first frame image in the target triplet data through a preset optical flow loss function;

[0232] Before determining that the joint learning framework has completed the framework training, the segmentation loss calculation results, the optical flow loss calculation results, and the joint loss calculation results are used to perform parameter optimization and update on the branch parameters corresponding to the optical flow branch and the segmentation branch.

[0233] It can be seen that in this optional embodiment, in the training stage of the joint learning framework, the input training data includes not only the image pair / binary data I1 and I2 composed of the previous and next frame images, but also the segmentation mask true value M1 corresponding to the previous frame image I1, which is used for the auxiliary calculation of the loss module (optical flow loss, joint loss, segmentation loss). In the optical flow branch, thanks to the constraint of the segmentation mask true value M1, the unsupervised optical flow loss L flow The optimization goal is shifted from global to local myocardial area.

[0234] In another optional embodiment, Figure 4 As shown, the device also includes a data preprocessing module 306 and an updating module 307, wherein:

[0235] The data preprocessing module 306 is used to perform data preprocessing on the training set data before the data input module 303 inputs the training set data into the pre-built joint learning framework to obtain a data preprocessing result corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing and data enhancement, and the data enhancement includes at least random brightness transformation and / or random affine transformation;

[0236] An updating module 307 is used to update the training set data according to the data preprocessing result, and trigger the execution of the above-mentioned operation corresponding to inputting the training set data into the pre-built joint learning framework;

[0237] And, the pre-built federated learning framework is built as follows:

[0238] Select target training models that meet the training conditions. The target training models include myocardial segmentation output model and optical flow estimation output model. The target training models have all passed the preliminary model training.

[0239] Determine training parameters and parameter values ​​for the target training model; the training parameters include at least one of the total number of training iterations, batch size, and optimizer parameters for the target training model; the optimizer parameters include a learning rate and a weight decay rate;

[0240] Determine a target loss function for the target training model, the target loss function including a segmentation loss function corresponding to the myocardial segmentation output model, an optical flow loss function corresponding to the optical flow estimation output model, and a joint loss function for connecting the myocardial segmentation output model and the optical flow estimation output model;

[0241] A joint learning framework is constructed based on the target training model, training parameters and their parameter values, and the target loss function.

[0242] In this optional embodiment, the calculation formula corresponding to the above segmentation loss function is as follows:

[0243] L seg =α*L WD +β*L CE

[0244] Among them, α and β are weight coefficients; L seg Refers to the segmentation loss function; L WD Refers to the Weighted Dice Loss function; L CE Refers to the CrossEntropy Loss function;

[0245] The calculation formula corresponding to the optical flow loss function is as follows:

[0246]

[0247] Among them, p represents the image pixel, i and j represent the pixel coordinates, d represents the partial derivatives in the x and y directions, Ψ is the robustness penalty function, q, ε, and α are the adjustable parameters of the optical flow loss function; λ is the weight coefficient:

[0248] The calculation formula corresponding to the joint loss function is as follows:

[0249] L=α*L flow +β*L seg +γ*L joint

[0250] Among them, α, β, and γ are weight coefficients; L is the final joint loss function; L joint is the initial joint loss function;

[0251] The initial joint loss function L joint The calculation formula is as follows:

[0252]

[0253] Among them, parameter l represents the subscript of different categories, i is used to indicate the i-th pixel, N is the total number of pixels, Indicates that the predicted segmentation mask P2 is used to predict the optical flow F 1→2 The deformation result mask under the action;

[0254] in:

[0255] and Represent the label value and predicted value of pixel i in category l respectively.

[0256] It can be seen that in this optional embodiment, before the training set data is input into the joint learning framework, a data preprocessing process is set up for the training set data. Through image scaling and normalization processing, the consistency of the training set data is improved, which is beneficial to improving the training efficiency of subsequent execution framework training (including model training); this data enhancement is beneficial to improving the generalization of the model; at the same time, for the joint learning framework, the parameter values ​​of the relevant training parameters can be accurately set according to the training requirements of the actual framework, thereby improving the framework construction accuracy of the joint learning framework.

[0257] Embodiment 4

[0258] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of the structure of another myocardial region data processing device based on a joint learning framework disclosed in an embodiment of the present invention. Figure 5 As shown, the myocardial region data processing device based on the joint learning framework may include:

[0259] A memory 401 storing executable program codes;

[0260] a processor 402 coupled to the memory 401;

[0261] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the myocardial region data processing method based on the joint learning framework described in the first embodiment of the present invention or the second embodiment of the present invention.

[0262] Embodiment 5

[0263] An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps in the myocardial region data processing method based on a joint learning framework described in Embodiment 1 or Embodiment 2 of the present invention.

[0264] Embodiment 6

[0265] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the myocardial region data processing method based on a joint learning framework described in Example 1 or Example 2.

[0266] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0267] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0268] Finally, it should be noted that the myocardial area data processing method and device based on a joint learning framework disclosed in the embodiment of the present invention discloses only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for processing myocardial region data based on a joint learning framework, characterized in that: The method comprises: Acquire cardiac image data for performing framework training, wherein the cardiac image data includes a plurality of frame images that have been processed by image annotation, and all the frame images include at least ED frame images and ES frame images; Performing an image processing operation on the cardiac image data to obtain target data, wherein the target data at least includes training set data for performing the framework training; the training set data includes multiple examples of data, each example of sub-data is a two-tuple data, and each of the two-tuple data consists of a front image frame and a rear image frame corresponding to the front image frame; Inputting the training set data into a pre-constructed joint learning framework, and performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, determining that the training task for the joint learning framework is completed; Among them, the joint learning framework includes a segmentation branch, an optical flow branch and a loss calculation module; the segmentation branch is used to perform myocardial area segmentation and extraction for each example of the sub-data; the optical flow branch is used to determine the motion optical flow information of all the sub-data on the myocardium; the loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch and a joint loss function, and the joint loss function is used to connect the segmentation branch and the optical flow branch.

2. The myocardial region data processing method based on the joint learning framework according to claim 1, characterized in that: The performing of an image processing operation on the cardiac image data to obtain target data comprises: Selecting a plurality of target frame images from the cardiac image data, wherein the target frame images are the ED frame images or the ES frame images; For each target frame image, the target frame image is taken as the first frame image, and the subsequent frame image of the target frame image is taken as the second frame image, to construct an image pair corresponding to the target frame image, which is recorded as a binary data; the subsequent frame number corresponding to the subsequent frame image of the target frame image is a preset frame number; Performing data partitioning on all the two-tuple data to obtain data partitioning results corresponding to all the two-tuple data, wherein the data partitioning results include validation set data for performing framework testing, test set data for performing framework testing, and training set data for performing framework training; The data division result is determined as target data.

3. The myocardial region data processing method based on the joint learning framework according to claim 2, characterized in that: Before performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework, the method further includes: For each of the two-tuple data, determine the segmentation mask true value corresponding to the first frame image in the two-tuple data; add the segmentation mask true value corresponding to the two-tuple data to the two-tuple data to obtain the three-tuple data corresponding to the two-tuple data; Determine all the triplet data as new training set data, and trigger the execution of the operation of performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework; Wherein, when performing branch training on the segmentation branch, the branch training data input to the segmentation branch is the training set data including the binary data; When performing branch training on the optical flow branch, the branch training data input to the optical flow branch is the new training set data including the triplet data.

4. The myocardial region data processing method based on a joint learning framework according to claim 3, characterized in that: The performing framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework includes: When branch training is performed on the segmentation branch, for the target binary data currently input into the segmentation branch, a target segmentation processing operation is performed on the target binary data to obtain a target segmentation result corresponding to the target binary data; the target segmentation processing operation includes myocardial segmentation processing based on a pre-constructed myocardial segmentation pre-training model and segmentation loss calculation based on a preset segmentation loss function; the target segmentation processing result includes a myocardial segmentation result corresponding to the myocardial segmentation processing and a segmentation loss calculation result corresponding to the segmentation loss calculation; When branch training is performed on the optical flow branch, for the target triplet data currently input into the optical flow branch, a target optical flow estimation processing operation is performed on the target triplet data to obtain a target estimation result corresponding to the target triplet data; the target optical flow estimation processing operation includes an optical flow estimation processing based on a pre-built optical flow estimation pre-training model and an optical flow loss calculation based on a preset optical flow loss function; the target estimation result includes an optical flow estimation result corresponding to the optical flow estimation processing and an optical flow loss calculation result corresponding to the optical flow loss calculation; According to a preset joint loss function, performing a joint loss calculation on the target segmentation result and the target estimation result to obtain a joint loss calculation result corresponding to the target segmentation result and the target estimation result; Determine the target segmentation result, the target estimation result, and the joint loss calculation result as a framework training result corresponding to the joint learning framework; When the segmentation loss calculation result, the optical flow loss calculation result, and the joint loss calculation result indicate that the joint learning framework has been trained to convergence, it is determined that the joint learning framework has completed training.

5. The myocardial region data processing method based on a joint learning framework according to claim 4, characterized in that: The myocardial segmentation result corresponding to the target binary data includes a first preset segmentation mask corresponding to the first frame image in the target binary data and a second preset segmentation mask corresponding to the second frame image; The segmentation loss calculation result corresponding to the target binary data is obtained by performing the segmentation loss calculation on the first preset segmentation mask and the true value of the segmentation mask corresponding to the target binary data through the preset segmentation loss function; The optical flow estimation result corresponding to the target triplet data includes a forward optical flow result corresponding to the target triplet data; The optical flow loss calculation result corresponding to the target triplet data is obtained by performing the optical flow loss calculation on the forward optical flow result, the segmentation mask true value corresponding to the target triplet data, and the first frame image in the target triplet data through the preset optical flow loss function; Before determining that the joint learning framework has completed framework training, the segmentation loss calculation result, the optical flow loss calculation result and the joint loss calculation result are used to perform parameter optimization and update on branch parameters corresponding to the optical flow branch and the segmentation branch.

6. The myocardial region data processing method based on a joint learning framework according to claim 1, 2, 4 or 5, characterized in that: Before inputting the training set data into the pre-built joint learning framework, the method further includes: Performing data preprocessing on the training set data to obtain data preprocessing results corresponding to the training set data; the data preprocessing includes image scaling based on a preset image size, normalization processing, and data enhancement, and the data enhancement includes at least random brightness transformation and / or random affine transformation; The training set data is updated according to the data preprocessing result, and the corresponding operation of inputting the training set data into the pre-built joint learning framework is triggered to be executed; And, the pre-built federated learning framework is built as follows: Selecting target training models that meet the training conditions, the target training models include a myocardial segmentation output model and an optical flow estimation output model; the target training models have all preliminarily passed model training; Determine the training parameters and parameter values ​​for the target training model; the training parameters include at least one of the total number of training iterations, batch size, and optimizer parameters for the target training model; the optimizer parameters include a learning rate and a weight decay rate; Determine a target loss function for the target training model, wherein the target loss function includes a segmentation loss function corresponding to the myocardial segmentation output model, an optical flow loss function corresponding to the optical flow estimation output model, and a joint loss function for connecting the myocardial segmentation output model and the optical flow estimation output model; A joint learning framework is constructed based on the target training model, the training parameters and their parameter values, and the target loss function.

7. The myocardial region data processing method based on a joint learning framework according to claim 6, characterized in that: The calculation formula corresponding to the segmentation loss function is as follows: L seg =α*L WD +β*L CE Among them, α and β are weight coefficients; L seg Refers to the segmentation loss function; L WD Refers to the Weighted Di ce Loss function; L CE Refers to the CrossEntropy Loss function; The calculation formula corresponding to the optical flow loss function is as follows: Among them, p represents the image pixel point, i and j represent the pixel coordinates, d represents the partial derivatives in the x and y directions, Ψ is the robustness penalty function, q, ε, and α are the adjustable parameters of the optical flow loss function respectively; λ is the weight coefficient: The calculation formula corresponding to the joint loss function is as follows: L=α*L flow +β*L seg +γ*L joint Among them, α, β, and γ are weight coefficients; L is the final joint loss function; L joint is the initial joint loss function; The initial joint loss function L joint The calculation formula is as follows: Among them, parameter l represents the subscript of different categories, i is used to indicate the i-th pixel, N is the total number of pixels, Indicates that the predicted segmentation mask P2 is used in predicting the optical flow F 1→2 The deformation result mask under the action; in: and Represent the label value and predicted value of pixel i in category l respectively.

8. A myocardial region data processing device based on a joint learning framework, characterized in that: The device comprises: An acquisition module, used for acquiring cardiac image data for executing framework training, wherein the cardiac image data includes a plurality of frame images that have been processed by image annotation, and all the frame images include at least ED frame images and ES frame images; An image processing module, configured to perform image processing operations on the cardiac image data to obtain target data, wherein the target data at least includes training set data for performing the framework training; the training set data includes multiple examples of data, each example of sub-data is a binary data, and each binary data is composed of a front image frame and a rear image frame corresponding to the front image frame; A data input module, used for inputting the training set data into a pre-built joint learning framework; A framework training module, used to perform framework training on the joint learning framework according to the training set data to obtain a framework training result corresponding to the joint learning framework; when the framework training result indicates that the joint learning framework has been trained to convergence, determine that the training task for the joint learning framework is completed; Among them, the joint learning framework includes a segmentation branch, an optical flow branch and a loss calculation module; the segmentation branch is used to perform myocardial area segmentation and extraction for each example of the sub-data; the optical flow branch is used to determine the motion optical flow information of all the sub-data on the myocardium; the loss calculation module includes a segmentation loss function corresponding to the segmentation branch, an optical flow loss function corresponding to the optical flow branch and a joint loss function, and the joint loss function is used to connect the segmentation branch and the optical flow branch.

9. A myocardial region data processing device based on a joint learning framework, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the myocardial region data processing method based on a joint learning framework as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the myocardial region data processing method based on the joint learning framework as described in any one of claims 1-7.

Images