A method and device for locating cardiac anatomical feature points based on heat map regression

Abstract

The present invention relates to the field of image processing, and discloses a method and device for locating cardiac anatomical feature points based on heat map regression, the method comprising: obtaining target image data to be analyzed; performing pre-processing operations on the target image data to obtain pre-processing results, the pre-processing operations are used to adjust the target image data to a tensor of a preset size; performing target processing on the pre-processing results according to a deep learning model to obtain corresponding predicted heat map data, the target processing operations at least include convolution processing, standardization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation; performing post-processing on the predicted heat map data according to view classification information to obtain multiple feature coordinate data; wherein the post-processing includes non-maximum suppression calculation, image scaling processing and coordinate selection; all feature coordinate data are used to determine the cardiac anatomical structure. It can be seen that the implementation of the present invention can improve the positioning accuracy of cardiac anatomical feature points.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for locating cardiac anatomical feature points based on heat map regression. Background Art

[0002] In the process of quantitative analysis of the heart based on echocardiography, the positioning of cardiac anatomical feature points is a very critical step. In the conventional processing flow, professionals manually position and annotate the feature points of the echocardiogram. For professionals, positioning and annotating feature points of echocardiograms is a time-consuming and laborious task, and the manpower and time costs required are high. In addition, due to inter-observer variations and intra-observer variations, different annotators also have different positioning and annotation results for the same echocardiogram. The differences and inconsistencies in their positioning and annotation results will eventually limit the use scenarios of the positioning and annotation results of the echocardiogram. In addition, if the positioning and annotation of the echocardiogram is performed by professionals with little experience, the accuracy of the positioning and annotation of the echocardiogram will be reduced. It can be seen that it is particularly important to design an algorithm to realize the automatic positioning and annotation of cardiac anatomical feature points to improve the accuracy of the positioning and annotation of cardiac anatomical feature points. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method and device for locating cardiac anatomical feature points based on heat map regression, which can improve the accuracy of locating and marking cardiac anatomical feature points.

[0004] In order to solve the above technical problems, the present invention discloses, in a first aspect, a method for locating cardiac anatomical feature points based on heat map regression, the method comprising:

[0005] Acquire target image data to be analyzed, wherein the target image data includes sub-image data corresponding to a plurality of frames of ventricular end-diastole and ventricular end-systole;

[0006] Performing a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data, wherein the pre-processing operation includes an image size conversion operation, and the pre-processing operation is used to adjust the target image data to a tensor of a preset size;

[0007] Performing a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model and obtain predicted heat map data corresponding to the pre-processing result, wherein the target processing operation at least includes convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation; the deep learning model is an improved model that optimizes the downsampling processing and adds coordinate information to the pre-processing result;

[0008] According to the preset view classification information, a preset post-processing operation is performed on the predicted heat map data to obtain a plurality of feature coordinate data corresponding to the predicted heat map data;

[0009] Wherein, the post-processing operation includes non-maximum suppression calculation, image scaling processing and coordinate selection operation; all the feature coordinate data include multiple feature point coordinates, and all the feature point coordinates are used to determine the anatomical structure of the heart.

[0010] As an optional implementation manner, in the first aspect of the present invention, performing a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data includes:

[0011] For each frame of the sub-image data, determining a first image size of the sub-image data of the frame;

[0012] Performing bilinear interpolation scaling processing on the frame sub-image data to scale the frame sub-image data from the first image size to a second image size, thereby obtaining a scaling processing result corresponding to the frame sub-image data;

[0013] Inputting the scaling processing result corresponding to the frame sub-image data into a preset exchange tensor channel to convert the frame sub-image data from the second image size to a third image size, thereby obtaining a tensor conversion result corresponding to the frame sub-image data;

[0014] Performing normalization processing and standardization processing on the tensor conversion result corresponding to the frame sub-image data in sequence to obtain a standardization processing result corresponding to the frame sub-image data;

[0015] The standardized processing results corresponding to all the sub-image data are determined as pre-processing results corresponding to the target image data.

[0016] As an optional implementation, in the first aspect of the present invention, performing a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model and obtain predicted heat map data corresponding to the pre-processing result includes:

[0017] Performing a preset data enhancement operation on the pre-processing result to obtain a data enhancement result corresponding to the pre-processing result, wherein the data enhancement operation includes an affine transformation and a pixel adjustment operation; the image parameter adjusted by the pixel adjustment operation includes at least one of brightness, contrast and saturation;

[0018] Performing a preset coordinate adding operation on the data enhancement result to obtain a coordinate adding result corresponding to the data enhancement result, wherein the coordinate adding operation includes adding coordinate information based on CoordConv;

[0019] Input the coordinate adding result into a preset deep learning model, so as to perform preset convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation on the coordinate adding result through the deep learning model, and obtain a model processing result corresponding to the coordinate adding result, each of the model processing results includes model output data corresponding to the coordinate adding result and a loss value corresponding to the loss calculation;

[0020] The target loss value and its corresponding target model output data are determined as predicted heat map data corresponding to the pre-processing result; wherein the target loss value is less than a preset loss threshold, and the target loss value is used to indicate that the deep learning model has completed model training.

[0021] As an optional implementation, in the first aspect of the present invention, the deep learning model includes a downsampling network, the downsampling network is used to perform the downsampling process, and the downsampling network is specifically a ResNet network;

[0022] Each frame of the sub-image data includes at least one set of feature point coordinates corresponding to the sub-image data;

[0023] After acquiring the target image data, the method further includes:

[0024] For each piece of the sub-image data, the coordinate probabilities corresponding to the coordinates of all the feature points in the sub-image data are calculated according to a preset probability conversion formula, and a pseudo-probability map corresponding to the sub-image data is generated according to all the coordinate probabilities;

[0025] Perform a bilinear interpolation scaling operation on the pseudo probability map corresponding to the sub-image data to obtain a true value heat map corresponding to the sub-image data;

[0026] All of the true value heat maps are used to compare with all of the predicted heat map data to evaluate and / or optimize the performance of the deep learning model in executing data prediction.

[0027] As an optional implementation, in the first aspect of the present invention, the preset view classification information includes a plurality of view categories; each of the sub-image data has at least one of the view categories corresponding thereto, and different view categories are used to indicate that the poses of the slices of the sub-image data are collected at different positions;

[0028] The step of performing a preset post-processing operation on the predicted heat map data according to the preset view classification information to obtain a plurality of feature coordinate data corresponding to the predicted heat map data includes:

[0029] According to a preset NMS calculation, a non-maximum suppression calculation is performed on the predicted heat map data belonging to the same view category to obtain an NMS calculation result corresponding to the predicted heat map data;

[0030] Performing bilinear scaling processing on the NMS calculation result to obtain a target scaling result corresponding to the NMS calculation result; the image size corresponding to the target scaling result is consistent with the initial image size, and the initial image size is the size of the target image data before performing the pre-processing operation;

[0031] According to a preset number, a plurality of target feature points and their feature coordinates corresponding to the preset number are determined from the target scaling result as a plurality of feature coordinate data corresponding to the predicted heat map data.

[0032] As an optional implementation, in the first aspect of the present invention, after determining a plurality of feature coordinate data corresponding to the predicted heat map data, the method further includes:

[0033] According to the preset feature evaluation parameters, positioning evaluation is performed on each of the feature coordinate data to obtain a positioning evaluation result corresponding to each of the feature coordinate data, and the positioning evaluation result corresponding to each of the feature coordinate data is used to determine the feature point positioning effect of the feature coordinate data; the feature evaluation parameters include feature point detection rate and average point error;

[0034] The feature point detection rate is used to represent the probability of each feature coordinate data being detected, and the calculation formula corresponding to the feature point detection rate is:

[0035]

[0036] Among them, the i-th sample image I (i) Through the method described above, feature points, image I (i) The number of true feature points in is N (i) ;

[0037] The average point error is used to represent the average Euclidean distance between the successfully detected feature points and the true feature points. The calculation formula corresponding to the average point error is:

[0038]

[0039] Among them, the above They are respectively represented as the jth pair of true feature point coordinates and predicted feature point coordinates on the i-th sample matching.

[0040] As an optional implementation, in the first aspect of the present invention, the coordinate adding operation is used to add the corresponding coordinate information to the data enhancement result, and the calculation formula corresponding to the CoordConv in the horizontal axis direction is:

[0041]

[0042] Wherein, the Coordx is used to represent the horizontal axis coordinate information corresponding to the position information channel constructed by the CoordConv, and the size corresponding to the position information channel is recorded as H′×W′. i,j Used to represent the value of any point on the position information channel;

[0043] And, when the loss calculation is performed through the deep learning network, the loss function used includes a pre-designed Weighted Smooth L1 Loss function and a Focal Smooth L1 Loss function;

[0044] The corresponding calculation formula of the Weighted Smooth L1 Loss function is:

[0045]

[0046] in, is the predicted heat map corresponding to the predicted heat map data, H is the true value heat map corresponding to the predicted heat map data, M=I(H d ≥0.2), I is the indicative function; H d is obtained by performing a 3×3 grayscale expansion operation on H; W is a size hyperparameter used to control the importance of the foreground point, and W·M+1 is the corresponding weight term, represents the Hadamard product;

[0047] The calculation formula corresponding to the Focal Smooth L1 Loss function is:

[0048]

[0049] Among them, y, are respectively the true value heat map corresponding to the predicted heat map data and the value of the predicted heat map at the same position; Thres is the threshold hyperparameter used to control the judgment of foreground points and background points; γ is the focusing hyperparameter.

[0050] The second aspect of the present invention discloses a cardiac anatomical feature point positioning device based on heat map regression, the device comprising:

[0051] An acquisition module, used for acquiring target image data to be analyzed, wherein the target image data includes sub-image data corresponding to a plurality of frames of ventricular end-diastole and ventricular end-systole;

[0052] A pre-processing module, used to perform a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data, wherein the pre-processing operation includes an image size conversion operation, and the pre-processing operation is used to adjust the target image data to a tensor of a preset size;

[0053] A heat map processing module, used to perform a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model and obtain predicted heat map data corresponding to the pre-processing result, wherein the target processing operation at least includes convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation; the deep learning model is an improved model that optimizes the downsampling processing and adds coordinate information to the pre-processing result;

[0054] A post-processing module, configured to perform a preset post-processing operation on the predicted heat map data according to preset view classification information, to obtain a plurality of feature coordinate data corresponding to the predicted heat map data;

[0055] Wherein, the post-processing operation includes non-maximum suppression calculation, image scaling processing and coordinate selection operation; all the feature coordinate data include multiple feature point coordinates, and all the feature point coordinates are used to determine the anatomical structure of the heart.

[0056] As an optional implementation, in the second aspect of the present invention, the pre-processing module performs a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data, specifically including:

[0057] For each frame of the sub-image data, determining a first image size of the sub-image data of the frame;

[0058] Performing bilinear interpolation scaling processing on the frame sub-image data to scale the frame sub-image data from the first image size to a second image size, thereby obtaining a scaling processing result corresponding to the frame sub-image data;

[0059] Inputting the scaling processing result corresponding to the frame sub-image data into a preset exchange tensor channel to convert the frame sub-image data from the second image size to a third image size, thereby obtaining a tensor conversion result corresponding to the frame sub-image data;

[0060] Performing normalization processing and standardization processing on the tensor conversion result corresponding to the frame sub-image data in sequence to obtain a standardization processing result corresponding to the frame sub-image data;

[0061] The standardized processing results corresponding to all the sub-image data are determined as pre-processing results corresponding to the target image data.

[0062] As an optional implementation, in the second aspect of the present invention, the heat map processing module performs a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model, and the method of obtaining the predicted heat map data corresponding to the pre-processing result specifically includes:

[0063] Performing a preset data enhancement operation on the pre-processing result to obtain a data enhancement result corresponding to the pre-processing result, wherein the data enhancement operation includes an affine transformation and a pixel adjustment operation; the image parameter adjusted by the pixel adjustment operation includes at least one of brightness, contrast and saturation;

[0064] Performing a preset coordinate adding operation on the data enhancement result to obtain a coordinate adding result corresponding to the data enhancement result, wherein the coordinate adding operation includes adding coordinate information based on CoordConv;

[0065] Input the coordinate adding result into a preset deep learning model, so as to perform preset convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation on the coordinate adding result through the deep learning model, and obtain a model processing result corresponding to the coordinate adding result, wherein the model processing result includes model output data corresponding to the coordinate adding result and a loss value corresponding to the loss calculation;

[0066] The target loss value and its corresponding target model output data are determined as predicted heat map data corresponding to the pre-processing result; wherein the target loss value is less than a preset loss threshold, and the target loss value is used to indicate that the deep learning model has completed model training.

[0067] As an optional implementation, in the second aspect of the present invention, the deep learning model includes a downsampling network, the downsampling network is used to perform the downsampling process, and the downsampling network is specifically a ResNet network;

[0068] Each frame of the sub-image data includes at least one set of feature point coordinates corresponding to the sub-image data;

[0069] The device also includes:

[0070] A calculation module, for calculating, after acquiring the target image data, for each sub-image data, the coordinate probability corresponding to the coordinates of all the feature points in the sub-image data according to a preset probability conversion formula;

[0071] A generating module, used for generating a pseudo probability map corresponding to the sub-image data according to all the coordinate probabilities;

[0072] A scaling processing module is used to perform a bilinear interpolation scaling operation on the pseudo-probability map corresponding to the sub-image data to obtain a true value heat map corresponding to the sub-image data; all the true value heat maps are used to compare with all the predicted heat map data to evaluate and / or optimize the performance of the execution data prediction of the deep learning model.

[0073] As an optional implementation, in the second aspect of the present invention, the preset view classification information includes a plurality of view categories; each of the sub-image data has at least one of the view categories corresponding thereto, and different view categories are used to indicate that the sub-image data has different slice acquisition postures;

[0074] The post-processing module performs a preset post-processing operation on the predicted heat map data according to the preset view classification information to obtain a plurality of feature coordinate data corresponding to the predicted heat map data, specifically including:

[0075] According to a preset NMS calculation, a non-maximum suppression calculation is performed on the predicted heat map data belonging to the same view category to obtain an NMS calculation result corresponding to the predicted heat map data;

[0076] Performing bilinear scaling processing on the NMS calculation result to obtain a target scaling result corresponding to the NMS calculation result; the image size corresponding to the target scaling result is consistent with the initial image size, and the initial image size is the size of the target image data before performing the pre-processing operation;

[0077] According to a preset number, a plurality of target feature points and their feature coordinates corresponding to the preset number are determined from the target scaling result as a plurality of feature coordinate data corresponding to the predicted heat map data.

[0078] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0079] An evaluation module is used for, after determining a plurality of feature coordinate data corresponding to the predicted heat map data, performing positioning evaluation on each of the feature coordinate data according to preset feature evaluation parameters to obtain a positioning evaluation result corresponding to each of the feature coordinate data, wherein the positioning evaluation result corresponding to each of the feature coordinate data is used to determine the feature point positioning effect of the feature coordinate data; the feature evaluation parameters include feature point detection rate and average point error;

[0080] The feature point detection rate is used to represent the probability of each feature coordinate data being detected, and the calculation formula corresponding to the feature point detection rate is:

[0081]

[0082] Among them, the i-th sample image I (i) After the device is matched to feature points, image I (i) The number of true feature points in is N (i) ;

[0083] The average point error is used to represent the average Euclidean distance between the successfully detected feature points and the true feature points. The calculation formula corresponding to the average point error is:

[0084]

[0085] Among them, the above They are respectively represented as the jth pair of true feature point coordinates and predicted feature point coordinates on the i-th sample matching.

[0086] As an optional implementation, in the second aspect of the present invention, the coordinate adding operation is used to add the corresponding coordinate information to the data enhancement result, and the calculation formula corresponding to the CoordConv in the horizontal axis direction is:

[0087]

[0088] Wherein, the Coordx is used to represent the horizontal axis coordinate information corresponding to the position information channel constructed by the CoordConv, and the size corresponding to the position information channel is recorded as H′×W′. i,j Used to represent the value of any point on the position information channel;

[0089] And, when the loss calculation is performed through the deep learning network, the loss function used includes a pre-designed Weighted Smooth L1 Loss function and a Focal Smooth L1 Loss function;

[0090] The corresponding calculation formula of the Weighted Smooth L1 Loss function is:

[0091]

[0092] in, is the predicted heat map corresponding to the predicted heat map data, H is the true value heat map corresponding to the predicted heat map data, M=I(H d ≥0.2), I is the indicative function; H d is obtained by performing a 3×3 grayscale expansion operation on H; W is a size hyperparameter used to control the importance of the foreground point, and W·M+1 is the corresponding weight term, represents the Hadamard product;

[0093] The calculation formula corresponding to the Focal Smooth L1 Loss function is:

[0094]

[0095] Among them, y, are respectively the true value heat map corresponding to the predicted heat map data and the value of the predicted heat map at the same position; Thres is the threshold hyperparameter used to control the judgment of foreground points and background points; γ is the focusing hyperparameter.

[0096] The third aspect of the present invention discloses another cardiac anatomical feature point positioning device based on heat map regression, the device comprising:

[0097] A memory storing executable program code;

[0098] a processor coupled to the memory;

[0099] The processor calls the executable program code stored in the memory to execute the cardiac anatomical feature point positioning method based on heat map regression disclosed in the first aspect of the present invention.

[0100] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the cardiac anatomical feature point positioning method based on heat map regression disclosed in the first aspect of the present invention.

[0101] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0102] In an embodiment of the present invention, a method for locating cardiac anatomical feature points based on heat map regression is provided, the method comprising: acquiring target image data to be analyzed, the target image data comprising sub-image data corresponding to a plurality of frames of ventricular end-diastole and ventricular end-systole; performing a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data, the pre-processing operation comprising an image size conversion operation, the pre-processing operation being used to adjust the target image data to a tensor of a preset size; performing a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model to obtain a predicted heat map corresponding to the pre-processing result The target processing operation at least includes convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation; the deep learning model is an improved model that optimizes the downsampling processing and adds coordinate information to the pre-processing result; according to the preset view classification information, the predicted heat map data is subjected to the preset post-processing operation to obtain a plurality of feature coordinate data corresponding to the predicted heat map data; wherein the post-processing operation includes non-maximum suppression calculation, image scaling processing and coordinate selection operation; all feature coordinate data include a plurality of feature point coordinates, and all feature point coordinates are used to determine the anatomical structure of the heart. It can be seen that, by implementing the present invention, after acquiring the target image data including the end of ventricular systole / diastole that needs to be analyzed, the target image data can be pre-processed in sequence based on the preset feature point processing process to obtain a tensor of uniform size, so as to improve the image processing efficiency of the subsequent process, reduce the difficulty of image processing, and reduce the probability of image processing errors; then the pre-processing result corresponding to the target image data is input into the set deep learning model, so as to output the predicted heat map data corresponding to the pre-processing result, and the predicted heat map data includes the feature points corresponding to various cardiac anatomical structures and their predicted probability distribution; finally, through the post-processing process of the feature point processing process, combined with the preset view classification information, the required multiple feature coordinate data are determined as the feature point coordinates corresponding to various cardiac anatomical structures to be analyzed; wherein, through the improved deep learning model, its optimized downsampling network is conducive to improving the feature extraction capability of the model; the increase of its coordinate information improves the feature point coordinate relationship extraction capability of the model; that is, through this solution, the positioning accuracy and positioning effect of the cardiac anatomical feature points are further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0104] Figure 1 It is a flow chart of a method for locating cardiac anatomical feature points based on heat map regression disclosed in an embodiment of the present invention;

[0105] Figure 2 It is a flow chart of another method for locating cardiac anatomical feature points based on heat map regression disclosed in an embodiment of the present invention;

[0106] Figure 3 It is a structural schematic diagram of a cardiac anatomical feature point positioning device based on heat map regression disclosed in an embodiment of the present invention;

[0107] Figure 4 It is a structural schematic diagram of another cardiac anatomical feature point positioning device based on heat map regression disclosed in an embodiment of the present invention;

[0108] Figure 5 It is a structural schematic diagram of another cardiac anatomical feature point positioning device based on heat map regression disclosed in an embodiment of the present invention;

[0109] Figure 6 It is a structural schematic diagram of another method for locating cardiac anatomical feature points based on heat map regression disclosed in an embodiment of the present invention;

[0110] Figure 7 It is a schematic diagram of the architecture corresponding to an improved deep learning network disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0111] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0112] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or ends.

[0113] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0114] The present invention discloses a method and device for locating cardiac anatomical feature points based on heat map regression. After acquiring target image data including the end of ventricular contraction / diastole that needs to be analyzed, the target image data can be pre-processed in sequence based on a preset feature point processing flow to obtain a tensor of uniform size, so as to improve the image processing efficiency of subsequent processes, reduce the difficulty of image processing, and reduce the probability of image processing errors; then the pre-processing result corresponding to the target image data is input into a set deep learning model, thereby outputting predicted heat map data corresponding to the pre-processing result, and the predicted heat map data includes feature points corresponding to various cardiac anatomical structures and their predicted probability distribution; finally, through the post-processing flow of the feature point processing flow, combined with the preset view classification information, the required multiple feature coordinate data are determined as the feature point coordinates corresponding to various cardiac anatomical structures to be analyzed; wherein, through the improved deep learning model, the optimized downsampling network is conducive to improving the feature extraction capability of the model; the increase of its coordinate information improves the feature point coordinate relationship extraction capability of the model; that is, through this scheme, the positioning accuracy and positioning effect of cardiac anatomical feature points are further improved. The following are detailed descriptions.

[0115] Embodiment 1

[0116] See also Figure 1 as well as Figure 6 , Figure 1 It is a flow chart of a method for locating cardiac anatomical feature points based on heat map regression disclosed in an embodiment of the present invention; Figure 6 FIG. 1 is a flow chart of another method for locating cardiac anatomical feature points based on heat map regression disclosed in an embodiment of the present invention. Figure 1 , Figure 6 The described cardiac anatomical feature point positioning method based on heat map regression can be applied to a cardiac anatomical feature point positioning device based on heat map regression, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the cardiac anatomical feature point positioning method based on heat map regression may include the following operations:

[0117] 101. Obtain target image data to be analyzed, where the target image data includes several frames of sub-image data corresponding to the end-diastole of the ventricle and the end-systole of the ventricle.

[0118] In an embodiment of the present invention, the target image data is acquired by anatomy professionals, such as cardiac ultrasound physicians, through relevant detection equipment (such as ultrasound detectors) to realize the acquisition and acquisition of ultrasound cardiograms. Specifically, when professionals initially perform ultrasound cardiogram detection through relevant detection equipment, the conventional recording time ranges from 2 seconds to 6 seconds, and at least includes a complete cardiac cycle, and the image size corresponding to the initially acquired ultrasound cardiogram ranges from 800×600 to 1600×1080. At the same time, the ultrasound cardiogram actually acquired includes at least six standard sections: apical four chambers, parasternal long axis left ventricular outflow tract, parasternal long axis right ventricular inflow tract, parasternal long axis right ventricular outflow tract, parasternal short axis central ventricle, and parasternal short axis apex; all the acquired ultrasound cardiograms can be directly used as target image data.

[0119] In the embodiment of the present invention, optionally, after all the echocardiograms are collected, the professional annotates them, and the specific annotation information added includes: right ventricular endocardium, right ventricular septum, right ventricular septum edge, tricuspid valve ring, base, apex, and pulmonary artery ring.

[0120] In an embodiment of the present invention, optionally, after completing the annotation of all echocardiograms, the data set can be divided according to a preset division ratio, such as dividing all echocardiograms into a training set, a validation set and a test set according to a ratio of 6:1:3; wherein the training set is used to train the model, the validation set is used to adjust the hyperparameters, and the test set is used to evaluate the final effect of the model, that is, the generalization ability.

[0121] 102. Perform a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data.

[0122] In the embodiment of the present invention, the pre-processing operation includes an image size conversion operation, and the pre-processing operation is used to adjust the target image data into a tensor of a preset size.

[0123] 103. Perform target processing operations on the pre-processing results according to the preset deep learning model to train the deep learning model and obtain predicted heat map data corresponding to the pre-processing results.

[0124] In the embodiment of the present invention, the target processing operation at least includes convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation;

[0125] Furthermore, the deep learning model is an improved model that optimizes the downsampling process and adds coordinate information to the pre-processing results.

[0126] 104. Perform a preset post-processing operation on the predicted heat map data according to the preset view classification information to obtain a plurality of feature coordinate data corresponding to the predicted heat map data.

[0127] In the embodiment of the present invention, the post-processing operation includes non-maximum suppression calculation, image scaling processing and coordinate selection operation; all feature coordinate data include multiple feature point coordinates, and all feature point coordinates are used to determine the cardiac anatomical structure.

[0128] It can be seen that implementation Figure 1 The described cardiac anatomical feature point positioning method based on heat map regression, after acquiring the target image data including the end of ventricular systole / diastole that needs to be analyzed, can perform pre-processing on the target image data in sequence based on the preset feature point processing process to obtain a tensor of uniform size, so as to improve the image processing efficiency of the subsequent process, reduce the difficulty of image processing and reduce the probability of image processing errors; then the pre-processing result corresponding to the target image data is input into the set deep learning model, so as to output the predicted heat map data corresponding to the pre-processing result, and the predicted heat map data includes the feature points corresponding to various cardiac anatomical structures and their predicted probability distribution; finally, through the post-processing process of the feature point processing process, combined with the preset view classification information, the required multiple feature coordinate data are determined as the feature point coordinates corresponding to the various cardiac anatomical structures to be analyzed; wherein, through the improved deep learning model, its optimized downsampling network is conducive to improving the feature extraction capability of the model; its coordinate information increases and improves the feature point coordinate relationship extraction capability of the model; that is, through this scheme, the positioning accuracy and positioning effect of the cardiac anatomical feature points are further improved.

[0129] In an optional embodiment, the above step 102 performs a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data, specifically including:

[0130] For each frame of sub-image data, determining a first image size of the frame of sub-image data;

[0131] Performing bilinear interpolation scaling processing on the frame sub-image data to scale the frame sub-image data from a first image size to a second image size, thereby obtaining a scaling processing result corresponding to the frame sub-image data;

[0132] Inputting the scaling processing result corresponding to the frame sub-image data into a preset exchange tensor channel to convert the frame sub-image data from the second image size to the third image size, thereby obtaining a tensor conversion result corresponding to the frame sub-image data;

[0133] Performing normalization processing and standardization processing on the tensor conversion result corresponding to the frame sub-image data in sequence to obtain a standardization processing result corresponding to the frame sub-image data;

[0134] The standardized processing results corresponding to all sub-image data are determined as pre-processing results corresponding to the target image data.

[0135] In this optional embodiment, the first image size corresponding to each frame of sub-image data input is recorded as H×W×C; after the bilinear interpolation scaling process, the first image size is scaled to a second image size corresponding to H′×W′×C; then the preset exchange tensor channel is input to adjust the second image size corresponding to the scaling result to a third image size corresponding to C×H′×W′; finally, normalization and standardization processes are performed in sequence to obtain the pre-processing result.

[0136] In this optional embodiment, the calculation formula used to perform the standardization process on each sub-image data is:

[0137]

[0138] Among them, I k,i,j Represents the element at position (k,i,j) in the tensor before transformation, I′ k,i,j Represents the element at position (k,i,j) in the transformed tensor, k∈{0,1,2}, μ k and σ k Represent the mean and variance of k-channel pixels, μ k and σ k The specific value of is obtained by counting the pixel brightness of the three RGB channels of the image in the dataset.

[0139] It can be seen that in this optional embodiment, an image pre-processing process is set up to achieve unified image format transformation and normalization and standardization processing before inputting the target image data into the deep learning model, which is beneficial to improving the model training efficiency when the deep learning model adopts the processed target image data and reducing the difficulty and error rate of model training.

[0140] In another optional embodiment, the above step 103 performs a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model, and the method of obtaining the predicted heat map data corresponding to the pre-processing result specifically includes:

[0141] Performing a preset data enhancement operation on the pre-processing result to obtain a data enhancement result corresponding to the pre-processing result, the data enhancement operation comprising an affine transformation and a pixel adjustment operation; the image parameter adjusted corresponding to the pixel adjustment operation comprises at least one of brightness, contrast and saturation;

[0142] Perform a preset coordinate adding operation on the data enhancement result to obtain a coordinate adding result corresponding to the data enhancement result, where the coordinate adding operation includes adding coordinate information based on CoordConv;

[0143] The coordinate adding result is input into a preset deep learning model, so as to perform preset convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation on the coordinate adding result through the deep learning model, and obtain a model processing result corresponding to the coordinate adding result, and the model processing result includes model output data corresponding to the coordinate adding result and a loss value corresponding to the loss calculation;

[0144] The target loss value and its corresponding target model output data are determined as predicted heat map data corresponding to the pre-processing result; wherein the target loss value is less than a preset loss threshold, and the target loss value is used to indicate that the deep learning model has completed model training.

[0145] In an embodiment of the present invention, when all echocardiograms are actually acquired, there are differences in the operating techniques of professionals for each section, and the data set included in the pre-processing result cannot cover all acquisition techniques. At the same time, the ultrasound images included in the pre-processing result have a lot of noise, and corresponding data enhancement means need to be set to improve the detection capability of the deep learning model.

[0146] In this optional embodiment, the above-mentioned affine transformation includes image rotation, translation and scaling operations; secondly, it is also considered to add random changes based on brightness, contrast and saturation on each pixel.

[0147] In this alternative embodiment, see Figure 7 , Figure 7 is a schematic diagram of the architecture corresponding to an improved deep learning network disclosed in an embodiment of the present invention, such as Figure 7 As shown in the figure, the improvements corresponding to the improved deep learning network are mainly: using ResNet to replace the downsampling part and adding CoordConv to add coordinate information to the input tensor.

[0148] in, Figure 7After the target image data is input, after the above-mentioned image enhancement processing, the coordinate information is added through CoordConv; then the deep learning network is processed in the vertical direction through the first layer, and the first layer includes at least one convolution layer, one batch normalization layer and one ReLu layer; the next layer is composed of a maximum pooling layer and three layers of residual blocks; the next layer is composed of a (2x downsampling) residual block and three layers of residual blocks; the subsequent analogy is composed of a layer of (2x downsampling) residual block and multiple layers of residual blocks; in the horizontal direction, each processing layer is composed of at least one convolution layer, a batch normalization layer and a ReLu layer, and they are superimposed, and a deconvolution layer is spliced ​​at the same time; at the final output, that is, before the output of the first layer in the vertical direction, the model output is performed through a convolution layer and a Sigmoid layer.

[0149] Among them, the first improvement to the U-Net model is to use ResNet50 to replace the downsampling part. This improvement solves the problem of insufficient feature extraction ability of the downsampling part of the U-Net model, which is conducive to improving the feature extraction ability and feature extraction effect of the model downsampling; the second improvement to the U-Net model is to add CoordConv after the input image, thereby adding coordinate information to the input tensor, alleviating the problem of not being able to fully utilize the positional relationship between feature points.

[0150] In this optional embodiment, the coordinate adding operation is used to add the corresponding coordinate information to the data enhancement result, and the calculation formula corresponding to the CoordConv in the horizontal axis direction is:

[0151]

[0152] Among them, Coordx is used to represent the horizontal axis coordinate information corresponding to the position information channel when the position information channel is constructed by CoordConv, and the size corresponding to the position information channel is recorded as H'×W'. i,j Used to represent the value of any point on the position information channel;

[0153] Similarly, the calculation formula corresponding to CoordConv can also include:

[0154]

[0155] Coordy is used to represent the vertical axis coordinate information corresponding to the position information channel when the position information channel is constructed by CoordConv.

[0156] And, when performing loss calculation through the deep learning network, the loss functions used include the pre-designed Weighted Smooth L1 Loss function and the Focal Smooth L1 Loss function;

[0157] The corresponding calculation formula of Weighted Smooth L1 Loss function is:

[0158]

[0159] in, is the predicted heat map corresponding to the predicted heat map data, is the true value heat map corresponding to the predicted heat map data, M = I (H d ≥0.2), I is the indicative function; H d is obtained by performing a 3×3 grayscale expansion operation on H; W is a size hyperparameter used to control the importance of foreground points, and W·M+1 is the corresponding weight term. represents the Hadamard product;

[0160] The calculation formula corresponding to the Focal Smooth L1 Loss function is:

[0161]

[0162] Among them, y, They are respectively the true value heat map corresponding to the predicted heat map data and the value of the predicted heat map at the same position; Thres is the threshold hyperparameter used to control the judgment of foreground points and background points; γ is the focusing hyperparameter.

[0163] In this optional embodiment, the two loss functions mentioned in this article are improved from Smooth L1 Loss. Let the true value of the previous layer of Sigmoid be y, then Smooth L1 Loss can be written as:

[0164]

[0165] In this optional embodiment, two loss functions are designed: Weighted Smooth L1 Loss and Focal Smooth L1 Loss. Both loss functions are designed based on Smooth L1 Loss and regress the value of the previous layer of Sigmoid. The difference is that Weighted Smooth L1 Loss uses the same method as AWLoss (Adaptive Wing Loss) to balance the foreground and background points, while Focal Smooth L1 Loss indirectly balances the foreground and background points by drawing on the idea of ​​Focal Loss.

[0166] It can be seen that in this optional embodiment, the setting of the image enhancement operation reduces the interference of image noise and the optimization of the downsampling is beneficial to improving the feature extraction ability and feature extraction effect of the model downsampling; the addition of CoordConv information improves the utilization rate of the positional relationship between image feature points.

[0167] Embodiment 2

[0168] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of another method for locating cardiac anatomical feature points based on heat map regression disclosed in an embodiment of the present invention. Figure 2 The described cardiac anatomical feature point positioning method based on heat map regression can be applied to a cardiac anatomical feature point positioning device based on heat map regression, and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the cardiac anatomical feature point positioning method based on heat map regression may include the following operations:

[0169] 201. Obtain target image data to be analyzed, where the target image data includes several frames of sub-image data corresponding to the end-diastole of the ventricle and the end-systole of the ventricle.

[0170] 202. Perform a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data.

[0171] 203. Perform a target processing operation on the pre-processing result according to a preset deep learning model to obtain predicted heat map data corresponding to the pre-processing result.

[0172] In the embodiment of the present invention, the deep learning model includes a downsampling network, which is used to perform downsampling processing. The downsampling network is specifically a ResNet network; preferably, the RestNet network may be RestNet50;

[0173] The predicted heat map data includes sub-predicted heat map data corresponding to each frame of sub-image data;

[0174] Each sub-image data includes at least one set of feature point coordinates corresponding to the sub-image data;

[0175] 204. After the target image data is acquired, for each sub-image data, the coordinate probability corresponding to the coordinates of all feature points in the sub-image data is calculated according to a preset probability conversion formula.

[0176] 205. Generate a pseudo probability map corresponding to the sub-image data according to all coordinate probabilities.

[0177] 206. Perform a bilinear interpolation scaling operation on the pseudo probability map corresponding to the sub-image data to obtain a true value heat map corresponding to the sub-image data.

[0178] In an embodiment of the present invention, all true value heat maps are used to compare with all predicted heat map data, that is, each true value heat map is used to compare with a corresponding sub-prediction heat map data to evaluate and / or optimize the performance of the deep learning model in executing data prediction.

[0179] 207. Perform a preset post-processing operation on the predicted heat map data according to the preset view classification information to obtain a plurality of feature coordinate data corresponding to the predicted heat map data.

[0180] In the embodiment of the present invention, for other descriptions of steps 201 to 203 and step 207, please refer to other specific descriptions of steps 101 to 103 and step 104 in the first embodiment, which will not be repeated in the embodiment of the present invention.

[0181] It can be seen that implementation Figure 2 The described cardiac anatomical feature point positioning method based on heat map regression, since the data to be processed in the subsequent post-processing operation is based on the premise that the predicted heat map data includes the predicted probability of the feature points, and the predicted heat map data actually initially output by the deep learning model contains the feature point coordinate data rather than the probability data of the feature points, a probability conversion and probability map generation operation for sub-image data is set up to convert the true value coordinates (sub-image data) into a true value heat map to compare all the true value heat maps with all the predicted heat map data, thereby evaluating the model performance of the currently trained deep learning model, or adjusting the model parameters, structure or training methods. That is, the setting of the probability conversion and probability map generation process improves the applicability of the overall deep learning model and the applicability of the overall solution.

[0182] In an optional embodiment, the preset view classification information includes a plurality of view categories; each sub-image data has at least one corresponding view category, and different view categories are used to indicate that the sub-image data has different slice acquisition postures;

[0183] The above-mentioned method of performing a preset post-processing operation on the predicted heat map data according to the preset view classification information to obtain a plurality of feature coordinate data corresponding to the predicted heat map data specifically includes:

[0184] According to the preset NMS calculation formula, non-maximum suppression calculation is performed on the predicted heat map data belonging to the same view category to obtain the NMS calculation result corresponding to the predicted heat map data;

[0185] Perform bilinear scaling on the NMS calculation result to obtain a target scaling result corresponding to the NMS calculation result; the image size corresponding to the target scaling result is consistent with the initial image size, and the initial image size is the size of the target image data before the pre-processing operation is performed;

[0186] According to a preset number, a plurality of target feature points and their feature coordinates corresponding to the preset number are determined from the target scaling result as a plurality of feature coordinate data corresponding to the predicted heat map data.

[0187] In this optional embodiment, the NMS (Non-Maximum Suppression) is used to refer to non-maximum suppression commonly used in deep learning networks.

[0188] In this optional embodiment, the preset number can be Among them, view is used to indicate the view category currently being processed.

[0189] In this optional embodiment, it can be clearly seen that the post-processing operation is also the heat map post-processing, and the process is divided into three steps: the first step is to perform two-dimensional non-maximum suppression on the predicted heat map data, in order to avoid the subsequent maximum value. The non-peak point in the region is obtained when the value is 0; the second step is to scale the target scaling result after NMS to the same size as the initial image through bilinear interpolation; the third step is to take the largest one from the target scaling result The coordinates of the values ​​are obtained to obtain the coordinates of the k-th feature point; wherein the k-th category is used to refer to the currently processed predicted heat map data, which all belong to the same view category.

[0190] In this optional embodiment, in the post-processing process for the predicted heat map data, the input is the predicted heat map (predicted heat map data) and the view category (view classification information), and the output is a set of feature point coordinates.

[0191] The NMS calculation formula can be specifically:

[0192]

[0193] Among them, the heat map after maximum pooling processing is The predicted heat map is The heat map after NMS processing is but The value of the previous position Calculated by the above formula.

[0194] In this optional embodiment, it can be seen that the essence of the NMS operation here is: if the value of the current position is the maximum value in the surrounding area, it is retained, otherwise it is 0. Considering the prediction heat map The size is large, and the size of the maximum pooling layer used in this paper is also large. More specifically, the maximum pooling layer size used in the heat map NMS process in this paper is 11x11, the step size is 1, and the padding is 5.

[0195] It can be seen that in this optional embodiment, in the post-processing process of the predicted heat map data, the calculation of the two-dimensional non-maximum suppression NMS is used to avoid selecting non-peak points in the area when a preset number of values ​​are subsequently taken; through the bilinear scaling process, the predicted heat map data after NMS is scaled to a first image size consistent with the initial image (target image data); and then the coordinates of the largest preset number of values ​​are taken from the target scaling result, thereby obtaining the coordinates of multiple types of feature points, thereby improving the selection accuracy of the multiple feature coordinates.

[0196] In another optional embodiment, after determining a plurality of feature coordinate data corresponding to the predicted heat map data, the method further includes:

[0197] According to the preset feature evaluation parameters, a positioning evaluation is performed on each feature coordinate data to obtain a positioning evaluation result corresponding to each feature coordinate data, and the positioning evaluation result corresponding to each feature coordinate data is used to determine the feature point positioning effect of the feature coordinate data; the feature evaluation parameters include the feature point detection rate and the average point error;

[0198] Among them, the feature point detection rate is used to indicate the probability of each feature coordinate data being detected, and the calculation formula corresponding to the feature point detection rate is:

[0199]

[0200] Among them, the i-th sample image I (i) Matched by method feature points, image I (i) The number of true feature points in is N (i) ;

[0201] The average point error is used to represent the average Euclidean distance between the successfully detected feature points and the true feature points. The calculation formula for the average point error is:

[0202]

[0203] Among them, the above They are respectively represented as the jth pair of true feature point coordinates and predicted feature point coordinates on the i-th sample matching.

[0204] In this optional embodiment, since the echocardiogram included in the target image data is collected by machines with different resolutions, it is necessary to first scale the image to 512 height while retaining the aspect ratio during the evaluation process. On this basis, the feature point detection rate (SDR) and the mean point error (MPE) are used in this optional embodiment to evaluate the feature point positioning effect. Among them, SDR refers to the probability of a feature point being detected. Specifically, the experiment stipulates that when the distance between the predicted feature point and the true feature point is less than a distance threshold, it is a successful detection, also known as the predicted feature point matching the true feature point. In the experiment, the distance threshold is 10 pixels, that is, 10px. When this threshold is met, we believe that the predicted feature point is close enough to the true feature point, and will not cause a predicted feature point to match multiple true feature points.

[0205] It can be seen that in this optional embodiment, two evaluation indicators, namely the feature point detection rate and the average point error, are set to evaluate the feature point positioning effect, which improves the overall feature point positioning scheme, realizes the quantification of the feature point positioning effect, and more intuitively reflects the feature point positioning effect, which is conducive to improving the applicability of the overall scheme.

[0206] Embodiment 3

[0207] See also Figure 3 , Figure 3 It is a schematic diagram of the structure of a cardiac anatomical feature point positioning device based on heat map regression disclosed in an embodiment of the present invention. The cardiac anatomical feature point positioning device based on heat map regression can be a cardiac anatomical feature point positioning terminal, device, system or server based on heat map regression. The server can be a local server, a remote server, or a cloud server (also called a cloud server). When the server is a non-cloud server, the non-cloud server can communicate with the cloud server, which is not limited in the embodiment of the present invention. Figure 3 As shown, the cardiac anatomical feature point positioning device based on heat map regression may include an acquisition module 301, a pre-processing module 302, a heat map processing module 303 and a post-processing module 304, wherein:

[0208] An acquisition module 301 is used to acquire target image data to be analyzed, wherein the target image data includes sub-image data corresponding to a plurality of frames of ventricular end-diastole and ventricular end-systole;

[0209] A pre-processing module 302 is used to perform a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data, wherein the pre-processing operation includes an image size conversion operation, and the pre-processing operation is used to adjust the target image data to a tensor of a preset size;

[0210] The heat map processing module 303 is used to perform a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model and obtain predicted heat map data corresponding to the pre-processing result, wherein the target processing operation at least includes convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation;

[0211] A post-processing module 304 is used to perform a preset post-processing operation on the predicted heat map data according to preset view classification information to obtain a plurality of feature coordinate data corresponding to the predicted heat map data;

[0212] Among them, the post-processing operations include non-maximum suppression calculation, image scaling processing and coordinate selection operations; all feature coordinate data include multiple feature point coordinates, and all feature point coordinates are used to determine the anatomical structure of the heart.

[0213] It can be seen that implementation Figure 3 The described cardiac anatomical feature point positioning device based on heat map regression, after acquiring the target image data including the end of ventricular contraction / diastole that needs to be analyzed, can perform pre-processing on the target image data in sequence based on the preset feature point processing process to obtain a tensor of uniform size, so as to improve the image processing efficiency of the subsequent process, reduce the difficulty of image processing and reduce the probability of image processing errors; then the pre-processing result corresponding to the target image data is input into the set deep learning model, so as to output the predicted heat map data corresponding to the pre-processing result, and the predicted heat map data includes the feature points corresponding to various cardiac anatomical structures and their predicted probability distribution; finally, through the post-processing process of the feature point processing process, combined with the preset view classification information, the required multiple feature coordinate data are determined as the feature point coordinates corresponding to the various cardiac anatomical structures to be analyzed; wherein, through the improved deep learning model, its optimized downsampling network is conducive to improving the feature extraction capability of the model; the increase of its coordinate information improves the feature point coordinate relationship extraction capability of the model; that is, through this scheme, the positioning accuracy and positioning effect of the cardiac anatomical feature points are further improved.

[0214] In an optional embodiment, the pre-processing module 302 performs a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data, specifically including:

[0215] For each frame of sub-image data, determining a first image size of the frame of sub-image data;

[0216] Performing bilinear interpolation scaling processing on the frame sub-image data to scale the frame sub-image data from a first image size to a second image size, thereby obtaining a scaling processing result corresponding to the frame sub-image data;

[0217] Inputting the scaling processing result corresponding to the frame sub-image data into a preset exchange tensor channel to convert the frame sub-image data from the second image size to the third image size, thereby obtaining a tensor conversion result corresponding to the frame sub-image data;

[0218] Performing normalization processing and standardization processing on the tensor conversion result corresponding to the frame sub-image data in sequence to obtain a standardization processing result corresponding to the frame sub-image data;

[0219] The standardized processing results corresponding to all sub-image data are determined as pre-processing results corresponding to the target image data.

[0220] It can be seen that in this optional embodiment, an image pre-processing process is set up to achieve unified image format transformation and normalization and standardization processing before inputting the target image data into the deep learning model, which is beneficial to improving the model training efficiency when the deep learning model adopts the processed target image data and reducing the difficulty and error rate of model training.

[0221] In another optional embodiment, the heat map processing module 303 performs a target processing operation on the pre-processing result according to a preset deep learning model, and obtains the predicted heat map data corresponding to the pre-processing result in a manner specifically including:

[0222] Performing a preset data enhancement operation on the pre-processing result to obtain a data enhancement result corresponding to the pre-processing result, the data enhancement operation comprising an affine transformation and a pixel adjustment operation; the image parameter adjusted corresponding to the pixel adjustment operation comprises at least one of brightness, contrast and saturation;

[0223] Perform a preset coordinate adding operation on the data enhancement result to obtain a coordinate adding result corresponding to the data enhancement result, where the coordinate adding operation includes adding coordinate information based on CoordConv;

[0224] Input the coordinate adding result into a preset deep learning model, so as to perform preset convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation on the coordinate adding result through the deep learning model, and obtain a model processing result corresponding to the coordinate adding result, wherein the model processing result includes model output data corresponding to the coordinate adding result and a loss value corresponding to the loss calculation;

[0225] The target loss value and its corresponding target model output data are determined as predicted heat map data corresponding to the pre-processing result; wherein the target loss value is less than a preset loss threshold, and the target loss value is used to indicate that the deep learning model has completed model training.

[0226] In this optional embodiment, the coordinate adding operation is used to add the corresponding coordinate information to the data enhancement result, and the calculation formula corresponding to the CoordConv in the horizontal axis direction is:

[0227]

[0228] Among them, Coordx is used to represent the horizontal axis coordinate information corresponding to the position information channel constructed by CoordConv, and the size corresponding to the position information channel is recorded as H'×W'. i,j Used to represent the value of any point on the position information channel;

[0229] Similarly, the calculation formula corresponding to CoordConv can also include:

[0230]

[0231] Coordy is used to represent the vertical axis coordinate information corresponding to the position information channel when the position information channel is constructed by CoordConv.

[0232] And, when performing loss calculation through the deep learning network, the loss functions used include the pre-designed Weighted Smooth L1 Loss function and the Focal Smooth L1 Loss function;

[0233] The corresponding calculation formula of Weighted Smooth L1 Loss function is:

[0234]

[0235] in, is the predicted heat map corresponding to the predicted heat map data, H is the true value heat map corresponding to the predicted heat map data, M = I (H d ≥0.2), I is the indicative function; H d is obtained by performing a 3×3 grayscale expansion operation on H; W is a size hyperparameter used to control the importance of foreground points, and W·M+1 is the corresponding weight term. represents the Hadamard product;

[0236] The calculation formula corresponding to the Focal Smooth L1 Loss function is:

[0237]

[0238] Among them, y, They are respectively the true value heat map corresponding to the predicted heat map data and the value of the predicted heat map at the same position; Thres is the threshold hyperparameter used to control the judgment of foreground points and background points; γ is the focusing hyperparameter.

[0239] In this optional embodiment, the two loss functions mentioned in this article are improved from Smooth L1 Loss. Let the true value of the previous layer of Sigmoid be y, then Smooth L1 Loss can be written as:

[0240]

[0241] It can be seen that in this optional embodiment, the setting of the image enhancement operation reduces the interference of image noise and the optimization of the downsampling is beneficial to improving the feature extraction ability and feature extraction effect of the model downsampling; the addition of CoordConv improves the utilization rate of the positional relationship between image feature points.

[0242] In another optional embodiment, the deep learning model includes a downsampling network, the downsampling network is used to perform downsampling processing, and the downsampling network is specifically a ResNet network;

[0243] The predicted heat map data includes sub-predicted heat map data corresponding to each frame of sub-image data;

[0244] Each sub-image data includes at least one set of feature point coordinates corresponding to the sub-image data;

[0245] like Figure 4 As shown, the device also includes a calculation module 305, a generation module 306, and a scaling processing module 307, wherein:

[0246] The calculation module 305 is used to calculate the coordinate probability corresponding to the coordinates of all feature points in each sub-image data according to a preset probability conversion formula after acquiring the target image data;

[0247] A generating module 306, configured to generate a pseudo probability map corresponding to the sub-image data according to all coordinate probabilities;

[0248] A scaling processing module 307 is used to perform a bilinear interpolation scaling operation on the pseudo probability map corresponding to the sub-image data to obtain a true value heat map corresponding to the sub-image data;

[0249] In an embodiment of the present invention, all true value heat maps are used to compare with all predicted heat map data to evaluate and / or optimize the performance of the deep learning model in executing data prediction.

[0250] It can be seen that in this optional embodiment, since the data to be processed by the subsequent post-processing operation is based on the premise that the predicted heat map data includes the predicted probability of the feature points, and the predicted heat map data actually initially output by the deep learning model contains the feature point coordinate data, rather than the probability data of the feature points, a probability conversion and probability map generation operation for the sub-image data is set to convert the true value coordinates (sub-image data) into a true value heat map to compare all the true value heat maps with all the predicted heat map data, so as to evaluate the model performance of the currently trained deep learning model, or adjust the model parameters, structure or training methods. That is, the setting of the probability conversion and probability map generation process improves the applicability of the overall deep learning model and the applicability of the overall solution.

[0251] In another optional embodiment, the preset view classification information includes a plurality of view categories; each sub-image data has at least one corresponding view category, and different view categories are used to indicate that the sub-image data has different slice acquisition postures;

[0252] The post-processing module 304 performs a preset post-processing operation on the predicted heat map data according to the preset view classification information to obtain a plurality of feature coordinate data corresponding to the predicted heat map data. Specifically, the method includes:

[0253] According to the preset NMS calculation formula, non-maximum suppression calculation is performed on the predicted heat map data belonging to the same view category to obtain the NMS calculation result corresponding to the predicted heat map data;

[0254] Perform bilinear scaling on the NMS calculation result to obtain a target scaling result corresponding to the NMS calculation result; the image size corresponding to the target scaling result is consistent with the initial image size, and the initial image size is the size of the target image data before the pre-processing operation is performed;

[0255] According to a preset number, a plurality of target feature points and their feature coordinates corresponding to the preset number are determined from the target scaling result as a plurality of feature coordinate data corresponding to the predicted heat map data.

[0256] It can be seen that in this optional embodiment, in the post-processing process of the predicted heat map data, the calculation of the two-dimensional non-maximum suppression NMS is used to avoid selecting non-peak points in the area when a preset number of values ​​are subsequently taken; through the bilinear scaling process, the predicted heat map data after NMS is scaled to a first image size consistent with the initial image (target image data); and then the coordinates of the largest preset number of values ​​are taken from the target scaling result, thereby obtaining the coordinates of multiple types of feature points, thereby improving the selection accuracy of the multiple feature coordinates.

[0257] In yet another optional embodiment, Figure 4As shown, the device further includes an evaluation module 309, wherein:

[0258] Evaluation module 309 is used to perform positioning evaluation on each feature coordinate data according to preset feature evaluation parameters after determining multiple feature coordinate data corresponding to the predicted heat map data, and obtain a positioning evaluation result corresponding to each feature coordinate data, wherein the positioning evaluation result corresponding to each feature coordinate data is used to determine the feature point positioning effect of the feature coordinate data; the feature evaluation parameters include feature point detection rate and average point error;

[0259] Among them, the feature point detection rate is used to indicate the probability of each feature coordinate data being detected, and the calculation formula corresponding to the feature point detection rate is:

[0260]

[0261] Among them, the i-th sample image I (i) After the device is matched to feature points, image I (i) The number of true feature points in is N (i) ;

[0262] The average point error is used to represent the average Euclidean distance between the successfully detected feature points and the true feature points. The calculation formula for the average point error is:

[0263]

[0264] Among them, the above They are respectively represented as the jth pair of true feature point coordinates and predicted feature point coordinates on the i-th sample matching.

[0265] It can be seen that in this optional embodiment, two evaluation indicators, namely the feature point detection rate and the average point error, are set to evaluate the feature point positioning effect, which improves the overall feature point positioning scheme, realizes the quantification of the feature point positioning effect, and more intuitively reflects the feature point positioning effect, which is conducive to improving the applicability of the overall scheme.

[0266] Embodiment 4

[0267] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of another structure of a cardiac anatomical feature point positioning device based on heat map regression disclosed in an embodiment of the present invention. Figure 5 As shown, the cardiac anatomical feature point positioning device based on heat map regression may include:

[0268] A memory 401 storing executable program codes;

[0269] a processor 402 coupled to the memory 401;

[0270] The processor 402 calls the executable program code stored in the memory 401 to execute the steps of the cardiac anatomical feature point positioning method based on heat map regression described in the first embodiment of the present invention or the second embodiment of the present invention.

[0271] Embodiment 5

[0272] An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps in the cardiac anatomical feature point positioning method based on heat map regression described in Embodiment 1 or Embodiment 2 of the present invention.

[0273] Embodiment 6

[0274] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps of the cardiac anatomical feature point positioning method based on heat map regression described in Example 1 or Example 2.

[0275] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0276] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0277] Finally, it should be noted that the method and device for locating cardiac anatomical feature points based on heat map regression disclosed in the embodiment of the present invention only discloses the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for locating cardiac anatomical feature points based on heat map regression, characterized in that: The method comprises: Acquire target image data to be analyzed, wherein the target image data includes sub-image data corresponding to a plurality of frames of ventricular end-diastole and ventricular end-systole; Performing a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data, wherein the pre-processing operation includes an image size conversion operation, and the pre-processing operation is used to adjust the target image data to a tensor of a preset size; Performing a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model and obtain predicted heat map data corresponding to the pre-processing result, wherein the target processing operation at least includes convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation; the deep learning model is an improved model that optimizes the downsampling processing and adds coordinate information to the pre-processing result; According to the preset view classification information, a preset post-processing operation is performed on the predicted heat map data to obtain a plurality of feature coordinate data corresponding to the predicted heat map data; Wherein, the post-processing operation includes non-maximum suppression calculation, image scaling processing and coordinate selection operation; all the feature coordinate data include multiple feature point coordinates, and all the feature point coordinates are used to determine the cardiac anatomical structure; The performing a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model and obtain predicted heat map data corresponding to the pre-processing result includes: Performing a preset data enhancement operation on the pre-processing result to obtain a data enhancement result corresponding to the pre-processing result, wherein the data enhancement operation includes an affine transformation and a pixel adjustment operation; the image parameter adjusted by the pixel adjustment operation includes at least one of brightness, contrast and saturation; Performing a preset coordinate adding operation on the data enhancement result to obtain a coordinate adding result corresponding to the data enhancement result, wherein the coordinate adding operation includes adding coordinate information based on CoordConv; Input the coordinate adding result into a preset deep learning model, so as to perform preset convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation on the coordinate adding result through the deep learning model, and obtain a model processing result corresponding to the coordinate adding result, wherein the model processing result includes model output data corresponding to the coordinate adding result and a loss value corresponding to the loss calculation; Determine the target loss value and its corresponding target model output data as the predicted heat map data corresponding to the pre-processing result; wherein the target loss value is less than a preset loss threshold, and the target loss value is used to indicate that the deep learning model has completed model training; Furthermore, the deep learning model includes a downsampling network, which is used to perform the downsampling processing, and the downsampling network is specifically a ResNet network.

2. The method for locating cardiac anatomical feature points based on heat map regression according to claim 1, characterized in that: The performing a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data includes: For each frame of the sub-image data, determining a first image size of the sub-image data of the frame; Performing bilinear interpolation scaling processing on the frame sub-image data to scale the frame sub-image data from the first image size to a second image size, thereby obtaining a scaling processing result corresponding to the frame sub-image data; Inputting the scaling processing result corresponding to the frame sub-image data into a preset exchange tensor channel to convert the frame sub-image data from the second image size to a third image size, thereby obtaining a tensor conversion result corresponding to the frame sub-image data; Performing normalization processing and standardization processing on the tensor conversion result corresponding to the frame sub-image data in sequence to obtain a standardization processing result corresponding to the frame sub-image data; The standardized processing results corresponding to all the sub-image data are determined as pre-processing results corresponding to the target image data.

3. The method for locating cardiac anatomical feature points based on heat map regression according to claim 1, characterized in that: Each frame of the sub-image data includes at least one set of feature point coordinates corresponding to the sub-image data; After acquiring the target image data, the method further includes: For each piece of the sub-image data, the coordinate probabilities corresponding to the coordinates of all the feature points in the sub-image data are calculated according to a preset probability conversion formula, and a pseudo-probability map corresponding to the sub-image data is generated according to all the coordinate probabilities; Perform a bilinear interpolation scaling operation on the pseudo probability map corresponding to the sub-image data to obtain a true value heat map corresponding to the sub-image data; All of the true value heat maps are used to compare with all of the predicted heat map data to evaluate and / or optimize the performance of the deep learning model in executing data prediction.

4. The method for locating cardiac anatomical feature points based on heat map regression according to claim 1, 2 or 3, characterized in that: The preset view classification information includes a plurality of view categories; each of the sub-image data has at least one of the view categories corresponding thereto, and different view categories are used to indicate that the sub-image data has different slice acquisition postures; The step of performing a preset post-processing operation on the predicted heat map data according to the preset view classification information to obtain a plurality of feature coordinate data corresponding to the predicted heat map data includes: According to a preset NMS calculation formula, non-maximum suppression calculation is performed on the predicted heat map data belonging to the same view category to obtain an NMS calculation result corresponding to the predicted heat map data; Performing bilinear scaling processing on the NMS calculation result to obtain a target scaling result corresponding to the NMS calculation result; the image size corresponding to the target scaling result is consistent with the initial image size, and the initial image size is the size of the target image data before performing the pre-processing operation; According to a preset number, a plurality of target feature points and their feature coordinates corresponding to the preset number are determined from the target scaling result as a plurality of feature coordinate data corresponding to the predicted heat map data.

5. The method for locating cardiac anatomical feature points based on heat map regression according to claim 4, characterized in that: After determining a plurality of feature coordinate data corresponding to the predicted heat map data, the method further includes: According to the preset feature evaluation parameters, positioning evaluation is performed on each of the feature coordinate data to obtain a positioning evaluation result corresponding to each of the feature coordinate data, and the positioning evaluation result corresponding to each of the feature coordinate data is used to determine the feature point positioning effect of the feature coordinate data; the feature evaluation parameters include feature point detection rate and average point error; The feature point detection rate is used to represent the probability of each feature coordinate data being detected, and the calculation formula corresponding to the feature point detection rate is: Among them, the i-th sample image I (i) Through the method described above, feature points, image I (i) The number of true feature points in is N (i) ; The average point error is used to represent the average Euclidean distance between the successfully detected feature points and the true feature points. The calculation formula corresponding to the average point error is: Among them, the above They are respectively represented as the jth pair of true feature point coordinates and predicted feature point coordinates on the i-th sample matching.

6. The method for locating cardiac anatomical feature points based on heat map regression according to claim 1, characterized in that: The coordinate adding operation is used to add the corresponding coordinate information to the data enhancement result. The calculation formula corresponding to the CoordConv in the horizontal axis direction is: Among them, Coordx is used to represent the horizontal axis coordinate information corresponding to the position information channel constructed by the CoordConv, and the size corresponding to the position information channel is recorded as H'×W'. i,j Used to represent the value of any point on the position information channel; And, when the loss calculation is performed by the deep learning model, the loss function used includes a pre-designed Weighted Smooth L1 Loss function and a Focal Smooth L1 Loss function; The corresponding calculation formula of the Weighted Smooth L1 Loss function is: in, is the predicted heat map corresponding to the predicted heat map data, H is the true value heat map corresponding to the predicted heat map data, M=I(H d ≥0.2), I is the indicative function; H d is obtained by performing a 3×3 grayscale expansion operation on H; W is a size hyperparameter used to control the importance of foreground points, and W·M+1 is the corresponding weight term. represents the Hadamard product; The calculation formula corresponding to the Focal Smooth L1 Loss function is: in, are respectively the true value heat map corresponding to the predicted heat map data and the value of the predicted heat map at the same position; Thres is the threshold hyperparameter used to control the judgment of foreground points and background points; γ is the focusing hyperparameter.

7. A cardiac anatomical feature point positioning device based on heat map regression, characterized in that: The device comprises: An acquisition module, used for acquiring target image data to be analyzed, wherein the target image data includes sub-image data corresponding to a plurality of frames of ventricular end-diastole and ventricular end-systole; A pre-processing module, used to perform a preset pre-processing operation on the target image data to obtain a pre-processing result corresponding to the target image data, wherein the pre-processing operation includes an image size conversion operation, and the pre-processing operation is used to adjust the target image data to a tensor of a preset size; A heat map processing module, used to perform a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model and obtain predicted heat map data corresponding to the pre-processing result, wherein the target processing operation at least includes convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation; the deep learning model is an improved model that optimizes the downsampling processing and adds coordinate information to the pre-processing result; A post-processing module, configured to perform a preset post-processing operation on the predicted heat map data according to preset view classification information, to obtain a plurality of feature coordinate data corresponding to the predicted heat map data; Wherein, the post-processing operation includes non-maximum suppression calculation, image scaling processing and coordinate selection operation; all the feature coordinate data include multiple feature point coordinates, and all the feature point coordinates are used to determine the cardiac anatomical structure; The heat map processing module performs a target processing operation on the pre-processing result according to a preset deep learning model to train the deep learning model, and the method of obtaining predicted heat map data corresponding to the pre-processing result specifically includes: Performing a preset data enhancement operation on the pre-processing result to obtain a data enhancement result corresponding to the pre-processing result, wherein the data enhancement operation includes an affine transformation and a pixel adjustment operation; the image parameter adjusted by the pixel adjustment operation includes at least one of brightness, contrast and saturation; Performing a preset coordinate adding operation on the data enhancement result to obtain a coordinate adding result corresponding to the data enhancement result, wherein the coordinate adding operation includes adding coordinate information based on CoordConv; Input the coordinate adding result into a preset deep learning model, so as to perform preset convolution processing, normalization processing, pooling processing, residual calculation, downsampling processing, activation processing and loss calculation on the coordinate adding result through the deep learning model, and obtain a model processing result corresponding to the coordinate adding result, wherein the model processing result includes model output data corresponding to the coordinate adding result and a loss value corresponding to the loss calculation; Determine the target loss value and its corresponding target model output data as the predicted heat map data corresponding to the pre-processing result; wherein the target loss value is less than a preset loss threshold, and the target loss value is used to indicate that the deep learning model has completed model training; The deep learning model includes a downsampling network, which is used to perform the downsampling process, and the downsampling network is specifically a ResNet network.

8. A cardiac anatomical feature point positioning device based on heat map regression, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the cardiac anatomical feature point positioning method based on heat map regression as described in any one of claims 1-6.

9. A computer storage medium, characterized in that The computer storage medium stores computer instructions, which, when called, are used to execute the cardiac anatomical feature point positioning method based on heat map regression as described in any one of claims 1-6.

Images