Heart ultrasound image key point constraint calibration method and system, readable storage medium and program product

Abstract

The present application relates to a kind of cardiac ultrasound image key point constraint calibration method, device / equipment / system, readable storage medium and program product, the method can not only accurately predict key point position, reduce the tracking error caused by occlusion and noise interference, also ensure the continuity and smoothness of key point tracking by automatic calibration mechanism.At the same time, considering the physiological characteristics of cardiac movement, so that the corrected key point position is more in line with the normal systolic and diastolic process of heart, improve the physiological rationality of diagnostic result, reduce the dependence on artificial operation, enhance the objectivity and consistency of diagnosis, provide strong support for the diagnosis and treatment of cardiovascular disease.

Description

Technical Field

[0001] This invention relates to the field of medical ultrasound report analysis technology, and in particular to a method, system, readable storage medium, and program product for constrained calibration of key points in cardiac ultrasound images. Background Technology

[0002] In existing technologies, echocardiography, as a non-invasive diagnostic technique, plays an irreplaceable role in assessing the structure and function of the heart. It serves a central function in the diagnosis and treatment of cardiovascular diseases. Compared to traditional ultrasound examinations, echocardiography not only focuses on organ morphology and lesions but also emphasizes the heart's performance. This performance assessment involves a detailed analysis of the morphological changes, motion trajectory, and functional status of the ventricular walls during different cardiac cycles. Through these analyses, a series of objective numerical indicators, such as left ventricular ejection fraction, can be generated, providing important evidence for clinical decision-making.

[0003] Currently, despite significant advancements in echocardiography for cardiac motion analysis, key point tracking remains a challenge. Traditional tracking algorithms, such as speckle-based methods, while capable of locating key points in certain situations, suffer a substantial drop in accuracy and reliability when encountering image occlusion, noise interference, or insufficient image quality. Furthermore, these methods often rely on physician experience and judgment, which is not only inefficient but also susceptible to subjective influences.

[0004] In recent years, with the rise of deep learning technology, keypoint detection models based on neural networks have been introduced into echocardiography analysis. This technology, by learning from large amounts of labeled data, achieves automatic prediction of keypoint locations, significantly improving the automation and accuracy of tracking. However, when processing consecutive frame images, these models still struggle to avoid discontinuities and jumps in keypoint location predictions, especially when cardiac motion is disturbed, where prediction errors can increase significantly. These problems limit the accuracy and reliability of echocardiography in clinical applications, necessitating a new technological solution to address these challenges. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a method, system, readable storage medium, and program product for constraining calibration of key points in cardiac ultrasound images.

[0006] To achieve the above objectives, this invention provides a method for constrained calibration of key points in cardiac ultrasound images, comprising: using a pre-trained neural network key point model to predict and locate key points in an input cardiac ultrasound sequence image, obtaining multiple initial predicted coordinates in each temporal image; calculating the changing trend of the position of each key point in multiple temporal images, and constructing a key point motion model based on these trends; comparing the initial predicted coordinates of the key points with the key point coordinates predicted by the motion model, and activating an error constraint correction module within the motion model when the deviation between the two coordinates exceeds a preset threshold; wherein, the constraint correction module uses a constraint function to calculate the correction amount of the key point coordinates, and generates the corrected key point coordinates as the final output based on the correction amount.

[0007] A further technical solution is that the training process of the neural network keypoint model includes the following steps:

[0008] I. In echocardiographic images, pre-determine and delineate the anatomical regions of the ventricular wall, and clearly define key points within these regions and their interrelationships;

[0009] II. Accurately label the collected echocardiogram sequences, marking the exact locations of key points to construct a training dataset;

[0010] III. Using a key point detection neural network structure, learn and identify the distribution patterns and motion laws of key points in echocardiographic sequence images;

[0011] IV. Complete model training to achieve automatic detection and trajectory prediction of key points in newly input cardiac ultrasound sequence images.

[0012] A further technical solution is that the construction process of the key point motion model includes the following steps:

[0013] I. Based on the key point model of neural network, analyze the changing trend of key point positions in consecutive frame echocardiogram images to identify and predict the periodic characteristics of cardiac motion;

[0014] II. Using the heart rate and frame rate information from echocardiography, a motion model of the key points is established to simulate the expected positional changes of the key points during cardiac contraction and relaxation.

[0015] A further technical solution is that the activation error constraint correction module includes the following specific steps:

[0016] Ⅰ. Calculate the weighted average acceleration vector a and weighted average velocity vector v of each key point in the three consecutive time-phase images t-2, t-1, and t;

[0017] II. Using the weighted average velocity vector v and the weighted average acceleration vector a, calculate the predicted displacement s of the key point at time t+1 according to the formula s=v*dt+(a*dt^2) / 2, where dt is the time interval between adjacent time phase images;

[0018] III. Calculate the predicted coordinates of the key point at time t+1, Point(t+1), by combining the current key point coordinates Point(t) at time t and the predicted displacement s, where Point(t+1) = Point(t) + s;

[0019] IV. Calculate the coordinate difference ds between the predicted coordinates Point(t+1) of the key point at time t+1 and the predicted coordinates Point(model) of the key point at time t+1, where ds = Point(model) - Point(t+1);

[0020] V. Calculate the displacement correction value r through the constraint function, which is r=w0*log(ds*w1+1)+b, where w0, w1, and b are parameters, and w0=1, w1=2, and b=6;

[0021] VI. Calculate the coordinates of the final constrained keypoint Point(modify) based on the predicted coordinates Point(t+1) and the correction value r as the final output, where Point(modify) = Point(t+1) + r.

[0022] A further technical solution is that the setting of the preset threshold includes, but is not limited to, the following factors:

[0023] I. Physiological parameters of cardiac activity: ventricular rate below 200 beats / minute;

[0024] II. Imaging characteristics of echocardiography: The frame rate of echocardiography is 20-100 frames / second;

[0025] III. In two consecutive time-phase images, the movement distance of the key point position is less than 1 cm.

[0026] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above-described methods.

[0027] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of any of the above-described methods.

[0028] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer program product, including a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of any of the above-described methods.

[0029] The method, system, readable storage medium, and program product for constrained calibration of key points in cardiac ultrasound images designed in this invention significantly improve the accuracy and reliability of cardiac ultrasound image analysis. This method not only accurately predicts key point locations and reduces tracking errors caused by occlusion and noise interference, but also ensures the continuity and smoothness of key point tracking through an automated calibration mechanism. Simultaneously, considering the physiological characteristics of cardiac motion, the corrected key point locations better conform to the normal contraction and relaxation process of the heart, improving the physiological rationality of diagnostic results, reducing reliance on manual operation, and enhancing the objectivity and consistency of diagnosis, thus providing strong support for the diagnosis and treatment of cardiovascular diseases. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the specific implementation of the key point constraint calibration method in Example 1;

[0031] Figure 2 This is a schematic diagram of the key point coordinate error constraint correction in Example 1. Detailed Implementation

[0032] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0033] Example 1.

[0034] like Figure 1 As shown, Figure 1 A specific implementation of a key point constraint calibration method for cardiac ultrasound images is shown.

[0035] It should be noted that if substantially the same result is obtained, the method of this invention is not based on... Figure 1 The process sequence shown is limited, and the method includes the following steps: S1: Using a pre-trained neural network keypoint model, keypoint prediction and localization are performed on the input cardiac ultrasound sequence image to obtain multiple initial predicted coordinates in each phase image.

[0036] Specifically, when using a pre-trained neural network keypoint model, the following steps are performed:

[0037] I. In echocardiographic images, pre-determine and delineate the anatomical regions of the ventricular wall, and clearly define key points within these regions and their interconnections.

[0038] In one specific embodiment, experienced doctors or technicians can first determine and delineate the anatomical region of the ventricular wall. Within the delineated anatomical region, key points such as the apex of the heart and specific locations of the ventricular wall are clearly defined, and the connection relationships between these key points are recorded, so as to obtain the relative positional relationship between the key points and to predict the movement path of the key points during the contraction and relaxation of the heart.

[0039] II. Accurately label the collected echocardiogram sequence images, marking the exact locations of key points to construct a training dataset.

[0040] In one specific embodiment, the collected echocardiographic image sequences can encompass echocardiographic images from patients of different ages, sexes, and cardiac conditions, ensuring the diversity and representativeness of the dataset. For example, it can include images of different pathological states such as normal hearts, myocardial infarction, and ventricular hypertrophy. These images are then precisely annotated using professional software or manual annotation tools to ensure that the positions of key points correspond to the actual cardiac motion, such as the position of the apex during systole relative to its position during diastole. After annotation, a training dataset containing key point position information is constructed for subsequent neural network training.

[0041] III. Using a key point detection neural network structure, learn and identify the distribution patterns and motion laws of key points in echocardiographic sequence images.

[0042] In one specific embodiment, a keypoint recognition neural network (e.g., a YOLOv8 neural network based on a CNN structure) is first trained to recognize images frame by frame and predict keypoint positions. The predicted keypoints of all frames are stored as a sequence array. In practice, since the accuracy of keypoint prediction by ordinary convolutional neural networks has not yet reached pixel-level tracking, there are prediction deviations for keypoints. Therefore, after obtaining the predicted keypoints of all frames, the obtained keypoints from all frames can be appended to the frame image. The operator confirms whether the keypoint prediction of the aortic closure frame (AVC frame) is correct by playing the image. If it is correct, this keypoint is used. If there is a deviation, the keypoint position is adjusted by manually dragging to determine the keypoint position. Then, a pixel-level optical flow algorithm (e.g., sparse optical flow calcOpticalFlowPyrLK in OpenCV) is used to track the adjusted keypoints frame by frame to obtain more accurate keypoint positions. In this optical flow tracking process, the position correction between two frames uses a weighted average of the estimated position and the actual optical flow tracking position proposed in this application to obtain a more accurate position that conforms to the motion characteristics. IV. Complete model training to achieve automatic detection and trajectory prediction of key points in newly input cardiac ultrasound sequence images.

[0043] In one specific embodiment, after model training is complete, the model is tested using an independent validation dataset to evaluate its accuracy and reliability in practical applications. For example, the model's prediction error can be calculated by comparing it with keypoint locations annotated by experts. Specifically, the trained model is deployed to an echocardiography analysis system, typically configured on a medical image processing server. The system receives new echocardiography sequence images as input, and the model automatically detects and predicts keypoint locations, generating keypoint trajectories. S2: Calculate the changing trends of each keypoint location in multiple temporal images, and construct a keypoint motion model based on these trends.

[0044] Specifically, the construction process of the key point motion model includes the following steps:

[0045] I. Based on the key point model of neural network, analyze the changing trend of key point positions in consecutive frame echocardiogram images to identify and predict the periodic characteristics of cardiac motion;

[0046] In one specific embodiment, the predicted locations of key points in consecutive frames of echocardiogram images are obtained from a pre-trained neural network key point model. For example, for key points of the left ventricular wall, the model predicts the location in a series of cardiac cycles, that is, the process of heart contraction and relaxation is simulated as a periodic waveform.

[0047] II. Using the heart rate and frame rate information from echocardiography, a motion model of the key points is established to simulate the expected positional changes of the key points during cardiac contraction and relaxation.

[0048] In one example, a YOLOv8 key prediction model trained on the aforementioned key points is used to obtain key points for each frame of the image. Once the model's accuracy reaches the expected level (generally 90%), the key point sequence is analyzed one by one. For example, the left ventricular wall key point sequence obtained from the analysis of left ventricular function in a four-chamber left ventricle is analyzed. For example, nine key points are used in the left ventricle, distributed at equal intervals on the left ventricular wall. For example, the positions of all frames of the aforementioned two types of key points are analyzed. By calculating the frame-by-frame changes in the left ventricular area and analyzing the changes in the position of this key point, the position, velocity, and acceleration vector information of the changes can be calculated. That is, by superimposing information from multiple samples, an approximate curve function is fitted, making the tracking results more consistent with physiological principles.

[0049] S3: Compare the initial predicted coordinates of the key points with the key point coordinates predicted by the motion model. When the difference between the two coordinates exceeds the preset threshold, activate the error constraint correction module in the motion model.

[0050] Specifically, the setting of the preset threshold includes, but is not limited to, the following factors:

[0051] I. Physiological parameters of cardiac activity: ventricular rate below 200 beats / minute;

[0052] II. Imaging characteristics of echocardiography: The frame rate of echocardiography is 20-100 frames / second;

[0053] III. In two consecutive time-phase images, the movement distance of the key point position is less than 1 cm.

[0054] Thus, the frame rate of echocardiography is between 20 and 100 frames per second, which determines the frequency at which the image captures cardiac motion. At this frame rate, the expected movement distance of key points between two consecutive frames should match the actual movement of the heart. For example, if the frame rate is 50 frames per second, then the movement distance of key points between two consecutive frames should be less than 1 cm to ensure image continuity and accuracy. When the coordinate deviation exceeds a preset threshold, the error constraint correction module is activated to adjust the predicted coordinates of the key points to reduce the deviation and make the coordinates closer to the actual trajectory of the heart's movement. S4: The constraint correction module uses constraint functions to calculate the correction amount of the key point coordinates and generates the corrected key point coordinates as the final output based on the correction amount.

[0055] Specifically, such as Figure 2 As shown, the specific steps of starting the error constraint correction module include:

[0056] Ⅰ. Calculate the weighted average acceleration vector a and weighted average velocity vector v of each key point in the three consecutive time-phase images t-2, t-1, and t;

[0057] In one specific embodiment, the details are as follows: The displacement of each keypoint in different temporal images is calculated by comparing the positions of the keypoints in these images. Then, based on the displacement of each keypoint, a weighted average acceleration vector *a* and a weighted average velocity vector *v* are calculated. For example, an exponentially weighted averaging method can be used, giving higher weight to the most recent temporal image to reflect the instantaneous changes in heart motion.

[0058] II. Using the weighted average velocity vector v and the weighted average acceleration vector a, calculate the predicted displacement s of the key point at time t+1 according to the formula s=v*dt+(a*dt^2) / 2, where dt is the time interval between adjacent time phase images.

[0059] III. Calculate the predicted coordinates of the key point at time t+1, Point(t+1), by combining the current key point coordinates Point(t) at time t and the predicted displacement s, where Point(t+1) = Point(t) + s.

[0060] For example, if the coordinates of Point(t) are (x, y) and the predicted displacement s is (Δx, Δy), then the coordinates of Point(t+1) are (x+Δx, y+Δy).

[0061] IV. Calculate the coordinate difference ds between the predicted coordinates Point(t+1) of the key point at time t+1 and the predicted coordinates Point(model) of the key point at time t+1, where ds = Point(model) - Point(t+1);

[0062] V. Calculate the displacement correction value r through the constraint function, which is r=w0*log(ds*w1+1)+b, where w0, w1, and b are parameters, and w0=1, w1=2, and b=6;

[0063] The purpose of this constraint function is to ensure that the value of r of the variable ds remains within a certain range regardless of its magnitude, whether it is large or small. This significantly suppresses large displacements, ensuring that the predicted position of the key point does not deviate too far from the actual trajectory of the heart. For small displacements, due to the characteristics of the log function, the change in the correction value r is small, thus maintaining the accuracy of the model output and avoiding overcorrection. This achieves smooth correction of the key point position, resulting in a smoother predicted position of the key point in consecutive frames. It effectively corrects prediction errors caused by factors such as occlusion, breathing movements, or equipment noise, making key point tracking more stable and reliable.

[0064] In this embodiment, w0, w1, and b are parameters used to adjust the shape of the function. w0=1, w1=2, and b=6 are used to approximate the motion characteristics of heart tissue, effectively constraining errors caused by large-amplitude movements while reducing the constraint on smaller-amplitude offsets. This results in a smooth, non-linear adjustment process, improving the accuracy of predictions and resolving the error of large displacement changes in predicted points caused by interference in existing keypoint algorithms. This provides stronger support for diagnosis and treatment.

[0065] VI. Calculate the coordinates of the final constrained keypoint Point(modify) based on the predicted coordinates Point(t+1) and the correction value r as the final output, where Point(modify) = Point(t+1) + r.

[0066] In summary, this method not only accurately predicts keypoint locations and reduces tracking errors caused by occlusion and noise interference, but also ensures the continuity and smoothness of keypoint tracking through an automated calibration mechanism. Furthermore, by considering the physiological characteristics of cardiac motion, the corrected keypoint locations better align with the normal contraction and relaxation process of the heart, improving the physiological rationality of the diagnostic results, reducing reliance on manual operation, and enhancing the objectivity and consistency of diagnosis, thus providing strong support for the diagnosis and treatment of cardiovascular diseases.

[0067] To address the aforementioned technical problems, this application also provides a computer system, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of any of the methods described above. The computer system can be a desktop computer, laptop, handheld computer, cloud server, or other computing device, which can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0068] To address the aforementioned technical problems, embodiments of this application also provide a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of any of the methods described above. The readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc.

[0069] To address the aforementioned technical problems, this application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of any of the methods described above. That is, the technical solution of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0070] In the description of this invention, it should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or accompanying drawings of this application are used to distinguish different objects, not to describe a particular order.

[0071] In the description of this invention, it should also be noted that the reference to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0072] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cardiac ultrasound image key point constraint calibration method, characterized in that, The application relates to a method for improving the accuracy of key point detection in echocardiogram sequence images. The training process of the neural network key point model comprises the following steps:

2. The cardiac ultrasound image keypoint constrained calibration method of claim 1, wherein, I. In echocardiogram images, the anatomical regions of the wall are determined and demarcated in advance, and the key points and the connection relationship between the key points are defined; II. The collected echocardiogram sequence images are accurately labeled to mark the exact positions of the key points and construct a training data set; III. The key point detection neural network structure is used to learn and identify the distribution mode and motion law of the key points in the echocardiogram sequence images; and IV. The model training is completed to realize automatic detection and trajectory prediction of the key points in the newly input echocardiogram sequence images.

3. The cardiac ultrasound image keypoint constrained calibration method of claim 2, wherein, The construction process of the key point motion model comprises the following steps: I. On the basis of the neural network key point model, the change trend of the key point positions in the continuous frame echocardiogram images is analyzed to identify and predict the periodic characteristics of the heart motion; II. The heart beating speed and the frame frequency information of the echocardiogram are used to establish the key point motion model to simulate the expected position change of the key points in the heart contraction and diastole process.

4. The cardiac ultrasound image keypoint constrained calibration method of claim 1, wherein, The specific steps of starting the error constraint correction module comprise: I. The weighted average acceleration vector a and the weighted average speed vector v of the key points in the continuous three time phase images of t-2, t-1 and t are calculated; II. The weighted average speed vector v and the weighted average acceleration vector a are used to calculate the predicted displacement s of the key point at the t+1 moment according to the formula s=v*dt+(a*dt^2) / 2, wherein dt is the time interval of the adjacent time phase images; III. The predicted coordinates Point(t+1) of the key point at the t+1 moment are calculated by combining the key point coordinates Point(t) at the current t moment and the predicted displacement s, wherein Point(t+1)=Point(t)+s; IV. The coordinate difference ds between the predicted coordinates Point(t+1) of the key point at the t+1 moment and the motion model predicted coordinates Point(model) of the key point at the t+1 moment is calculated, wherein ds=Point(model)-Point(t+1); V. The displacement correction value r is calculated through a constraint function, and the function is r=w0*log(ds*w1+1)+b, wherein w0, w1 and b are parameters, and w0=1, w1=2 and b=6. VI. Calculate the coordinate of the key point Point(modify) after final constraint correction as the final output according to the predicted coordinate Point(t+1) and the correction value r, wherein Point(modify)=Point(t+1)+r.

5. The cardiac ultrasound image keypoint constrained calibration method of claim 4, wherein, The setting of the preset threshold value includes but is not limited to the following factors: I. Physiological parameters of heart movement, ventricular rate below 200 beats per minute; II. Imaging characteristics of echocardiogram, frame frequency of echocardiogram is 20-100 frames per second; III. In the continuous two frame phase images, the moving distance of the key point position is less than 1 cm.

6. A computer system comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 5. The processor executes the computer program to realize the steps of the method of any one of claims 1-5.

7. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the method of any one of claims 1-5.

8. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the method of any one of claims 1-5.

Images