Auxiliary diagnostic systems, methods, procedures, products, and equipment for heart valve repair

Abstract

This invention discloses an auxiliary diagnostic system, method, program product, and device for transcatheter aortic valve repair (TAVR). By analyzing surgical images and physician questions, it automatically answers relevant diagnostic, treatment, and interpretation questions, and provides real-time surgical suggestions and optimization plans, thereby improving the safety and effectiveness of TAVR surgery. The auxiliary diagnostic system is based on a pre-trained visual question-answering model and includes an input module, a visual module, a language module, and an output module. The input module receives medical images and natural language questions as system input. The visual Transformer model processes the medical images and extracts visual features; the language Transformer model processes the questions and generates answers based on the visual features extracted by the visual module; the output module outputs the generated answers in text or speech form.

Description

Technical Field

[0001] This invention relates to the field of medical auxiliary technology, and more specifically, to an auxiliary diagnostic system, method, procedure product and equipment for heart valve contralateral repair surgery. Background Technology

[0002] Heart valve disease is a common cardiovascular condition, primarily including aortic stenosis and aortic regurgitation. Traditional treatments include open-heart surgery and percutaneous aortic valve replacement, but these methods have drawbacks such as significant trauma, long recovery times, and high risk of complications. In contrast, transcatheter aortic valve replacement (TAVR), as a non-invasive treatment, has a lower risk of complications and a faster recovery time, making it an important option for treating heart valve disease.

[0003] However, TAVR surgery still requires highly precise and specialized procedures, including valve localization, valve deployment, valve positioning, and fixation. These steps require physicians to make judgments and decisions based on the patient's cardiac structure and valvular disease, thus demanding extensive experience and expertise. Furthermore, due to individual patient differences and varying surgical complexity, physicians must adjust and optimize the procedure based on real-time image information to ensure its safety and effectiveness.

[0004] To address challenges during surgery, researchers have explored the use of computer vision and natural language processing (NLP) technologies to assist in procedures. In the field of laparoscopic surgery, Chen Zhen of the Hong Kong Innovation Institute of the Chinese Academy of Sciences and Bai Long of the Chinese University of Hong Kong, among others, have achieved accurate generation of answers and suggestions by analyzing surgical images and doctors' natural language questions, thereby aiding doctors in surgical decision-making and operations. For example, during surgery, doctors may encounter complex situations such as improper instrument placement or slow surgical progress. In such cases, a visual question-and-answer system can automatically answer relevant questions and provide real-time suggestions and optimization solutions by analyzing surgical images and doctors' questions, helping doctors quickly resolve problems and improve surgical efficiency and safety. However, due to the unique nature of minimally invasive catheter-based interventional procedures, cardiac valve interventional surgery currently lacks a practically applicable visual question-and-answer system suitable for TAVR (Transcatheter Aortic Valve Replacement) procedures. Summary of the Invention

[0005] In view of the problems existing in the prior art, the present invention proposes an auxiliary diagnostic system, method, program product and equipment for heart valve edge repair surgery. By analyzing surgical images and doctors' questions, it automatically answers relevant diagnostic, treatment and interpretation questions, and provides real-time surgical suggestions and optimization plans, which can improve the safety and effectiveness of TAVR surgery.

[0006] To achieve the above objectives, in a first aspect, the present invention provides an auxiliary diagnostic system for heart valve resection surgery, characterized in that it is based on a trained basic visual question-and-answer model, comprising:

[0007] Input module: Used to receive medical images and natural language questions as input;

[0008] Visual module: The visual Transformer model is used to process medical images and extract visual features from the images; the encoder of the visual Transformer model automatically learns important features in the image through a self-attention mechanism and performs weighted fusion of different features; the final hidden state of the decoder of the visual Transformer model is used as the visual feature.

[0009] Language Module: The language Transformer model is used to process the question and generate the answer based on the visual features extracted by the visual module. The encoder of the language Transformer model automatically learns the keywords in the question through a self-attention mechanism and performs weighted fusion of different keywords to handle ambiguity and polysemy in natural language. The decoder of the language Transformer model fuses the visual features to generate the answer.

[0010] Output module: Outputs the generated answers in text or voice format.

[0011] In some embodiments of the first aspect of this application, the language Transformer model integrates a large language model.

[0012] Secondly, the present invention provides an auxiliary diagnostic method based on the auxiliary diagnostic system for heart valve edge repair surgery as described above, characterized by comprising the following steps:

[0013] The auxiliary diagnostic system inputs medical images and the doctor's natural language questions;

[0014] The visual module of the auxiliary diagnostic system processes the medical images to generate visual features;

[0015] The language module of the auxiliary diagnostic system processes natural language questions and generates answers based on the visual features;

[0016] Output the answer generated by the language module in text or voice format.

[0017] In some embodiments of the second aspect of this application, the vision module first segments the input image into fixed-size image blocks, then performs linear projection on each image block to generate an image embedding, then embeds positional encoding information into the image embedding, processes the image embedding using an encoder based on a multi-layer self-attention mechanism and a feedforward neural network, and finally outputs the final hidden state of the visual transformer decoder as a visual feature.

[0018] In some embodiments of the second aspect of this application, the language module first segments the input question into words or sub-words, then converts each word into a word vector, adds positional encoding information to the word vector, processes the word vectors using an encoder based on a multi-layer self-attention mechanism and a feedforward neural network, and finally generates an answer based on the decoder output of the language transformer and visual features.

[0019] In some embodiments of the second aspect of this application, the training of the basic visual question-answering model is carried out in an end-to-end manner, and the training steps include:

[0020] Data preprocessing: standardizing and enhancing medical images, segmenting and encoding the questions;

[0021] Loss function design: Use the cross-entropy loss function to calculate the difference between the predicted answer and the true answer;

[0022] Optimization algorithm: The Adam optimizer is used for parameter updates;

[0023] Learning rate scheduling: A learning rate scheduling strategy using warmup and cosine decay;

[0024] Regularization: Use weight decay and dropout to prevent overfitting.

[0025] In some embodiments of the second aspect of this application, medical knowledge graphs and / or expert experience are encoded into a basic visual question-answering model.

[0026] In some embodiments of the second aspect of this application, the auxiliary diagnostic system visualizes the attention distribution of the basic visual question-answering model using a heatmap.

[0027] Thirdly, the present invention provides a computer program product, characterized in that, when the computer program product is run on a computer, it causes the computer to perform the auxiliary diagnostic method as described above.

[0028] Fourthly, the present invention provides an electronic device, characterized in that it includes a memory storing executable program code; a processor coupled to the memory; and the processor calling the executable program code stored in the memory to execute the auxiliary diagnostic method described above.

[0029] Compared with the prior art, the present invention has the following technical effects:

[0030] This invention can automatically analyze medical images and answer doctors' diagnostic questions through a transformer-type deep learning model, thereby improving diagnostic accuracy and efficiency and providing strong support for TAVR surgery decisions. The system can provide customized diagnostic suggestions based on the specific conditions of different patients, achieving personalized diagnosis. In addition, the system can be updated online, continuously learning new medical knowledge and cases to improve diagnostic capabilities and maintain the system's advanced nature and reliability. The system can encode medical knowledge graphs and / or expert experience into the model, enabling the widespread dissemination and application of knowledge and improving medical standards. Attached Figure Description

[0031] Figure 1 This is an architecture diagram of an auxiliary diagnostic system according to an embodiment of the present invention.

[0032] Figure 2 This is a flowchart of an auxiliary diagnostic method in one embodiment of the present invention.

[0033] Figure 3 This is a diagram illustrating the implementation steps of the vision module in one embodiment of the present invention.

[0034] Figure 4 This is a diagram illustrating the implementation steps of the language module in one embodiment of the present invention. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but these are not intended to limit the scope of the invention.

[0036] In the following detailed description, numerous specific details are set forth to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that well-known algorithms are not shown in detail to avoid obscuring the spirit of the invention.

[0037] Furthermore, the execution order of actions, steps, etc. in the apparatus and methods shown in the claims, specification, and drawings can be implemented in any order, as long as there is no specific express limitation on the order and the output of the preceding processing is not used in the subsequent processing.

[0038] In the embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0039] Example 1

[0040] This embodiment provides an auxiliary diagnostic system for heart valve margin repair surgery. (See also...) Figure 1 It is based on a pre-trained basic visual question answering model, including:

[0041] Input module 100: used to receive medical images and natural language questions as input; medical images may include various image types such as cardiac CT, MRI, and X-ray, and natural language questions may be doctors' descriptions of patients' conditions or analysis questions of images.

[0042] Visual module 200: The visual Transformer model is used to process medical images and extract visual features of the images; the encoder of the visual Transformer model automatically learns important features in the image through a self-attention mechanism and performs weighted fusion of different features; the final hidden state of the decoder of the visual Transformer model is used as the visual feature.

[0043] Language Module 300: This module uses a language Transformer model to process questions and generates answers based on the visual features extracted by the vision module. The encoder of the language Transformer model automatically learns keywords in the question through a self-attention mechanism and performs weighted fusion of different keywords to handle ambiguity and polysemy in natural language. The decoder of the language Transformer model fuses the visual features to generate answers. At the same time, this module can also handle ambiguity and polysemy in natural language, improving the system's comprehension ability.

[0044] Output module 400: Outputs the generated answers in text or voice format. This output module can flexibly select the output method according to the doctor's needs. Simultaneously, the system can also associate the answers with relevant medical knowledge and case studies to provide more detailed diagnostic information.

[0045] To support more natural conversational interaction and improve the ease of use of the system, the language Transformer model can preferably integrate known large language models.

[0046] See Figure 2 The method for performing auxiliary diagnosis using an auxiliary diagnostic system in heart valve repair surgery as described above includes the following steps:

[0047] The auxiliary diagnostic system inputs medical images and the doctor's natural language questions;

[0048] The visual module of the auxiliary diagnostic system processes the medical images to generate visual features;

[0049] The language module of the auxiliary diagnostic system processes natural language questions and generates answers based on the visual features;

[0050] Output the answer generated by the language module in text or voice format.

[0051] As can be seen from the above-described auxiliary diagnostic system architecture and its implementation method, the system can automatically analyze medical images and answer diagnostic questions raised by doctors based on the analysis results, thereby assisting in TAVR surgery decisions.

[0052] As a preferred technical solution, medical knowledge graphs and / or expert experience can be encoded into the basic visual question-answering model to further enhance the system's reasoning ability.

[0053] As a preferred technical solution, the auxiliary diagnostic system visualizes the attention distribution of the basic visual question-answering model using a heatmap, further improving the interpretability of the system.

[0054] As an example, this embodiment uses a specialized TAVR surgery-related dataset for model training and testing. The dataset includes:

[0055] (1) Medical Imaging:

[0056] Images of valvular heart disease (such as aortic stenosis, aortic regurgitation, etc.)

[0057] Images of the valve implantation process (e.g., valve deployment, positioning, etc.)

[0058] Images of cardiac structures (such as the aorta, left ventricle, etc.)

[0059] (2) Problem type:

[0060] Diagnostic questions (e.g., "Is there aortic stenosis in this image?")

[0061] Treatment questions (e.g., "What is the best TAVR procedure for this type of aortic stenosis?")

[0062] Explain the question (e.g., "What is the location and angle of the valve implantation in this image?").

[0063] The vision module adopts a vision Transformer structure, such as... Figure 3 As shown, its main steps include:

[0064] (1) Image slicing: Divide the input image into image blocks of a fixed size.

[0065] (2) Linear projection: Perform linear projection on each image block to generate an image embedding.

[0066] (3) Position encoding: Add position encoding information to the image embedding.

[0067] (4) Employ Transformer encoder: Use multi-layer self-attention mechanism and feedforward neural network to process image embedding.

[0068] (5) Visual feature generation: The final hidden state of the output Transformer decoder is used as the visual feature.

[0069] The language module adopts a language Transformer structure, such as... Figure 4 As shown, its main steps include:

[0070] (1) Word segmentation: The input question is divided into words or sub-words.

[0071] (2) Word embedding: Convert each word into a word vector.

[0072] (3) Position encoding: Add position encoding information to the word vector.

[0073] (4) Transformer encoder: uses a multi-layer self-attention mechanism and a feedforward neural network to process word vectors.

[0074] (5) Generate answers by combining the visual features of the image: Generate answers based on the output of the Transformer decoder and visual features.

[0075] Training of basic visual question answering models

[0076] Using an end-to-end approach, the main steps are as follows:

[0077] (1) Data preprocessing: standardize and enhance medical images, and segment and encode questions.

[0078] (2) Loss function: The cross-entropy loss function is used to calculate the difference between the predicted answer and the true answer.

[0079] (3) Optimization algorithm: The Adam optimizer is used for parameter updates.

[0080] (4) Learning rate scheduling: a learning rate scheduling strategy using warmup and cosine decay.

[0081] (5) Regularization: Use weight decay and dropout to prevent overfitting.

[0082] Performance evaluation

[0083] Use the following metrics to evaluate model performance:

[0084] (1) Accuracy: The proportion of questions answered correctly by the model.

[0085] (2) Recall: The proportion of correct answers in the model's responses.

[0086] (3) F1 score: the harmonic mean of precision and recall.

[0087] (4) Average response time: The average time it takes for the model to answer questions.

[0088] Example 1: Diagnosis of aortic stenosis

[0089] enter:

[0090] Medical Imaging: Patient's cardiac CT images

[0091] Question: "Is there aortic stenosis in this image?"

[0092] Processing procedure:

[0093] The vision module processes CT images to generate visual features.

[0094] The language module processes the question and generates an answer based on visual features.

[0095] Output:

[0096] "Yes, the images show aortic stenosis. The aortic valve orifice area is significantly reduced, and the valve leaflets show calcification and thickening, all typical characteristics of aortic stenosis. Based on image analysis, the stenosis is severe, and further evaluation is recommended to determine suitability for TAVR surgery."

[0097] Example 2: TAVR surgical plan development

[0098] enter:

[0099] Medical imaging: Echocardiogram and CT angiography images of the patient.

[0100] Question: "Based on these images, what is the recommended TAVR surgical procedure?"

[0101] Processing procedure:

[0102] The vision module processes ultrasound and CT images separately to generate comprehensive visual features.

[0103] The language module processes the question and generates an answer based on comprehensive visual features.

[0104] Output:

[0105] Based on image analysis, the transfemoral TAVR procedure is recommended. The reasons are as follows:

[0106] The patient has severe aortic stenosis and is a suitable candidate for TAVR surgery.

[0107] The patient's femoral artery diameter is greater than 6 mm, and there is no obvious tortuosity or calcification, making the femoral artery approach suitable.

[0108] The aortic arch has no obvious angulation, which facilitates the smooth passage of the catheter.

[0109] We recommend using a 26mm self-expanding valve to ensure good fit and seal.

[0110] Example 3: Intraoperative guidance for TAVR

[0111] enter:

[0112] Medical Imaging: Real-time X-ray Fluoroscopy Images

[0113] Question: "Is the current position and angle of the valve appropriate?"

[0114] Processing procedure:

[0115] The vision module processes real-time X-ray images to generate visual features.

[0116] The language module processes the question and generates an answer based on visual features.

[0117] Output:

[0118] The current position and angle of the valve need to be fine-tuned:

[0119] Location: The valve is slightly lower than the aortic valve annulus by about 2mm; it is recommended to adjust it upwards.

[0120] Angle: The valve forms an angle of approximately 5 degrees with the long axis of the aorta, which is slightly tilted. It is recommended to rotate it slightly to make it more vertical.

[0121] Deployment extent: The valve has deployed approximately 80%, and will continue to slowly deploy until it is fully deployed.

[0122] Please make the above adjustments carefully under fluorescence guidance to ensure optimal valve implantation results.

[0123] Understandably, the aforementioned auxiliary diagnostic system can also be combined with multiple imaging modalities such as CT, ultrasound, and X-ray to perform multimodal fusion, thereby improving the comprehensiveness and accuracy of diagnosis.

[0124] Example 2

[0125] This embodiment provides a computer program product that, when run on a computer, causes the computer to perform the auxiliary diagnostic method as described in Embodiment 1.

[0126] The essence of this technical solution, or the part that contributes to the prior art, or a portion of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic system (which may be a personal computer, server, or network system, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] Example 3

[0128] This embodiment provides an electronic device, including a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the auxiliary diagnostic method as described in Embodiment 1.

[0129] The aforementioned basic visual question answering model and its training method can be embodied in the form of a computer program product or a software functional unit. If the aforementioned basic visual question answering model and its training method are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0130] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] In summary, this invention provides an auxiliary diagnostic system, method, program product, and device for TAVR (Transcatheter Aortic Valve Replacement) surgery. By analyzing surgical images and physician questions, it automatically answers relevant diagnostic, treatment, and interpretation questions, and provides real-time surgical suggestions and optimization plans, thereby improving the safety and effectiveness of TAVR surgery. The auxiliary diagnostic system is based on a pre-trained visual question-answering model and includes an input module, a visual module, a language module, and an output module. The input module receives medical images and natural language questions as system input. The visual Transformer model processes the medical images and extracts visual features; the language Transformer model processes the questions and generates answers based on the visual features extracted by the visual module; the output module outputs the generated answers in text or speech form.

[0132] Those skilled in the art should understand that variations can be implemented by combining existing technology with the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention, and will not be elaborated here either.

[0133] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above. Systems and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention are still within the scope of protection of the present invention.

Claims

1. An auxiliary diagnostic system for heart valve margin repair surgery, characterized in that, It is based on a pre-trained basic visual question answering model, including: Input module: Used to receive medical images and natural language questions as input; Visual module: The visual Transformer model is used to process medical images and extract visual features from the images; the encoder of the visual Transformer model automatically learns important features in the image through a self-attention mechanism and performs weighted fusion of different features; the final hidden state of the decoder of the visual Transformer model is used as the visual feature. Language Module: The language Transformer model is used to process the question and generate the answer based on the visual features extracted by the visual module. The encoder of the language Transformer model automatically learns the keywords in the question through a self-attention mechanism and performs weighted fusion of different keywords to handle ambiguity and polysemy in natural language. The decoder of the language Transformer model fuses the visual features to generate the answer. Output module: Outputs the generated answers in text or voice format.

2. The auxiliary diagnostic system for heart valve margin repair according to claim 1, characterized in that, The language Transformer model integrates a large language model.

3. An auxiliary diagnostic method based on the auxiliary diagnostic system for heart valve repair as described in claim 1 or 2, characterized in that, Includes the following steps: The auxiliary diagnostic system inputs medical images and the doctor's natural language questions; The visual module of the auxiliary diagnostic system processes the medical images to generate visual features; The language module of the auxiliary diagnostic system processes natural language questions and generates answers based on the visual features; Output the answer generated by the language module in text or voice format.

4. The auxiliary diagnostic method according to claim 3, characterized in that, The vision module first segments the input image into fixed-size image blocks, then performs linear projection on each image block to generate an image embedding, and then embeds positional encoding information into the image embedding. The image embedding is processed by an encoder based on a multi-layer self-attention mechanism and a feedforward neural network. Finally, the final hidden state of the visual transformer decoder is output as a visual feature.

5. The auxiliary diagnostic method according to claim 3, characterized in that, The language module first segments the input question into words or sub-words, then converts each word into a word vector, adds positional encoding information to the word vectors, processes the word vectors using an encoder based on a multi-layer self-attention mechanism and a feedforward neural network, and finally generates the answer based on the decoder output of the language transformer and visual features.

6. The auxiliary diagnostic method according to claim 3, characterized in that, The training of the basic visual question answering model adopts an end-to-end approach, and the training steps include: Data preprocessing: standardizing and enhancing medical images, segmenting and encoding the questions; Loss function design: Use the cross-entropy loss function to calculate the difference between the predicted answer and the true answer; Optimization algorithm: The Adam optimizer is used for parameter updates; Learning rate scheduling: A learning rate scheduling strategy using warmup and cosine decay; Regularization: Use weight decay and dropout to prevent overfitting.

7. The auxiliary diagnostic method according to claim 3, characterized in that, Encode medical knowledge graphs and / or expert experience into the basic visual question answering model.

8. The auxiliary diagnostic method according to claim 3, characterized in that, The auxiliary diagnostic system visualizes the attention distribution of the basic visual question-answering model using a heatmap.

9. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the auxiliary diagnostic method as described in any one of claims 3 to 8.

10. An electronic device, characterized in that, A memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the auxiliary diagnostic method according to any one of claims 3 to 8.

Images