Auxiliary diagnostic system, method, program product and device for heart valve para-marginal repair

Abstract

The invention discloses an auxiliary diagnosis system, an auxiliary diagnosis method, an auxiliary diagnosis program product and auxiliary diagnosis equipment for heart valve alignment repair, which can improve the safety and the effectiveness of a TAVR operation by analyzing operation images and problems of doctors, automatically answering related diagnosis, treatment and explaining problems and providing real-time operation suggestions and optimization schemes. The auxiliary diagnosis system is based on a trained basic visual question and answer model and comprises an input module, a visual module, a language module and an output module. The input module receives medical images and natural language questions as system inputs. The visual Transform model is used for processing the medical image and extracting visual features of the image; processing the question by a language Transform model, and generating an answer based on the visual features extracted by the visual module; and the output module outputs the generated answer in a text or voice form.

Description

Technical Field

[0001] The present invention relates to the field of medical assistance technology, and more specifically, to an auxiliary diagnosis system, method, program product and equipment for heart valve rim repair. Background Art

[0002] Heart valve disease is a common cardiovascular disease, mainly including aortic valve stenosis, aortic valve regurgitation and other types of lesions. Traditional treatments include open-chest surgery and percutaneous aortic valve replacement, but these methods have disadvantages such as large trauma, long recovery time, and high risk of complications. In contrast, transcatheter aortic valve replacement (TAVR), as a non-invasive treatment method, has a lower risk of complications and a faster recovery time, and has become an important choice for the treatment of heart valve disease.

[0003] However, TAVR surgery still requires highly precise and professional operations, including valve positioning, valve deployment, valve positioning and fixation. These steps require doctors to make judgments and decisions based on the patient's heart structure and valve lesions, so a lot of experience and expertise are needed. In addition, due to individual differences in patients and the difficulty of surgery, doctors also need to adjust and optimize according to real-time image information during the operation to ensure the safety and effectiveness of the operation.

[0004] In order to solve problems during surgery, researchers have tried to use computer vision and natural language processing technologies to assist surgery. In the field of laparoscopic surgery, Chen Zhen from the Hong Kong Innovation Institute of the Chinese Academy of Sciences and Bai Long and others from the Chinese University of Hong Kong have accurately generated answers and suggestions by analyzing surgical images and doctors' natural language questions, thereby helping doctors make surgical decisions and operations. For example, during an operation, doctors may encounter some complex situations, such as improper positioning of surgical instruments and slow progress of the operation. At this time, the visual question-answering system can automatically answer relevant questions by analyzing surgical images and doctors' questions, and provide real-time suggestions and optimization solutions to help doctors quickly solve problems and improve surgical efficiency and safety. However, due to the uniqueness of minimally invasive catheter intervention, heart valve intervention surgery currently does not have a practical visual question-answering system that is compatible with TAVR. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention proposes an auxiliary diagnosis system, method, program product and equipment for heart valve apposition repair, which automatically answers relevant diagnosis, treatment and interpretation questions by analyzing surgical images and doctors' questions, and provides real-time surgical suggestions and optimization plans, thereby improving the safety and effectiveness of TAVR surgery.

[0006] To achieve the above objectives, in a first aspect, the present invention provides an auxiliary diagnosis system for heart valve prosthesis, characterized in that it is based on a trained basic visual question answering model and includes:

[0007] Input module: used to receive medical images and natural language questions as input;

[0008] Vision module: The visual Transformer model is used to process medical images and extract visual features of the images. The encoder of the visual Transformer model automatically learns important features in the image through a self-attention mechanism and performs weighted fusion of different features. The final hidden state of the decoder of the visual Transformer model is used as the visual feature.

[0009] Language module: A language Transformer model is used to process questions and generate answers based on the visual features extracted by the visual module; the encoder of the language Transformer model automatically learns the keywords in the question through a self-attention mechanism and performs weighted fusion of different keywords to handle ambiguity and polysemy in natural language; the decoder of the language Transformer model fuses the visual features to generate answers;

[0010] Output module: Outputs the generated answer in the form of text or voice.

[0011] In some embodiments of the first aspect of the present application, the language Transformer model integrates a large language model.

[0012] In a second aspect, the present invention provides an auxiliary diagnosis method based on the auxiliary diagnosis system for heart valve rim repair as described above, characterized in that it comprises the following steps:

[0013] The auxiliary diagnosis system inputs medical images and natural language questions of doctors;

[0014] The visual module of the auxiliary diagnosis system processes the medical image to generate visual features;

[0015] The language module of the auxiliary diagnosis system processes natural language questions and generates answers based on the visual features;

[0016] The answer generated by the language module is output in the form of text or speech.

[0017] In some embodiments of the second aspect of the present application, the visual module first divides the input image into image blocks of a fixed size, then performs linear projection on each image block to generate an image embedding, then embeds and adds position encoding information to the image embedding, uses an encoder of a visual transformer based on a multi-layer self-attention mechanism and a feedforward neural network to process the image embedding, and finally outputs the final hidden state of the decoder of the visual transformer as a visual feature.

[0018] In some embodiments of the second aspect of the present application, the language module first segments the input question into words or sub-words, then converts each word into a word vector, adds position encoding information to the word vector, processes the word vector using an encoder of a language transformer based on a multi-layer self-attention mechanism and a feedforward neural network, and finally generates an answer based on the decoder output of the language transformer and visual features.

[0019] In some embodiments of the second aspect of the present application, the basic visual question answering model is trained in an end-to-end manner, and the training steps include:

[0020] Data preprocessing: standardize and enhance medical images, segment and encode questions;

[0021] Loss function design: Use the cross entropy loss function to calculate the difference between the predicted answer and the true answer;

[0022] Optimization algorithm: Adam optimizer is used to update parameters;

[0023] Learning rate scheduling: Using warmup and cosine decay learning rate scheduling strategy;

[0024] Regularization: Use weight decay and dropout to prevent overfitting.

[0025] In some embodiments of the second aspect of the present application, medical knowledge graphs and / or expert experience are encoded into a basic visual question answering model.

[0026] In some embodiments of the second aspect of the present application, the auxiliary diagnosis system visualizes the attention distribution of the basic visual question-answering model by means of a heat map.

[0027] In a third aspect, the present invention provides a computer program product, characterized in that when the computer program product is run on a computer, the computer is enabled to execute the auxiliary diagnosis method as described above.

[0028] In a fourth aspect, the present invention provides an electronic device, characterized by a memory storing executable program code; a processor coupled to the memory; and the processor calling the executable program code stored in the memory to execute the auxiliary diagnosis method as described above.

[0029] Compared with the prior art, the present invention has the following technical effects:

[0030] The present invention can automatically analyze medical images through transformer-type deep learning models and answer doctors' diagnostic questions, thereby improving diagnostic accuracy and efficiency and providing strong support for TAVR surgery decision-making; the system can provide customized diagnostic suggestions based on the specific conditions of different patients to achieve personalized diagnosis; in addition, the system can also be updated online, continuously learn new medical knowledge and cases, improve diagnostic capabilities, and maintain the system's advancement and reliability; the system can encode medical knowledge graphs and / or expert experience into the model to achieve widespread dissemination and application of knowledge and improve medical standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 FIG. 4 is a schematic diagram of an auxiliary diagnosis system according to an embodiment of the present invention.

[0032] Figure 2 Flow chart of an auxiliary diagnosis method in one embodiment of the present invention.

[0033] Figure 3 1 is a diagram of the implementation steps of the visual module in one embodiment of the present invention.

[0034] Figure 4 FIG. 4 is a diagram of implementation steps of a language module in one embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0036] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that well-known algorithms are not shown in detail to avoid obscuring the subject matter of the present invention.

[0037] In addition, the order of execution of actions, steps, etc. in the devices and methods shown in the claims, specifications and drawings can be implemented in any order as long as there is no special explicit limitation on the order and the output of the previous processing is not used in the subsequent processing.

[0038] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0039] Example 1

[0040] This embodiment provides an auxiliary diagnosis system for heart valve rim repair, see Figure 1 , which is based on the trained basic visual question answering model, including:

[0041] Input module 100: used to receive medical images and natural language questions as input; medical images may include various image types such as cardiac CT, MRI, X-ray, etc., and natural language questions may be a doctor's description of the patient's condition or an analysis question of the image.

[0042] Visual module 200: Processing medical images using a visual Transformer model to extract visual features of the image; the encoder of the visual Transformer model automatically learns important features in the image through a self-attention mechanism and performs weighted fusion on different features; the final hidden state of the decoder of the visual Transformer model is used as the visual feature;

[0043] Language module 300: uses a language Transformer model to process questions and generates answers based on the visual features extracted by the visual module; the encoder of the language Transformer model automatically learns the keywords in the question through a self-attention mechanism, and performs weighted fusion on different keywords to handle ambiguity and polysemy in natural language; the decoder of the language Transformer model fuses the visual features to generate answers; at the same time, this module can also handle ambiguity and polysemy in natural language and improve the system's understanding ability.

[0044] Output module 400: Output the generated answer in the form of text or voice. The output module can flexibly select the output mode according to the doctor's needs. At the same time, the system can also associate the answer with relevant medical knowledge and cases to provide more detailed diagnostic information.

[0045] In order to support more natural conversational interactions and improve the usability of the system, the language Transformer model may preferably integrate a known large language model.

[0046] See also Figure 2 The method for implementing auxiliary diagnosis by the auxiliary diagnosis system for heart valve prosthesis as described above comprises the following steps:

[0047] The auxiliary diagnosis system inputs medical images and natural language questions of doctors;

[0048] The visual module of the auxiliary diagnosis system processes the medical image to generate visual features;

[0049] The language module of the auxiliary diagnosis system processes natural language questions and generates answers based on the visual features;

[0050] The answer generated by the language module is output in the form of text or speech.

[0051] From the above-mentioned auxiliary diagnosis system architecture and its implementation method, it can be seen that the system can automatically analyze medical images and answer diagnostic questions raised by doctors based on the analysis results, thereby assisting TAVR surgery decision-making.

[0052] As a preferred technical solution, medical knowledge graphs and / or expert experience can be encoded into the basic visual question-answering model to further enhance the reasoning ability of the system.

[0053] As a preferred technical solution, the auxiliary diagnosis system visualizes the attention distribution of the basic visual question-answering model through a heat map to further improve the interpretability of the system.

[0054] As an example, this embodiment uses a dedicated TAVR surgery-related dataset for model training and testing. The dataset includes:

[0055] (1) Medical imaging:

[0056] Heart valve disease images (such as aortic stenosis, aortic regurgitation, etc.)

[0057] Images of the valve implantation process (such as valve deployment, positioning, etc.)

[0058] Cardiac structural images (such as aorta, left ventricle, etc.)

[0059] (2) Question type:

[0060] Diagnostic questions (eg, "Is aortic stenosis present in this image?")

[0061] Treatment questions (eg, "What is the best TAVR procedure for this type of aortic stenosis?")

[0062] Interpretation questions (e.g., "How is the position and angle of the valve implanted in this image?")

[0063] The visual module adopts the visual Transformer structure, such as Figure 3 As shown, the main steps include:

[0064] (1) Image segmentation: Divide the input image into image blocks of fixed size.

[0065] (2) Linear projection: Linear projection is performed on each image block to generate image embedding.

[0066] (3) Position encoding: adding position encoding information to image embedding.

[0067] (4) Using Transformer encoder: Use multi-layer self-attention mechanism and feed-forward neural network to process image embedding.

[0068] (5) Visual feature generation: Output the final hidden state of the Transformer decoder as the visual feature.

[0069] The language module uses the language Transformer structure, such as Figure 4 As shown, the main steps include:

[0070] (1) Word segmentation: split the input question into words or subwords.

[0071] (2) Word embedding: Convert each word into a word vector.

[0072] (3) Position encoding: Add position encoding information to the word vector.

[0073] (4) Transformer encoder: uses a multi-layer self-attention mechanism and a feedforward neural network to process word vectors.

[0074] (5) Generate answers based on the visual features of the image: Generate answers based on the Transformer decoder output and visual features.

[0075] Basic Visual Question Answering Model Training

[0076] Using an end-to-end approach, the main steps are as follows:

[0077] (1) Data preprocessing: standardize and enhance medical images, and segment and encode questions.

[0078] (2) Loss function: The cross entropy loss function is used to calculate the difference between the predicted answer and the true answer.

[0079] (3) Optimization algorithm: Adam optimizer is used to update parameters.

[0080] (4) Learning rate scheduling: Use the learning rate scheduling strategy of warmup and cosine decay.

[0081] (5) Regularization: Use weight decay and dropout to prevent overfitting.

[0082] Performance Evaluation

[0083] The model performance was evaluated using the following metrics:

[0084] (1) Accuracy: The proportion of questions that the model answers correctly.

[0085] (2) Recall: The proportion of questions answered by the model that contain correct answers.

[0086] (3) F1 score: the harmonic mean of precision and recall.

[0087] (4) Average answer time: the average time it takes the model to answer questions.

[0088] Example 1: Aortic stenosis diagnosis

[0089] enter:

[0090] Medical imaging: CT images of a patient's heart

[0091] Question: "Is aortic stenosis present in this image?"

[0092] Processing process:

[0093] The visual module processes the CT images and generates visual features.

[0094] The language module processes the questions and generates answers based on visual features.

[0095] Output:

[0096] "Yes, the image shows aortic stenosis. The aortic valve area is significantly reduced, and the valve leaflets are calcified and thickened, which are typical features of aortic stenosis. According to image analysis, the stenosis is severe, and further evaluation is recommended for suitability for TAVR surgery."

[0097] Example 2: TAVR surgery planning

[0098] enter:

[0099] Medical imaging: ultrasound images and CT angiography images of patients’ hearts

[0100] Question: "Based on these images, what is the recommended TAVR procedure?"

[0101] Processing process:

[0102] The visual module processes ultrasound images and CT images respectively to generate comprehensive visual features.

[0103] The language module processes the questions and generates answers based on the integrated visual features.

[0104] Output:

[0105] "Based on the image analysis, the transfemoral TAVR approach is recommended. The reasons are as follows:

[0106] Patients with severe aortic valve stenosis are suitable for TAVR surgery.

[0107] The patient's femoral artery diameter was greater than 6 mm, without obvious tortuosity and calcification, and was suitable for the transfemoral artery approach.

[0108] The aortic arch has no obvious angle, which is conducive to the smooth passage of the catheter.

[0109] It is recommended to use a 26mm self-expanding valve to ensure a good fit and seal.

[0110] Example 3: TAVR Intraoperative Guidance

[0111] enter:

[0112] Medical imaging: real-time X-ray fluoroscopy images

[0113] Question: "Is the current valve position and angle appropriate?"

[0114] Processing process:

[0115] The visual module processes real-time X-ray images and generates visual features.

[0116] The language module processes the questions and generates answers based on visual features.

[0117] Output:

[0118] "The current valve position and angle need to be fine-tuned:

[0119] Position: The valve is slightly lower than the aortic valve ring by about 2 mm, and it is recommended to adjust it upward.

[0120] Angle: The angle between the valve and the long axis of the aorta is about 5 degrees, which is slightly tilted. It is recommended to rotate it slightly to make it more vertical.

[0121] Degree of expansion: The valve has expanded by about 80%. Continue to release slowly until it is fully expanded.

[0122] Please make the above adjustments carefully under fluorescent guidance to ensure the best implantation effect of the valve.

[0123] It is understandable that the above-mentioned auxiliary diagnosis system can also combine multiple imaging modalities such as CT, ultrasound, and X-ray to perform multimodal fusion to improve the comprehensiveness and accuracy of diagnosis.

[0124] Example 2

[0125] This embodiment provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the auxiliary diagnosis method as described in Embodiment 1.

[0126] The essence of this technical solution or the part that contributes to the prior art or the part of this technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling an electronic system (which may be a personal computer, a server, or a network system, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0127] Example 3

[0128] This embodiment provides an electronic device, comprising a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the auxiliary diagnosis method as described in Example 1.

[0129] The above basic visual question answering model and its training method can be embodied in the form of a computer program product or a software functional unit. If the above basic visual question answering model and its training method are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0130] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0131] In summary, the present invention provides an auxiliary diagnosis system, method, program product and device for heart valve apposition repair, which automatically answers relevant diagnosis, treatment and interpretation questions by analyzing surgical images and doctors' questions, and provides real-time surgical suggestions and optimization solutions, which can improve the safety and effectiveness of TAVR surgery. The auxiliary diagnosis system is based on a trained basic visual question-answering model, including an input module, a visual module, a language module and an output module. The input module receives medical images and natural language questions as system input. The visual Transformer model processes the medical image and extracts the visual features of the image; the language Transformer model processes the question and generates an answer based on the visual features extracted by the visual module; the output module outputs the generated answer in the form of text or voice.

[0132] Those skilled in the art should understand that those skilled in the art can implement variations by combining the prior art and the above embodiments, which will not be described in detail here. Such variations do not affect the essential content of the present invention, and will not be described in detail here.

[0133] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the systems and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. An auxiliary diagnosis system for heart valve prosthesis, characterized in that: It is based on the trained basic visual question answering model, including: Input module: used to receive medical images and natural language questions as input; Vision module: The visual Transformer model is used to process medical images and extract visual features of the images. The encoder of the visual Transformer model automatically learns important features in the image through a self-attention mechanism and performs weighted fusion of different features. The final hidden state of the decoder of the visual Transformer model is used as the visual feature. Language module: A language Transformer model is used to process questions and generate answers based on the visual features extracted by the visual module; the encoder of the language Transformer model automatically learns the keywords in the question through a self-attention mechanism and performs weighted fusion of different keywords to handle ambiguity and polysemy in natural language; the decoder of the language Transformer model fuses the visual features to generate answers; Output module: Outputs the generated answer in the form of text or voice.

2. The auxiliary diagnosis system for heart valve rim repair according to claim 1, characterized in that: The language Transformer model integrates a large language model.

3. An auxiliary diagnosis method based on the auxiliary diagnosis system for heart valve rim repair according to claim 1 or 2, characterized in that: The following steps are involved: The auxiliary diagnosis system inputs medical images and natural language questions of doctors; The visual module of the auxiliary diagnosis system processes the medical image to generate visual features; The language module of the auxiliary diagnosis system processes natural language questions and generates answers based on the visual features; The answer generated by the language module is output in the form of text or speech.

4. The auxiliary diagnosis method according to claim 3, characterized in that: The visual module first divides the input image into image blocks of fixed size, then performs linear projection on each image block to generate image embedding, then embeds and adds position encoding information to the image embedding, uses the encoder of the visual transformer based on the multi-layer self-attention mechanism and feedforward neural network to process the image embedding, and finally outputs the final hidden state of the decoder of the visual transformer as the visual feature.

5. The auxiliary diagnosis method according to claim 3, characterized in that: The language module first divides the input question into words or subwords, then converts each word into a word vector, adds position encoding information to the word vector, processes the word vector using an encoder of a language transformer based on a multi-layer self-attention mechanism and a feedforward neural network, and finally generates an answer based on the decoder output of the language transformer and visual features.

6. The auxiliary diagnosis method according to claim 3, characterized in that: The basic visual question answering model is trained in an end-to-end manner, and the training steps include: Data preprocessing: standardize and enhance medical images, segment and encode questions; Loss function design: Use the cross entropy loss function to calculate the difference between the predicted answer and the true answer; Optimization algorithm: Adam optimizer is used to update parameters; Learning rate scheduling: Using warmup and cosine decay learning rate scheduling strategy; Regularization: Use weight decay and dropout to prevent overfitting.

7. The auxiliary diagnosis method according to claim 3, characterized in that: Encoding medical knowledge graphs and / or expert experience into basic visual question answering models.

8. The auxiliary diagnosis method according to claim 3, characterized in that: The auxiliary diagnosis system visualizes the attention distribution of the basic visual question answering model in the form of a heat map.

9. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the auxiliary diagnosis method according to any one of claims 3 to 8.

10. An electronic device, characterized in that: A memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the auxiliary diagnosis method according to any one of claims 3 to 8.

Images