Method, device, system, equipment and medium for generating guidance information

Abstract

The disclosure provides a guidance information generation method, device, system, equipment and medium, the generation method comprises: obtaining a target intracardiac ultrasound image corresponding to a simulated heart based on a pre-constructed three-dimensional simulated heart model; inputting the target intracardiac ultrasound image into a pre-constructed sector analysis model to output a target sector analysis result; inputting the target sector analysis result into a pre-trained question and answer model to generate target guidance information matched with intracardiac ultrasound operation of the simulated heart. According to the target intracardiac ultrasound image, the disclosure obtains the target sector analysis result, inputs the target sector analysis result into the pre-trained question and answer model, and outputs the target guidance information matched with the intracardiac ultrasound operation of the simulated heart, realizes real-time and automatic feedback of the intracardiac ultrasound operation, improves the learning efficiency of the operation object, and can better promote the teaching guidance of the intracardiac ultrasound operation.

Description

Technical Field

[0001] This disclosure relates to the field of intracardiac ultrasound operation technology, and more particularly to a method, apparatus, system, device, and medium for generating guidance information. Background Technology

[0002] Currently, intracardiac ultrasound (ICE) teaching generally relies on traditional text-based tutorials or simple two-dimensional image displays, which cannot effectively present the three-dimensional anatomical structure of the heart and provide real-time feedback on catheter manipulation. Although some companies have launched two-dimensional or three-dimensional simulation systems, these systems typically rely on the guidance of professional instructors and cannot achieve automated feedback and personalized learning. Traditional simulators also suffer from the drawbacks of lacking feedback or providing insufficiently intelligent feedback, which is particularly inconvenient when efficiently training intracardiac ultrasound operation skills. Summary of the Invention

[0003] The technical problem to be solved by this disclosure is to overcome the shortcomings of existing technologies that use two-dimensional imaging, two-dimensional or three-dimensional simulation systems for ultrasound teaching in centers, resulting in untimely and unintelligent feedback on intracardiac ultrasound operations. This disclosure provides a method, device, system, equipment and medium for generating guidance information.

[0004] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0005] This disclosure provides a method for generating guidance information, the method comprising:

[0006] Based on a pre-built three-dimensional simulated heart model, a target intracardiac ultrasound image of the corresponding simulated heart is obtained;

[0007] The target intracardiac ultrasound image is input into a pre-constructed sector analysis model to output the target sector analysis results;

[0008] The target sector analysis results are input into a pre-trained question-and-answer model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart.

[0009] Optionally, the step of inputting the target sector analysis results into the question-answering model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart includes:

[0010] Acquire the image information of the operation object, wherein the operation object is the object corresponding to the target intracardiac ultrasound image for which intracardiac ultrasound operation is performed on the simulated heart;

[0011] The target sector analysis results and the image information are input into the question-and-answer model to generate the target guidance information that matches the intracardiac ultrasound operation of the simulated heart;

[0012] And / or,

[0013] After the step of inputting the target sector analysis results into the question-answering model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart, the method further includes:

[0014] Based on the target guidance information, several types of feedback information are generated;

[0015] The types include at least one of text type, voice type, vibration type and image type.

[0016] Optionally, the training steps of the question-answering model include:

[0017] Based on the pre-constructed three-dimensional simulated heart model, several sample intracardiac ultrasound images of the corresponding simulated heart are obtained.

[0018] The intracardiac ultrasound images of the sample are input into the pre-constructed sector analysis model to output the sample sector analysis results;

[0019] Obtain sample association information that matches the sample sector analysis results, wherein the sample association information is information associated with the operation object performing intracardiac ultrasound on the simulated heart;

[0020] The results of the analysis of several sets of sample sectors and the matching sample association information are input into a preset large language model to train a question-and-answer model for guiding intracardiac ultrasound operations.

[0021] Optionally, the sample association information includes sample guidance information that matches the intracardiac ultrasound operation and sample operation status information of the intracardiac ultrasound operation.

[0022] The sample operation status information includes at least one of the following: operation stage information of the intracardiac ultrasound operation, error type information of the intracardiac ultrasound operation, image information of the operation object, historical operation information of the operation object, and instruction information of the operation object.

[0023] Optionally, the step of inputting several sets of sample sector analysis results and the matched association information into a preset large language model to train a question-and-answer model for guiding intracardiac ultrasound operations includes:

[0024] Based on the sample sector analysis results and the sample operation status information, prompt words are constructed;

[0025] The prompt words and the sample guidance information are input into the preset large language model to train the question-answering model.

[0026] Optionally, after the step of inputting several sets of sample sector analysis results and the matched sample association information into a preset large language model to train a question-and-answer model for guiding intracardiac ultrasound operations, the method further includes:

[0027] Acquire operational knowledge of intracardiac ultrasound procedures from several experts;

[0028] The operational knowledge includes at least one of the following: expert guidance records for intracardiac ultrasound operation, expert operational thinking for intracardiac ultrasound operation, and expert teaching and training content for intracardiac ultrasound operation.

[0029] Based on the operational knowledge, the question-answering model is adjusted to obtain the adjusted question-answering model;

[0030] And / or,

[0031] The training steps for the sector analysis model include:

[0032] Obtain the image segmentation model trained using several sets of sample training data;

[0033] Each set of sample training data includes intracardiac ultrasound images of the sample and the corresponding sample segmentation results;

[0034] The image segmentation model is used as the teacher model, and the task corresponding to the teacher model is decoupled to obtain several branch tasks.

[0035] Based on the aforementioned branch tasks, several student models are distilled in parallel.

[0036] Each of the branch tasks corresponds to a student model, which is trained using a geometric consistency loss function and includes at least one of a detection model, a segmentation model, and a classification model.

[0037] By sharing the network weights of several student models, the sector analysis model is obtained.

[0038] This disclosure provides a guidance information generation apparatus, the apparatus comprising:

[0039] The target image acquisition module is used to obtain the target intracardiac ultrasound image of the corresponding simulated heart based on a pre-built three-dimensional simulated heart model.

[0040] The target result acquisition module is used to input the target intracardiac ultrasound image into a pre-constructed sector analysis model to output the target sector analysis result;

[0041] The guidance information generation module is used to input the target sector analysis results into a pre-trained question-and-answer model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart.

[0042] Optionally, the guidance information generation module includes:

[0043] The image information acquisition unit is used to acquire the image information of the operation object, wherein the operation object is the object that performs intracardiac ultrasound operation on the simulated heart corresponding to the target intracardiac ultrasound image;

[0044] The guidance information generation unit is used to input the target sector analysis results and the portrait information into the question-and-answer model to generate the target guidance information that matches the intracardiac ultrasound operation of the simulated heart;

[0045] And / or,

[0046] The generating apparatus further includes:

[0047] The feedback information generation module is used to generate several types of feedback information based on the target guidance information;

[0048] The types include at least one of text type, voice type, vibration type and image type.

[0049] Optionally, the generating apparatus further includes:

[0050] The sample image acquisition module is used to obtain several sample intracardiac ultrasound images of the corresponding simulated heart based on the pre-constructed three-dimensional simulated heart model.

[0051] The sample result acquisition module is used to input the intracardiac ultrasound image of the sample into the pre-constructed sector analysis model to output the sample sector analysis result;

[0052] The association information acquisition module is used to acquire sample association information that matches the sample sector analysis results. The sample association information is information associated with the operation object performing intracardiac ultrasound operation on the simulated heart.

[0053] The question-answering model training module is used to input several sets of sample sector analysis results and the matched sample association information into a preset large language model to train a question-answering model for guiding intracardiac ultrasound operations.

[0054] Optionally, the sample association information includes sample guidance information that matches the intracardiac ultrasound operation and sample operation status information of the intracardiac ultrasound operation.

[0055] The sample operation status information includes at least one of the following: operation stage information of the intracardiac ultrasound operation, error type information of the intracardiac ultrasound operation, image information of the operation object, historical operation information of the operation object, and instruction information of the operation object.

[0056] Optionally, the question-answering model training module includes:

[0057] The prompt word construction unit is used to construct prompt words based on the sample sector analysis results and the sample operation status information;

[0058] The question-answering model training unit is used to input the prompt words and the sample guidance information into the preset large language model to train the question-answering model.

[0059] Optionally, the generating apparatus further includes:

[0060] The operation knowledge acquisition module is used to acquire the operation knowledge of intracardiac ultrasound operations described by several experts.

[0061] The operational knowledge includes at least one of the following: expert guidance records for intracardiac ultrasound operation, expert operational thinking for intracardiac ultrasound operation, and expert teaching and training content for intracardiac ultrasound operation.

[0062] The model adjustment module is used to adjust the question-answering model based on the operational knowledge to obtain the adjusted question-answering model;

[0063] And / or,

[0064] The generating apparatus further includes:

[0065] The segmentation model acquisition module is used to acquire the image segmentation model trained using several sets of sample training data.

[0066] Each set of sample training data includes intracardiac ultrasound images of the sample and the corresponding sample segmentation results;

[0067] The task decoupling module is used to take the image segmentation model as the teacher model and perform task decoupling processing on the task corresponding to the teacher model to obtain several branch tasks.

[0068] The model distillation module is used to distill several student models in parallel based on several of the aforementioned branch tasks;

[0069] Each of the branch tasks corresponds to a student model, which is trained using a geometric consistency loss function and includes at least one of a detection model, a segmentation model, and a classification model.

[0070] The weight sharing module is used to share the network weights of several student models to obtain the sector analysis model.

[0071] This disclosure also provides a guidance information generation system, the generation system including the generation device as described above, as well as a catheter, a handle for operating the catheter, and a data acquisition module;

[0072] The handle is used to move the catheter within the receiving space of the data acquisition module based on external operation, so as to change the position information of the catheter.

[0073] The data acquisition module is used to acquire the pose information of the catheter and output it to the generation device to change the display state of the catheter in the three-dimensional simulated heart model. The display state is used to characterize the pose information of the catheter.

[0074] The generating device is also used to output a target intracardiac ultrasound image corresponding to the simulated heart based on the display state.

[0075] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the above-described method for generating guidance information.

[0076] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for generating guidance information.

[0077] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method for generating guidance information as described above.

[0078] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0079] The positive and progressive effects of this disclosure are as follows:

[0080] This disclosure obtains target sector analysis results from target intracardiac ultrasound images, inputs the target sector analysis results into a pre-trained question-and-answer model, and outputs target guidance information that matches the intracardiac ultrasound operation of the simulated heart. This realizes real-time and automated feedback of intracardiac ultrasound operation, improves the learning efficiency of the operation subjects, and can better promote the teaching and guidance of intracardiac ultrasound operation. Attached Figure Description

[0081] Figure 1 This is a first flowchart of a method for generating guidance information according to Embodiment 1 of this disclosure;

[0082] Figure 2 This is a first flowchart of the training process of the question-answering model in the method for generating guidance information according to Embodiment 1 of this disclosure;

[0083] Figure 3 This is a flowchart of step S134 of the training process of the question-answering model in the method for generating guidance information according to Embodiment 1 of this disclosure;

[0084] Figure 4 This is a second flowchart of the training process of the question-answering model in the method for generating guidance information according to Embodiment 1 of this disclosure;

[0085] Figure 5 This is a third flowchart illustrating the training process of the question-answering model in the method for generating guidance information according to Embodiment 1 of this disclosure;

[0086] Figure 6 This is a flowchart of step S13 in the method for generating guidance information according to Embodiment 1 of this disclosure;

[0087] Figure 7 This is a second flowchart of the method for generating guidance information according to Embodiment 1 of this disclosure;

[0088] Figure 8 This is a schematic diagram of the first module of the guidance information generation device according to Embodiment 2 of this disclosure;

[0089] Figure 9 This is a schematic diagram of the second module of the guidance information generation device according to Embodiment 2 of this disclosure;

[0090] Figure 10 This is a schematic diagram of the structure of the guidance information generation system of Embodiment 3 of this disclosure;

[0091] Figure 11 This is a schematic diagram of the modules of the guidance information generation system of Embodiment 3 of this disclosure;

[0092] Figure 12 This is a schematic diagram of the structure of the electronic device according to Embodiment 4 of this disclosure. Detailed Implementation

[0093] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0094] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the context of the embodiments, and the use of such prefixes should not constitute unnecessary restrictions. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0095] In this embodiment of the disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of personal information of the operational objects involved all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0096] Example 1

[0097] This embodiment provides a method for generating guidance information, such as... Figure 1 As shown, the generation method includes:

[0098] S11. Based on a pre-built three-dimensional simulated heart model, obtain the target intracardiac ultrasound image of the corresponding simulated heart;

[0099] S12. Input the target intracardiac ultrasound image into the pre-constructed sector analysis model to output the target sector analysis results;

[0100] S13. Input the target sector analysis results into the pre-trained question-and-answer model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart.

[0101] Specifically, during catheter navigation and image localization in intracardiac ultrasound, the physical location of the catheter is synchronized in real time with a three-dimensional simulated heart model in the virtual world, thereby guiding the acquisition of intracardiac ultrasound images.

[0102] First, the patient operates the catheter handle. The force feedback sensor attached to the catheter transmits the patient's hand movements, such as pushing, pulling, rotating, and bending, to the catheter body via the handle, ultimately changing the position and orientation of the catheter tip inside the simulated heart. Force feedback and position sensors installed on the catheter handle or tip transmit these changes in real-time to the 3D simulated heart model via the catheter connection cable. The 3D simulated heart model then displays and updates the position and orientation of a virtual icon representing the ultrasound catheter tip. The tip of the intracardiac ultrasound catheter has a transducer that emits and receives ultrasound waves, forming a fan-shaped two-dimensional ultrasound cross-section. The orientation of this fan is entirely determined by the position and angle of the catheter tip. Using the newly acquired and updated virtual position of the catheter tip, the 3D simulated heart model calculates the fan-shaped plane seen by the ultrasound transducer at this position and angle, and visually displays this calculated ultrasound scanning area—the intracardiac ultrasound image—as a semi-transparent fan-shaped plane. Through the 3D simulated heart model and biomechanical simulation, accurate 3D anatomical simulation is provided, helping the patient better understand the spatial relationships within the heart.

[0103] The target intracardiac ultrasound image can be obtained through the above-described method of acquiring intracardiac ultrasound images.

[0104] The target intracardiac ultrasound image is input into a pre-built sector analysis model for sector analysis, outputting the corresponding target sector analysis results. The sector analysis model includes at least one of a detection model, a segmentation model, and a classification model. The target intracardiac ultrasound image is input into the detection model to output the detection result, i.e., the target sector analysis result, such as the puncture needle and its orientation at 30 degrees. The target intracardiac ultrasound image is input into the segmentation model to output the segmentation result, i.e., the target sector analysis result, such as left atrium, right atrium, and atrial septum. The target intracardiac ultrasound image is input into the classification model to output the classification result, i.e., the target sector analysis result, such as the sector being at the puncture position. By using the pre-built sector analysis model to quantify key parameters such as catheter angle and insertion depth, the operator can complete clinically standard-compliant skills training in an unsupervised environment.

[0105] For a target intracardiac ultrasound image, by using a sector analysis model to extract the structure of the left atrium, we can know that there is a left atrium in the image, which may mean that the probe is located inside the left atrium.

[0106] The detection results of the sector analysis model, i.e. the target sector analysis results, such as the left atrium, are input into the question-and-answer model. The model can then infer and analyze the results to conclude that the probe is located in the left atrium, and analyze the data to output target guidance information.

[0107] In this scheme, the target sector analysis results are obtained based on the target intracardiac ultrasound image. The target sector analysis results are then input into a pre-trained question-and-answer model to output target guidance information that matches the intracardiac ultrasound operation of the simulated heart. This achieves real-time and automated feedback for intracardiac ultrasound operation, improves the learning efficiency of the operator, and can better promote the teaching and guidance of intracardiac ultrasound operation.

[0108] In a feasible solution, such as Figure 2 As shown, before step S13, there is also a training step for the question-answering model, which specifically includes:

[0109] S131. Based on a pre-constructed three-dimensional simulated heart model, obtain several sample intracardiac ultrasound images of the corresponding simulated heart.

[0110] S132. Input the intracardiac ultrasound images of the sample into the pre-constructed sector analysis model to output the sample sector analysis results;

[0111] S133. Obtain sample association information that matches the sample sector analysis results. The sample association information is information related to the intracardiac ultrasound operation performed on the simulated heart by the operation object.

[0112] S134. Input the results of sector analysis of several sets of samples and the matching sample association information into the preset large language model to train a question-and-answer model for guiding intracardiac ultrasound operation.

[0113] Specifically, the acquisition of intracardiac ultrasound images of the sample is similar to that of the target intracardiac ultrasound images, and will not be described in detail here.

[0114] By using different methods of acquiring intracardiac ultrasound images, several sample intracardiac ultrasound images can be obtained based on the different hand movements of the subject.

[0115] Different hand movements of the simulated heart generate information associated with intracardiac ultrasound manipulation, known as sample correlation information. This sample correlation information corresponds to the intracardiac ultrasound images of the sample, and thus matches the results of the sample sector analysis.

[0116] The results of sector analysis of several sets of samples and the matching sample association information are input into a pre-set large language model (LLM) to train a question-and-answer model for guiding intracardiac ultrasound operations.

[0117] This question-answering model, trained with a large amount of data, is able to provide personalized guidance while maintaining professional opinions, based on the analysis results of several sets of sample sectors and the matching sample association information.

[0118] In this scheme, a question-and-answer model for guiding intracardiac ultrasound operations is trained by analyzing several sets of sample intracardiac ultrasound images and matching information related to intracardiac ultrasound operations performed on a simulated heart. This ensures the accuracy and reliability of the question-and-answer model in guiding intracardiac ultrasound operations.

[0119] In one feasible approach, the sample association information includes sample guidance information that matches the intracardiac ultrasound procedure and sample operation status information of the intracardiac ultrasound procedure.

[0120] The sample operation status information includes at least one of the following: operation stage information of intracardiac ultrasound operation, error type information of intracardiac ultrasound operation, image information of the operation object, historical operation information of the operation object, and instruction information of the operation object.

[0121] Specifically, the different hand movements of the patient during the procedure encompass information from multiple stages of intracardiac ultrasound, such as probe insertion, section acquisition, and pathological assessment. Errors can occur due to these hand movements; recording these errors creates error type information for intracardiac ultrasound, including common mistakes, which can be stored in an error pattern recognition database. Different patients are associated with different profiles, representing their background, such as medical students, residents, or specialist trainees. The patient's operational trajectory is recorded to create historical operational information. During the intracardiac ultrasound procedure, the patient can also issue commands, such as asking questions.

[0122] The above information serves as sample operation status information for intracardiac ultrasound procedures. Together with sample guidance information, it forms sample association information. The sample association information, along with the sample sector analysis results, is input into a pre-set large language model to train a question-and-answer model for guiding intracardiac ultrasound procedures.

[0123] This question-answering model offers a new technological path for medical education innovation. Trained on massive amounts of medical data, the model can generate personalized teaching plans. Its multimodal processing capabilities allow for the simultaneous parsing of image data and operational instructions, enabling users to ask questions at any time during the process to obtain guidance on the current status. This model overcomes the limitations of traditional teaching in real-time interaction and adaptive learning.

[0124] Integrating domain-specific LLMs finely tuned from tens of thousands of clinical cases, it generates personalized guidance plans that conform to expert diagnosis and treatment logic by dynamically analyzing operational status information, sector parameters (i.e., sample sector analysis results) and student interaction questions. This enables dynamic display of the three-dimensional structure of the heart and visual guidance of the operational path.

[0125] In this solution, the sample association information includes operational stage perception, user profile adaptation, error pattern recognition library, context perception module, and user instruction information, which enables the question answering model to provide personalized guidance while maintaining professional opinions, thereby improving the accuracy and reliability of the question answering model.

[0126] In a feasible solution, such as Figure 3 As shown, step S134 includes:

[0127] S1341. Based on the sample sector analysis results and sample operation status information, construct prompt words;

[0128] S1342. Input the prompt words and sample guidance information into the preset large language model to train the question answering model.

[0129] Specifically, the sample sector analysis results are combined with information on the operation stage of intracardiac ultrasound, error type of intracardiac ultrasound, portrait information of the operator, historical operation information of the operator, and instruction information of the operator to construct a multi-dimensional prompt template. Based on the prompt and sample guidance information, a question-and-answer model is trained to ensure that the question-and-answer model can automatically adjust the content of the guidance information according to the different backgrounds, operation stages, and error types of the operator, thus ensuring a personalized teaching experience.

[0130] In this approach, prompt words for a large language model are constructed, which are then used to train the question-answering model. This ensures the effectiveness of model training and improves the accuracy and reliability of the question-answering model.

[0131] In a feasible solution, such as Figure 4 As shown, after step S134 and before step S13, the following steps are also included:

[0132] S135. Acquire operational knowledge of intracardiac ultrasound procedures from several experts;

[0133] Among them, operational knowledge includes at least one of the following: expert guidance records for intracardiac ultrasound operation, expert operational thinking for intracardiac ultrasound operation, and teaching and training content for expert intracardiac ultrasound operation.

[0134] S136. Based on operational knowledge, adjust the question-answering model to obtain the adjusted question-answering model.

[0135] Specifically, it integrates and fine-tunes the medical LLM with personalized guidance records from different experts to achieve intelligent Q&A functionality, providing trainees with real-time question answers and learning support; it also fine-tunes the LLM based on the experts' operational approaches and teaching and training content, making it easier for experts to teach and promote it.

[0136] In this solution, the question-and-answer model is fine-tuned by experts to facilitate teaching and promotion by experts.

[0137] In a feasible solution, such as Figure 5 As shown, before step S132, there is also a training step for the sector analysis model, which specifically includes:

[0138] S1321. Obtain the image segmentation model trained using several sets of sample training data;

[0139] Each set of training data includes intracardiac ultrasound images of the samples and the corresponding sample segmentation results;

[0140] S1322. Using the image segmentation model as the teacher model, the tasks corresponding to the teacher model are decoupled to obtain several branch tasks.

[0141] S1323. Based on several branch tasks, several student models are distilled in parallel.

[0142] Each branch task corresponds to a student model, which is trained using the geometric consistency loss function. The student model includes at least one of the detection model, segmentation model, and classification model.

[0143] S1324. By sharing the network weights of several student models, a sector analysis model is obtained.

[0144] Specifically, we obtain the image segmentation model ICE-SAM (Segment Anything Model, an AI general model focused on image segmentation tasks) that has been pre-trained on a large-scale ICE dataset. The encoder of ICE-SAM has an ultrasound-specific global-local feature space.

[0145] Using the image segmentation model as the teacher model, a task decoupling distillation framework is constructed. The tasks corresponding to the teacher model are decoupled to obtain several branch tasks. This framework includes at least a detection branch, a segmentation branch, and a classification branch.

[0146] For each branch, only the sub-feature map of the corresponding task-related channel in the ICE-SAM encoder is extracted, and the spatial error between the sub-feature map and the standard anatomical plane is constrained by the geometric consistency loss function.

[0147] Parallel distillation of each branch is performed using a small number of labeled samples to obtain the corresponding detection model, segmentation model, and classification model. The detection model, segmentation model, and classification model only learn the latent space submanifolds in the image segmentation model that are relevant to their own task.

[0148] By sharing the backbone network weights of the detection model, segmentation model, and classification model, a lightweight, multi-task sector analysis model is formed.

[0149] By employing knowledge distillation technology, large-scale medical models are compressed into lightweight inference engines, reducing operational feedback latency to the millisecond level. This ensures real-time synchronous analysis of catheter movement trajectory and ultrasound imaging. While guaranteeing real-time analysis, the accuracy of large-parameter models is preserved, and a high-efficiency model with a small number of parameters is used as the core of real-time operational feedback, providing operational guidance and skills assessment for the user.

[0150] In this approach, ultrasound-specific SAM is used as the teacher model. Through task decoupling, geometric consistency loss function constraints, and shared shallow weights, a sector analysis model that includes three real-time models—detection, segmentation, and classification—can be obtained simultaneously with only a small number of annotations, significantly reducing the threshold for clinical implementation.

[0151] In a feasible solution, such as Figure 6 As shown, step S13 includes:

[0152] S137. Obtain the image information of the operation object, which is the object corresponding to the target intracardiac ultrasound image for performing intracardiac ultrasound operation on the simulated heart.

[0153] S138. Input the target sector analysis results and image information into the question-and-answer model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart.

[0154] During the training of the question-answering model, a multi-dimensional prompt word template was constructed based on the operational stage information, error type information, patient profile information, patient history information, and instruction information of the patient. During the application of the question-answering model, the multi-dimensional prompt word template can be adjusted according to the patient profile information to obtain real-time prompt words. Based on these prompt words, the question-answering model can generate target guidance information that matches the simulated cardiac ultrasound operation. The patient profile information represents the patient's background, such as medical student, resident physician, or specialist training.

[0155] The user can first output instructions, such as the text command "What is the current sector doing?". The question-answering model understands the text command, calls the sector analysis model, and outputs the target sector analysis result, such as "puncture needle," puncture needle direction "30 degrees," "left atrium," "right atrium," and "atrial septum." The target sector analysis result is input into the question-answering model, which combines the key information contained in the text command, the user's profile information, and historical dialogues to answer the user's text command. Role attributes can be pre-set for the question-answering model so that its responses to the user's text commands conform to the characteristics of the role. For example, if the role attribute is a legal consulting assistant, the question-answering model can summarize and answer legal-related questions. Rules can also be pre-set for the question-answering model, such as outputting healthy and positive content, and using language that is friendly, logical, and summarizing.

[0156] The question-answering model can dynamically identify the current operation stage, such as probe insertion, section acquisition, and pathological judgment; adapt to the profile information of the operation object and customize learning content according to the background of the operation object, such as medical students, resident physicians, and specialist training; identify common operation errors and provide corrective opinions in real time; record the operation trajectory of the operation object to achieve context awareness and provide accurate guidance based on historical operation data.

[0157] In this solution, the learning content is customized according to the background of the patient, thus achieving personalized guidance for intracardiac ultrasound operation.

[0158] In a feasible solution, such as Figure 7 As shown, after step S13, the following steps are also included:

[0159] S14. Based on the target guidance information, generate several types of feedback information;

[0160] The types include at least one of text type, voice type, vibration type and image type.

[0161] Specifically, the interactive terminal can provide feedback on the target guidance information output by the question-answering model in a multimodal manner, including text, voice, vibration, and images.

[0162] This solution provides comprehensive operational guidance to users by using multimodal feedback methods, including text, voice, vibration, and images, thereby enhancing the user experience.

[0163] In this embodiment, the target sector analysis results are obtained based on the target intracardiac ultrasound image. The target sector analysis results are then input into a pre-trained question-and-answer model to output target guidance information that matches the intracardiac ultrasound operation of the simulated heart. This achieves real-time and automated feedback for intracardiac ultrasound operation, improves the learning efficiency of the operator, and can better promote the teaching and guidance of intracardiac ultrasound operation.

[0164] The working principle of the guidance information generation method in this embodiment is explained below with specific examples:

[0165] Example 1: The operator manipulates the handle of the catheter to simulate atrial septal puncture exploration, using a puncture needle to simulate passing from the right atrium through the interatrial septum into the left atrium. During the operation, there are three consecutive angular deviations greater than 15 degrees. The handle outputs the operator's operation status to the catheter, changing the catheter's pose information. The force feedback sensor attached to the catheter collects the catheter's pose information and outputs it to a 3D simulated heart model to change the display status of the 3D simulated heart model. Based on the display status of the 3D simulated heart model, a target intracardiac ultrasound image of the corresponding simulated heart is obtained. The target intracardiac ultrasound image is input into a pre-constructed sector analysis model to output the target sector analysis result. The target sector analysis result is input into a question-and-answer model, which outputs target guidance information corresponding to the operator's operation status, including freezing the operation and activating the 3D simulated heart model to display an error comparison view, generating a voice prompt "Please rotate the probe 20 degrees towards the specified sector," and driving the computer's vibration module to generate a rightward pulse guidance.

[0166] Example 2: The operator manipulates the catheter handle to simulate atrial septal puncture, using a puncture needle to simulate passing through the interatrial septum from the right atrium into the left atrium. The handle outputs the operator's operational status to the catheter, changing the catheter's pose information. The force feedback sensor attached to the catheter collects the catheter's pose information and outputs it to a 3D simulated heart model, changing the display status of the 3D simulated heart model. Based on the display status of the 3D simulated heart model, a target intracardiac ultrasound image of the corresponding simulated heart is obtained. The target intracardiac ultrasound image is input into a pre-constructed sector analysis model to output the target sector analysis result. The target sector analysis result is input into a question-and-answer model, which outputs target guidance information corresponding to the operator's operational status, including the text message "Suspected thrombus detected," automatically marking abnormal areas in the 3D simulated heart model with red flashing, and pushing differential diagnosis question-and-answer options, such as: A thrombus is currently detected in the left atrial appendage, and the following measures can be taken: This is located in the left atrial appendage, which is generally a thrombus-prone area, and the detected suspicious object may be a thrombus; ECG, X-ray, and other operations can be performed to further determine whether it is a thrombus.

[0167] Example 2

[0168] Corresponding to the aforementioned embodiment of a method for generating guidance information, this disclosure also provides an embodiment of an apparatus for generating guidance information.

[0169] like Figure 8 As shown, the generating apparatus includes:

[0170] The target image acquisition module 11 is used to obtain the target intracardiac ultrasound image of the corresponding simulated heart based on a pre-built three-dimensional simulated heart model.

[0171] The target result acquisition module 12 is used to input the target intracardiac ultrasound image into the pre-constructed sector analysis model to output the target sector analysis result;

[0172] The guidance information generation module 13 is used to input the target sector analysis results into a pre-trained question-and-answer model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart.

[0173] In a feasible solution, such as Figure 9 As shown, the guidance information generation module 13 includes:

[0174] The image information acquisition unit 131 is used to acquire the image information of the operation object, which is the object that performs intracardiac ultrasound operation on the simulated heart corresponding to the target intracardiac ultrasound image.

[0175] The guidance information generation unit 132 is used to input the target sector analysis results and image information into the question-and-answer model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart.

[0176] And / or,

[0177] The generating apparatus also includes:

[0178] Feedback information generation module 14 is used to generate several types of feedback information based on target guidance information;

[0179] The types include at least one of text type, voice type, vibration type and image type.

[0180] In one feasible embodiment, the generating apparatus further includes:

[0181] The sample image acquisition module 15 is used to obtain several sample intracardiac ultrasound images of the corresponding simulated heart based on a pre-built three-dimensional simulated heart model.

[0182] The sample result acquisition module 16 is used to input the intracardiac ultrasound images of the sample into a pre-built sector analysis model to output the sample sector analysis results;

[0183] The association information acquisition module 17 is used to acquire sample association information that matches the sample sector analysis results. The sample association information is information associated with the operation object performing intracardiac ultrasound on the simulated heart.

[0184] The question-answering model training module 18 is used to input the results of several sets of sample sector analysis and the matching sample association information into a preset large language model to train a question-answering model for guiding intracardiac ultrasound operations.

[0185] In one feasible approach, the sample association information includes sample guidance information that matches the intracardiac ultrasound procedure and sample operation status information of the intracardiac ultrasound procedure.

[0186] The sample operation status information includes at least one of the following: operation stage information of intracardiac ultrasound operation, error type information of intracardiac ultrasound operation, image information of the operation object, historical operation information of the operation object, and instruction information of the operation object.

[0187] In a feasible solution, the question-answering model training module 18 includes:

[0188] The prompt word construction unit 181 is used to construct prompt words based on the sample sector analysis results and sample operation status information;

[0189] The question-answering model training unit 182 is used to input prompt words and sample guidance information into a preset large language model to train a question-answering model.

[0190] In one feasible embodiment, the generating apparatus further includes:

[0191] Operational knowledge acquisition module 19 is used to acquire operational knowledge of intracardiac ultrasound operations from several experts.

[0192] Among them, operational knowledge includes at least one of the following: expert guidance records for intracardiac ultrasound operation, expert operational thinking for intracardiac ultrasound operation, and teaching and training content for expert intracardiac ultrasound operation.

[0193] The model adjustment module 20 is used to adjust the question-answering model based on operational knowledge to obtain the adjusted question-answering model;

[0194] And / or,

[0195] The generating apparatus also includes:

[0196] The segmentation model acquisition module 21 is used to acquire the image segmentation model trained using several sets of sample training data;

[0197] Each set of training data includes intracardiac ultrasound images of the samples and the corresponding sample segmentation results;

[0198] The task decoupling module 22 is used to decouple the image segmentation model as the teacher model and perform task decoupling on the corresponding task of the teacher model to obtain several branch tasks.

[0199] Model distillation module 23 is used to distill several student models in parallel based on several branch tasks;

[0200] Each branch task corresponds to a student model, which is trained using the geometric consistency loss function. The student model includes at least one of the detection model, segmentation model, and classification model.

[0201] The weight sharing module 24 is used to share the network weights of several student models to obtain the sector analysis model.

[0202] In this embodiment, the target sector analysis results are obtained based on the target intracardiac ultrasound image. The target sector analysis results are then input into a pre-trained question-and-answer model to output target guidance information that matches the intracardiac ultrasound operation of the simulated heart. This achieves real-time and automated feedback for intracardiac ultrasound operation, improves the learning efficiency of the operator, and can better promote the teaching and guidance of intracardiac ultrasound operation.

[0203] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0204] Example 3

[0205] This embodiment provides a system for generating guidance information, such as Figure 10 As shown, the generation system includes a guidance information generation device 2 as described above, a catheter 31, a handle 32 for operating the catheter 31, and a data acquisition module 33;

[0206] The handle 32 is used to move the catheter 31 within the receiving space of the data acquisition module 33 based on external operation, so as to change the position information of the catheter 31.

[0207] The data acquisition module 33 is used to acquire the pose information of the catheter 31 and output it to the generation device 2 to change the display state of the catheter 31 in the three-dimensional simulated heart model. The display state is used to characterize the pose information of the catheter 31.

[0208] The generating device 2 is also used to output a target intracardiac ultrasound image corresponding to the simulated heart based on the display state.

[0209] Specifically, during catheter navigation and image localization in intracardiac ultrasound, the physical catheter position is synchronized in real time with a virtual 3D simulated heart model, thereby guiding the acquisition of intracardiac ultrasound images. The data acquisition module includes force feedback sensors and position sensors. The 3D simulated heart model is set up in a computer or other terminal device.

[0210] The operator manipulates the catheter handle. A force feedback sensor attached to the catheter transmits the operator's hand movements, such as pushing, pulling, rotating, and bending, to the catheter body via the handle, ultimately changing the position and orientation of the catheter tip within the simulated heart—its pose information. Force feedback and position sensors mounted on the catheter handle or tip transmit these changes in real-time to the 3D simulated heart model via the catheter connection cable. The 3D simulated heart model then displays and updates the position and orientation of a virtual icon representing the ultrasound catheter tip, thus changing the model's display state. The intracardiac ultrasound catheter tip has a transducer that emits and receives ultrasound waves, forming a fan-shaped two-dimensional ultrasound cross-section. The orientation of this fan is entirely determined by the position and angle of the catheter tip. The 3D simulated heart model, using the newly acquired and updated virtual position of the catheter tip, calculates the fan-shaped plane seen by the ultrasound transducer at that position and angle, and displays this calculated ultrasound scanning area—the intracardiac ultrasound image—visually through a semi-transparent fan-shaped plane.

[0211] like Figure 11 The diagram shows the modules of the guidance information generation system. The system comprises a physical layer, a simulation layer, and an algorithm layer. The physical layer includes a catheter, a data acquisition module, and an interactive terminal. The interactive terminal can provide feedback on the target guidance information in multiple modalities, including text, voice, vibration, and images. The simulation layer includes a three-dimensional simulated heart model. The user operates the catheter handle, and the data acquisition module transmits the catheter's state changes to the three-dimensional simulated heart model. The three-dimensional simulated heart model performs a sector simulation and outputs the target intracardiac ultrasound image.

[0212] The algorithm layer includes a sector analysis model and a question-answering model. The sector analysis model analyzes the target intracardiac ultrasound image and outputs the target sector analysis results. The question-answering model analyzes and provides feedback on the target sector analysis results based on user dialogue commands, outputting target guidance information, which can be visualized on the interactive terminal.

[0213] In this embodiment, by integrating a data acquisition module, tactile feedback in real operation can be simulated, improving the actual operation perception of the object being operated on and ensuring the accurate acquisition of the target intracardiac ultrasound image.

[0214] Example 4

[0215] Figure 12 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the method for generating guidance information as described in any of the above embodiments. Figure 12 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0216] like Figure 12 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).

[0217] Bus 93 includes a data bus, an address bus, and a control bus.

[0218] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0219] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, such program module 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0220] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92, such as the method for generating guidance information provided in any of the above embodiments.

[0221] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 96. As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0222] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0223] Example 5

[0224] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating guidance information provided in any of the above embodiments.

[0225] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0226] Example 6

[0227] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method for generating guidance information as described in any of the preceding embodiments.

[0228] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0229] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A method for generating guidance information, characterized in that, The generation method includes: Based on a pre-built three-dimensional simulated heart model, a target intracardiac ultrasound image of the corresponding simulated heart is obtained; The target intracardiac ultrasound image is input into a pre-constructed sector analysis model to output the target sector analysis results; The target sector analysis results are input into a pre-trained question-and-answer model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart.

2. The method for generating guidance information as described in claim 1, characterized in that, The step of inputting the target sector analysis results into the question-answering model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart includes: Acquire the image information of the operation object, wherein the operation object is the object corresponding to the target intracardiac ultrasound image for which intracardiac ultrasound operation is performed on the simulated heart; The target sector analysis results and the image information are input into the question-and-answer model to generate the target guidance information that matches the intracardiac ultrasound operation of the simulated heart; And / or, After the step of inputting the target sector analysis results into the question-answering model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart, the method further includes: Based on the target guidance information, several types of feedback information are generated; The types include at least one of text type, voice type, vibration type and image type.

3. The method for generating guidance information as described in claim 1, characterized in that, The training steps of the question-answering model include: Based on the pre-constructed three-dimensional simulated heart model, several sample intracardiac ultrasound images of the corresponding simulated heart are obtained. The intracardiac ultrasound images of the sample are input into the pre-constructed sector analysis model to output the sample sector analysis results; Obtain sample association information that matches the sample sector analysis results, wherein the sample association information is information associated with the operation object performing intracardiac ultrasound on the simulated heart; The results of the analysis of several sets of sample sectors and the matching sample association information are input into a preset large language model to train a question-and-answer model for guiding intracardiac ultrasound operations.

4. The method for generating guidance information as described in claim 3, characterized in that, The sample association information includes sample guidance information that matches the intracardiac ultrasound operation and sample operation status information of the intracardiac ultrasound operation. The sample operation status information includes at least one of the following: operation stage information of the intracardiac ultrasound operation, error type information of the intracardiac ultrasound operation, image information of the operation object, historical operation information of the operation object, and instruction information of the operation object.

5. The method for generating guidance information as described in claim 4, characterized in that, The step of inputting several sets of sample sector analysis results and the matched correlation information into a preset large language model to train a question-answering model for guiding intracardiac ultrasound operations includes: Based on the sample sector analysis results and the sample operation status information, prompt words are constructed; The prompt words and the sample guidance information are input into the preset large language model to train the question-answering model.

6. The method for generating guidance information as described in any one of claims 3-5, characterized in that, After the step of inputting several sets of sample sector analysis results and the matched sample association information into a preset large language model to train a question-and-answer model for guiding intracardiac ultrasound operations, the method further includes: Acquire operational knowledge of intracardiac ultrasound procedures from several experts; The operational knowledge includes at least one of the following: expert guidance records for intracardiac ultrasound operation, expert operational thinking for intracardiac ultrasound operation, and expert teaching and training content for intracardiac ultrasound operation. Based on the operational knowledge, the question-answering model is adjusted to obtain the adjusted question-answering model.

7. A device for generating guidance information, characterized in that, The generating apparatus includes: The target image acquisition module is used to obtain the target intracardiac ultrasound image of the corresponding simulated heart based on a pre-built three-dimensional simulated heart model. The target result acquisition module is used to input the target intracardiac ultrasound image into a pre-constructed sector analysis model to output the target sector analysis result; The guidance information generation module is used to input the target sector analysis results into a pre-trained question-and-answer model to generate target guidance information that matches the intracardiac ultrasound operation of the simulated heart.

8. A system for generating guidance information, characterized in that, The generation system includes the generation device as described in claim 7, as well as a catheter, a handle for operating the catheter, and a data acquisition module; The handle is used to move the catheter within the receiving space of the data acquisition module based on external operation, so as to change the position information of the catheter. The data acquisition module is used to acquire the pose information of the catheter and output it to the generation device to change the display state of the catheter in the three-dimensional simulated heart model. The display state is used to characterize the pose information of the catheter. The generating device is also used to output a target intracardiac ultrasound image corresponding to the simulated heart based on the display state.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating guidance information as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for generating guidance information as described in any one of claims 1-6.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for generating guidance information as described in any one of claims 1-6.

Images