Digestive endoscope virtual simulation training system and digestive endoscope operation evaluation method

Abstract

The invention belongs to the technical field of medical simulation. According to the digestive endoscope virtual simulation training system and the digestive endoscope operation evaluation method, a background control terminal carries a visual interaction platform and a user assessment system, an attitude sensor is arranged in a box body, a handle is used for simulating key operation of an endoscope, and the visual interaction platform is connected with the user assessment system. A virtual digestive tract modeling scene is configured in the visual interaction platform; the handle and the attitude sensor are both in communication connection with the control terminal; the control terminal is in communication connection with the background control terminal; the visual interaction platform restores the operation behavior track of the endoscope in the virtual digestive tract modeling scene according to the received posture data of the endoscope and in combination with the virtual digestive tract modeling scene; an endoscope video is collected through a camera on a handle, and a user assessment system performs operation evaluation on the digestive endoscope according to an operation behavior track, the endoscope video and a diagnosis report of a user; and the accurate restoration of the endoscope operation process is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical simulation, in particular to a virtual simulation training system for a digestive endoscope and an evaluation method for operation of the digestive endoscope. BACKGROUND

[0002] The statements in this section merely provide background technology related to the present application and do not necessarily constitute prior art.

[0003] In the modern medical system, endoscopy and treatment as a key technology have been widely and deeply applied to many medical fields and play an irreplaceable important role in the diagnosis and treatment of diseases. From the aspect of disease diagnosis, an endoscope can directly observe the surface morphology, color, texture and other details of an organ by virtue of its long tube and precise lens at the front end, which can enter the body through natural cavities or small incisions. Taking the digestive system as an example, a gastroscope can clearly present the lesions of esophageal, gastric and duodenal mucosa, accurately find early gastric cancer, esophageal cancer and various inflammations, ulcers and other diseases; a colonoscope can comprehensively examine the large intestine and timely find intestinal polyps, intestinal cancer and other lesions. Compared with traditional open surgery, endoscopic treatment has significant advantages such as small trauma, quick recovery and fewer complications, which greatly reduces the pain and medical burden of patients. However, the operation of an endoscope is extremely complex, which requires the operator not only to master the basic operation skills of the endoscope such as advancing, retreating, rotating and bending, but also to have precise hand-eye coordination ability to make correct judgments and operations according to the images under the lens; at the same time, the operation methods and precautions are different for different parts and different types of diseases, which puts high requirements on the skills and experience of the operator.

[0004] The traditional endoscope training mode mainly relies on operation practice on real patients, but this mode has many disadvantages: on the one hand, the training cost is high, each operation on a patient needs to use disposable consumables, disinfection equipment and professional medical team for monitoring, which increases the operating cost of medical institutions and the economic burden of patients; on the other hand, the risk is huge, and the students may cause serious complications such as misinsertion of organs and massive hemorrhage due to lack of experience in the operation process, which causes irreparable harm to the patients; in addition, the limited nature of patient resources also seriously restricts the training opportunities of students, not all patients are willing to be training objects, and the patient's condition and physical condition are also different, which is difficult to meet the systematic and comprehensive training needs of students, which makes it difficult to meet the needs of current medical education for efficient and safe training. SUMMARY

[0005] In order to solve the problems of insufficient practice opportunities, high operation risk and lack of effective feedback in the traditional training mode, the present application provides a kind of digestive endoscope virtual simulation training system and digestive endoscope operation evaluation method, which adopts the mode of software and hardware combination to build highly simulated training environment for students;The hardware part simulates the real endoscope operation process, and provides realistic physical feedback experience for students by reproducing insertion, steering and adjusting operations;The software part collects operation data (including mirror position and angle parameters) in real time, and presents intuitively through visual interface, and fully restores the operation process of endoscope.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a kind of digestive endoscope virtual simulation training system.

[0007] A kind of digestive endoscope virtual simulation training system, comprising: Control terminal, handle, attitude sensor, box and background control terminal, background control terminal is loaded with visual interactive platform and user examination system, attitude sensor is arranged in the box, handle is used to simulate the key operation of endoscope, visual interactive platform is configured with virtual digestive tract modeling scene; Handle and attitude sensor are in communication connection with control terminal, and control terminal is in communication connection with background control terminal; Visual interactive platform restores the operation behavior trajectory of endoscope in virtual digestive tract modeling scene according to the received attitude data of endoscope, in combination with virtual digestive tract modeling scene; Endoscope video acquisition is carried out through the camera on handle, and user examination system carries out operation evaluation of digestive endoscope according to operation behavior trajectory, endoscope video and user's diagnosis report.

[0008] In an implementation manner of the first aspect of the present application, the key operation includes: lens direction control, water spray, air blowing, photographing, air suction, marking and force feedback.

[0009] In an implementation manner of the first aspect of the present application, further comprising a pedal in communication connection with the control terminal, and the pedal is stepped down to realize the set operation action.

[0010] In an implementation manner of the first aspect of the present application, further comprising an X-ray module in communication connection with the control terminal, and the X-ray module is used to guide X-ray image when mirror operation is carried out.

[0011] In an implementation manner of the first aspect of the present application, visual interactive platform restores the operation behavior trajectory of endoscope in virtual digestive tract modeling scene, then generates global navigation view, and dynamically displays real-time depth and attitude of endoscope and insertion tube.

[0012] In an implementation form of the first aspect of the present application, the visual interaction platform simulates the field of view of the endoscope front camera by adjusting the angle of the handle, dynamically presents the details of the digestive tract wall and the lesion characteristics, and displays the camera-captured image in the central area of the visual interface in real time, so that the operator can observe the details of different parts of the digestive tract through the endoscope pushing and 360-degree rotation of the lens to obtain the lesion condition.

[0013] In an implementation form of the first aspect of the present application, the visual interaction platform integrates a pain value simulation unit to provide real-time feedback of virtual pain perception.

[0014] In an implementation form of the first aspect of the present application, the background control terminal is also equipped with an editor for setting the position, size and shape of the lesion in the virtual digestive tract modeling scene.

[0015] In a second aspect, the present application provides a method for evaluating the operation of a digestive endoscope.

[0016] A method for evaluating the operation of a digestive endoscope, which uses the virtual simulation training system for a digestive endoscope according to the first aspect of the present application, includes the following processes: Obtaining the operation behavior trajectory, the endoscope video and the user's diagnosis report; Encoding the operation behavior trajectory using an operation encoder, encoding the endoscope video using a video encoder, and encoding the diagnosis report using a text encoder; Performing unified dimension processing and splicing on the operation behavior trajectory encoding result, the endoscope video encoding result and the diagnosis report encoding result to obtain the spliced features; Performing multi-head self-attention processing on the spliced features, and processing the result of the multi-head self-attention processing through a sparse-gated hybrid expert mechanism to obtain multi-modal fusion features; Obtaining the final operation evaluation result according to the multi-modal fusion features and a pre-trained decoder.

[0017] In an implementation form of the second aspect of the present application, the multi-head self-attention processing is performed on the spliced features, and the result of the multi-head self-attention processing is processed through a sparse-gated hybrid expert mechanism to obtain multi-modal fusion features, including: , , , , , wherein, represents the spliced features, is the feature representation after multi-head self-attention processing, a routing coefficient of an input multi-head self-attention processed feature representation relative to each expert model, a routing coefficient after taking the maximum K values, and FFN represents a feedforward neural network, an output of the i-th expert model, an output of the i-th expert model, a multi-modal fusion feature.

[0018] In an implementation form of the second aspect of the present application, the total loss function is: wherein, , , is a prediction result at position i, is an encoding result of a true value at position i, is a total text length, is a total load number of the i-th expert model, is a total number of expert models. Compared with the prior art, the present application has the following beneficial effects: The present application innovatively proposes a digestive endoscope virtual simulation training system and a digestive endoscope operation evaluation method, solves the problems of insufficient practice opportunities, high operation risk and lack of effective feedback in the traditional training mode, adopts a combination of software and hardware to build a highly simulated training environment for trainees; the hardware part simulates the real endoscope operation process, provides a realistic physical feedback experience for trainees by reproducing operations such as insertion, turning and adjusting; the software part collects operation data (including parameters such as mirror insertion position and angle) in real time and presents them intuitively through a visual interface, fully restoring the operation process of the endoscope.

[0019] Through efficient cooperation of software and hardware, the present application creates a safe, low-cost and repeatable operation training environment, enabling trainees to master the key technologies and process details of endoscope operation under risk-free conditions, thereby improving operation proficiency and the ability to cope with actual clinical scenarios; the present application overcomes the shortcomings of the traditional training mode and provides an intelligent and precise training tool for medical education, significantly improving the training efficiency and quality of endoscope operation skills.

[0020] The present application creates a safe, low-cost and repeatable operation training environment, enabling trainees to master the key technologies and process details of endoscope operation under risk-free conditions, thereby improving operation proficiency and the ability to cope with actual clinical scenarios; the present application overcomes the shortcomings of the traditional training mode and provides an intelligent and precise training tool for medical education, significantly improving the training efficiency and quality of endoscope operation skills.

[0021] ​​​The application can dynamically reproduce the operation process of the endoscope, including key links such as entering the mirror, rotating, and lesion examination, by accurate data collection of hardware and real-time simulation of software, and provide students with an intuitive and real operation training environment, the hardware simulates the real operation behavior of the endoscope and provides physical feedback, and the software generates a virtual scene based on real-time data, presents the endoscope front end view and the digestive tract structure, so that students are familiar with the technical points in actual operation, effectively solve the problems of high risk, high cost and insufficient opportunities in traditional training, and significantly improve the skill level of students through repeatable and low-risk simulation operation.

[0022] The application supports the creation and editing of cases, provides diversified training scenes, and can simulate the condition changes of real patients, and provides a flexible and efficient solution for medical teaching and in-service training of hospitals.

[0023] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings accompanying the specification of the application form a part of the application and serve to provide further understanding of the application, the illustrative embodiments of the application and their description serve to explain the application without constituting an improper limitation thereof.

[0025] Figure 1 The hardware framework schematic diagram of the digestive endoscope virtual simulation training system provided for an exemplary embodiment of the application is shown in the figure; Figure 2 The software framework schematic diagram of the digestive endoscope virtual simulation training system provided for an exemplary embodiment of the application is shown in the figure; Figure 3 The interface schematic diagram of the visual interactive platform provided for an exemplary embodiment of the application is shown in the figure; Figure 4 The schematic diagram of the endoscope operation evaluation model embedded in the user examination system provided for an exemplary embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0026] The application will be further described below in combination with the drawings and embodiments.

[0027] It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0028] The present embodiment proposes a virtual simulation training system for digestive endoscopy, which comprises a hardware part and a software part, wherein the hardware part is used for collecting user operation data and simulating real endoscopy operation experience, and the software part is responsible for processing the collected data, visual presentation, and personalized evaluation of user operation in combination with an endoscopy operation evaluation model.

[0029] More specifically, the hardware part of the present embodiment comprises a control terminal, a handle, a posture sensor, a box, a pedal, an X-ray module and a background control terminal, as shown in the figure, to realize precise simulation and data collection of endoscopy operation. Figure 1

[0030] In the present embodiment, the control terminal is the core of the hardware system, responsible for collecting data from various functional modules and communicating with the background control terminal through a USB interface to realize efficient transmission of user operation information and ensure real-time synchronization to the virtual digestive tract modeling scene in the visual interactive platform of the software part.

[0031] In the present embodiment, the handle is used to simulate key operations of the endoscope, including lens direction control, water spraying, air blowing, photographing, air suction, marking, force feedback and other functions. The handle is built-in with a signal collection and transmission module, which can capture user operation instructions in real time and upload them to the virtual digestive tract modeling scene in the visual interactive platform through the control terminal, ensuring accurate presentation of each operation in the virtual environment.

[0032] In the present embodiment, the box is integrated with a high-precision posture sensor to record the dynamic position and angle of the insertion tube, ensuring that the system can accurately reproduce the endoscopy trajectory and provide a real mirror insertion simulation experience for students.

[0033] In the present embodiment, the pedal serves as an auxiliary control device, allowing users to trigger specific functions through their feet while operating with their hands, further improving the flexibility and smoothness of the operation. In addition, the hardware part of the present embodiment also designs an X-ray module to simulate the image assistance function in endoscopy examination, helping students better understand the collaborative application of endoscopy operation and image guidance technology.

[0034] The software part of the present invention designs a complete functional module system, including a login system, a local data management system, a user assessment system, an editor and a visual interactive platform, all of which are installed on the background control terminal, as shown in the figure. Figure 2

[0035] ​​ProjectMgr total manager (media class singleton), which is the core management module of the software, exists in singleton mode, ensures that there is only one instance in the system, facilitates global management and coordination of other modules, receives event-driven signals from the input controller (i.e. input control), and processes corresponding events according to the event type. This module also interacts with multiple sub-modules, including: UI user interface, responsible for interacting with users, displaying various information and operation interfaces of the system; case editor, used to edit and manage case-related data; data storage, responsible for storing various data of the system, such as case data, user operation records, etc.; training evaluation, evaluating and feeding back user operations to help users improve their skills.

[0036] GameSceneMgr scene manager (singleton), also in singleton mode, is responsible for managing related matters of game or simulation scenes, interacts with ProjectMgr to receive its instructions to manage scene switching, initialization, etc. It has jurisdiction over multiple functional modules: level management, responsible for managing different levels, including level loading, switching, and state saving; simulation, performing various simulation operations in cooperation with organ simulation modules to simulate human organs and other objects; mirror body control, controlling the mirror body in the simulation scene (such as the simulation of medical equipment such as endoscopes), which may cooperate with the internal and external cameras and auxiliary instrument modules to achieve more realistic simulation effects; internal and external cameras, simulating the perspective and functions of internal and external cameras to provide visual presentation for the scene; auxiliary instruments, managing various auxiliary instruments in the simulation scene, such as surgical tools.

[0037] In this implementation, the login system is the entrance for users to enter the system, responsible for identity verification and permission management. The login system provides three login methods for users: administrators can enter the editor with the highest permission account to edit lesion data, and view student operation records and case diagnosis reports through the local data management system; ordinary users can enter the visualization interaction platform after logging in to select the case scene preset by the tutor for operation practice; external users can experience system functions as visitors and can enter the visualization interaction platform to complete basic operations but have no right to save data.

[0038] In this implementation, the visualization interaction platform is the core module of the system. After the student selects a specific patient case, the system simulates the operation environment of the endoscope through the visualization interaction platform, combines high-precision digestive tract modeling technology with data collected by hardware, and dynamically reproduces the movement process of the endoscope and the picture of the front camera. The visualization interaction platform realizes multi-aspect simulation of endoscope operation, such as Figure 3 As shown, it includes the following functions: (1) The visual interaction platform relies on the attitude sensor in the box to collect the attitude data of the endoscope in real time, and accurately restores the process of the endoscope in the digestive tract through the virtual digestive tract modeling scene, such as pushing and rotating. The visual interaction platform generates a global navigation view to dynamically display the real-time depth and attitude of the endoscope and the insertion tube, enabling students to clearly and intuitively perceive the spatial changes of the mirror insertion process.

[0039] (2) The visual interaction platform simulates the field of view of the front camera of the endoscope by adjusting the angle of the handle, dynamically presenting the details of the digestive tract wall and the characteristics of the lesion. The central area of the interface of the visual interaction platform displays the picture captured by the camera in real time. Students can observe the details of different parts of the digestive tract and understand the lesion by pushing the endoscope and rotating the lens 360 degrees.

[0040] (3) The visual interaction platform also supports interactive operations with the digestive tract, such as biopsy, water spraying, and air blowing, which are controlled by the handle. These operations are dynamically updated by the system according to the position and direction of the handle, such as controlling the direction and coverage of water spraying, or selecting specific parts for biopsy, simulating the real clinical operation process.

[0041] (4) Based on the interactive experience, the visual interaction platform also integrates a pain value simulation function, which dynamically feedbacks the patient's possible pain perception through a progress bar. The color of the progress bar gradually transitions from green to red to prompt students whether the operation is too intense, helping them adjust the intensity, standardize the operation behavior, and improve the safety of the training.

[0042] In this implementation, the editor is a case creation and editing tool designed specifically for administrators, aiming to provide flexible lesion simulation functions. The system is built-in with a variety of lesion materials, including common clinical lesions such as ulcers, polyps, and erosions, and supports the expansion of more lesion types according to actual clinical needs. Administrators can set the location, size, shape, and other key parameters of the lesion through the editor to highly simulate the patient's condition. The editor supports three-dimensional perspective switching function, and administrators can accurately position and place lesions in the three-dimensional model of the digestive system through drag-and-drop method, and make necessary adjustments. After completion, the case data will be stored in the data management system for the visual interaction platform to call, thereby providing students with diversified and clinically relevant training scenarios.

[0043] In this implementation, the local data management system stores the following core data to ensure complete records of the operation process and traceability for subsequent analysis. First, the user's account information, including the account name, registration time, and recent login record, can be edited or deleted by the administrator through the management interface. Second, the patient's medical record edited by the administrator, covering the location information, size, shape, and related description of the lesion, provides students with diverse operation scenarios. In addition, the system records the student's operation process and generated case diagnosis report, including operation steps, time consumption, screenshots, and final diagnosis results, to support subsequent analysis and review. Finally, the system stores the evaluation results of the student's operation by the administrator or tutor, including the score, improvement suggestions, and comprehensive assessment, providing clear learning feedback and growth direction for students.

[0044] In this implementation, the user assessment system retrieves the operation records, case diagnosis reports, and other related data from the local data management system, uses a pre-trained endoscopic operation evaluation model to comprehensively assess the student's operation, and generates an operation evaluation report. The evaluation report not only provides detailed learning feedback for students but also provides important reference for tutors. The operation evaluation report generated by the system can help tutors quickly judge the student's skill mastery and learning progress, and combined with the evaluation report and actual teaching goals, the student's learning achievements are confirmed and scored, ensuring the flexibility and professionalism of the evaluation process.

[0045] Based on the above-mentioned digestive endoscopy virtual simulation training system, this implementation proposes a digestive endoscopy operation evaluation method. First, the endoscopic operation evaluation model needs to be trained, including: Collect multiple groups of data to construct a user operation dataset , which is composed of the following: (1); (2); Among them, is the operation behavior trajectory, is the endoscopic motion trajectory, including multiple frames of three-dimensional position coordinates and Euler angles, is the operation event sequence, which is obtained by multi-hot encoding of specific operation events (water spraying, photographing, using instruments, etc.), and if there is no event in the current frame, it is defaulted to all 0, is the pain value sequence, containing multiple frames of pain values, is the total number of frames recorded during the endoscopic operation process, is the total feature dimension after splicing, represents the dimension of the operation event sequence; is the video collected in real time during the endoscopic operation process, and are the height and width of each frame of image, respectively; is the diagnosis report, which is the diagnosis report made by the user according to the endoscopy operation, in the form of a string; is the operation evaluation report, which is obtained by an expert according to the endoscopy operation process and the diagnosis report of the user, and is used to evaluate the accuracy and deficiency of the operation process, and is also in the form of a string.

[0046] The structure of the endoscopy operation evaluation model is shown in Figure 4 , which adopts the structure of encoding-fusion-decoding, wherein the encoder is divided into three parts, including an operation encoder, a video encoder and a text encoder, which respectively encode the operation behavior trajectory, the endoscopy video and the diagnosis report.

[0047] For the operation encoder, the operation behavior trajectory is used as the input, which is first mapped to a unified hidden dimension by a linear layer, then a trainable position encoding is added to retain the time sequence information, and finally input into a multi-layer Transformer Encoder to extract the global time sequence features of the operation behavior, and the output of the operation encoder is represented as operation behavior encoding .

[0048] For the video encoder, the endoscopy video sequence is used as the input, each frame of image is first extracted by a CNN network to obtain the image-level feature representation, and then combined into a frame sequence feature according to the time sequence. The sequence is then input into a multi-layer Transformer Encoder to capture the cross-frame spatio-temporal features, and the output of the video encoder is the video feature encoding .

[0049] For the text encoder, the diagnosis report text is used as the input, the text is first converted into a token ID sequence by a word segmentation encoder, and then input into a pre-trained language model BERT to obtain the semantic representation of the whole text, which is used to represent the global features of the diagnosis information, and the output of the text encoder is the diagnosis text encoding .

[0050] In the multi-modal fusion module of the system, the operation behavior encoding, the video feature encoding and the diagnosis text encoding are first subjected to unified dimension processing, and the three are spliced along the feature dimension to form a fusion input sequence; then, the fusion sequence passes through a layer of multi-head self-attention mechanism (MSHA) for modeling the dependency relationship and interaction structure between different modalities, so as to enhance the context connection between modalities; on this basis, a sparse gated mixture of experts mechanism (Mixture of Experts, MoE) is introduced to further model the attention output. Specifically, the system constructs a plurality of feed-forward neural networks (FFN) with the same structure but independent parameters as expert sub-networks, and dynamically allocates expert weights according to the input features of each time step through a learnable router module, so that only part of the experts are activated to participate in the calculation. Finally, the outputs of each expert model are weighted and fused according to the router weights to obtain a multi-modal fusion representation with semantic integration capability. The fusion strategy has good expression ability and expansibility, and can adapt to different modal combinations and time sequence length changes, providing strong feature support for the downstream operation evaluation task. The specific formula is as follows: (3); (4); (5); (6); (7); (8); wherein, represents the spliced feature, is the feature representation processed by the multi-head attention mechanism, is the routing coefficient of the input relative to each expert model, represents the routing coefficient after taking the maximum of values, represents the feed-forward neural network, represents the output of the expert model, represents the multi-modal fusion feature.

[0051] In the above scheme, the operation behavior encoding, video encoding and text encoding are uniformly processed in the same dimension and spliced to form a fusion input sequence. This step provides a basic data structure for subsequent multi-modal fusion, so that data of different modalities can be processed in the same framework, breaking down the barriers between modalities and laying the foundation for fully utilizing multi-modal information. By processing the fusion input sequence using MSHA, the dependency relationships and interaction structures between different modalities can be modeled. There is a close relationship between operation behavior, video and text description. MSHA can capture these relationships, such as the association between operation steps and corresponding actions in the video, and the correspondence between the description of the operation in the text and the actual behavior, thereby enhancing the contextual connection between modalities and enabling the model to better understand the overall semantics of multi-modal data. MSHA can focus on data from multiple subspaces to extract more rich feature information, which helps the model to better understand the data and improve the representation ability of complex scenes, providing more discriminative features for subsequent fusion and downstream tasks. The MoE mechanism is introduced to construct multiple feed-forward neural networks (FFN) with the same structure but independent parameters as expert subnetworks, and through a learnable router module, the expert weights are dynamically allocated according to the input features, and only a part of the experts are activated for calculation. This dynamic allocation method can flexibly select the most suitable expert for processing according to different input features, improving the processing efficiency and relevance of the model.

[0052] The MoE mechanism enables the model to adapt to different combinations of modalities and changes in time length. When facing diverse operation evaluation tasks, whether it is a simple operation or a complex process, the model can effectively handle different situations by reasonably allocating expert resources, providing strong feature support for downstream tasks and enhancing the applicability and scalability of the model. The outputs of each expert model are weighted and fused according to the router weights to obtain a multi-modal fusion representation with semantic integration capability. This fusion representation integrates the advantages of different experts and can more comprehensively and accurately express the semantic information of multi-modal data. For example, in operation evaluation, the fusion representation can cover the accuracy of operation behavior, the details of video and the key points of text description, providing a rich feature basis for generating accurate operation evaluation reports.

[0053] Overall, this multi-modal fusion scheme effectively integrates information from different modalities through the synergistic effect of multiple technologies, improving the model's understanding and processing ability of multi-modal data and providing high-quality feature support for downstream operation evaluation tasks, which helps to generate more accurate and comprehensive operation evaluation results.

[0054] The present implementation uses an autoregressive text generation decoder based on a Transformer architecture to generate an operation evaluation report from the fused multi-modal features The operation evaluation report is generated, the decoder is composed of multiple decoding layers stacked, each layer includes a masked self-attention mechanism, a cross-attention mechanism and a feedforward network structure, and can dynamically focus on the fusion features of the input when generating each word, thereby realizing conditional text generation. In the training stage, the system adopts a teacher forcing strategy, segments the operation evaluation text, constructs an input sequence and a target sequence, the input sequence is used as the initial token of the decoder, and the target sequence is used as a supervision signal to guide the model to predict the subsequent content word by word. The model output is the word probability distribution at each position, which is optimized by cross-entropy loss. The input of the decoder is the last position The generated token sequence , the output is the predicted token sequence at the current position , and the complete operation evaluation text (composed of the predicted token sequence at each time) is finally generated: (9); (10).

[0055] The present embodiment utilizes an autoregressive text generation decoder based on the Transformer architecture to generate operation evaluation reports from fused multi-modal features, and has significant technical effects. The masked self-attention mechanism in the multi-layer structure ensures that the generated words only depend on historical information, maintaining the autoregressive property. The cross-attention mechanism enables the generation process to dynamically focus on the fused features, enabling conditional text generation and making the report content closely related to the multi-modal input. During training, the teacher forcing strategy is used to accelerate model convergence, and the cross-entropy loss is optimized to enable the model to accurately predict words one by one, ultimately generating a logically coherent and content-relevant operation evaluation text that effectively improves the accuracy and comprehensiveness of operation evaluation.

[0056] In the present embodiment, the trained endoscope operation evaluation model is used to generate the final operation evaluation result with multi-modal fusion features as input.

[0057] The training goal of the system is to generate accurate operation evaluation text, so the main loss function is the cross-entropy loss between the predicted text sequence and the true text sequence. In addition, to ensure balanced use of experts in the MoE mechanism, a MoE balance loss is further introduced as a regularization term.

[0058] In the text generation part, the decoder outputs the word prediction distribution at each step and matches it with the one-hot encoding of the true label to calculate the standard sequence-level cross-entropy loss, which can effectively guide the model to learn to extract key semantics from the multi-modal fusion features and generate evaluation text that conforms to the actual operation performance: (11); wherein, is the prediction result of the model at position is the one-hot encoding result of the true value at position is the total text length. Specifically, in the text generation process, the decoder outputs the word prediction distribution at each step and matches it with the one-hot encoding of the true label to calculate the sequence-level cross-entropy loss, which can measure the difference between the model's prediction and the true text. Through backpropagation, the model adjusts its parameters to extract key semantic information from multi-modal fusion features. For example, when generating an operation evaluation report, the model can more accurately capture key descriptive words related to actual operations, making the generated evaluation text more consistent with actual operation performance. By continuously optimizing the cross-entropy loss, the model can learn a prediction pattern that better conforms to the distribution of the true text, thereby improving the accuracy of the generated text. This allows the model to more accurately generate descriptions of surgical steps, operation highlights, and other aspects, reducing the deviation between the generated text and the true evaluation.

[0059] Meanwhile, the sparse routing mechanism is adopted in the MoE fusion module. To avoid the problem of uneven utilization of experts caused by the activation of only a small number of experts during model training, a balance loss is introduced to constrain the activation probability distribution of different experts and encourage the model to be more stable and diverse in expert selection:

[0060] (12); wherein, is the total load of the th expert model, is the total number of expert models.

[0061] ​​More specifically, the MoE (Mixture of Experts) fusion module adopts a sparse routing mechanism. During model training, there may be a case where only a small number of experts are activated, resulting in uneven utilization of experts. After introducing a balance loss, it can constrain the activation probability distribution of different experts, encouraging the model to be more stable and diverse in expert model selection. The balance loss can encourage the model to more reasonably call different expert models, avoiding the situation where some experts are overused while other experts are not fully utilized. By balancing the activation probabilities of different expert models, the model can more stably select appropriate expert model combinations when processing various inputs, thereby improving the model's adaptability to different types of inputs and enhancing the model's generalization performance.

[0062] The final total loss function is represented as: (13).

[0063] The total loss function of the present implementation is composed of a cross-entropy loss and a MoE balance loss. It considers both the accuracy of text generation and the balance of expert selection. By optimizing the loss of both aspects at the same time, the model can generate accurate operation evaluation text while reasonably utilizing expert knowledge, improving overall performance, ensuring the accuracy of the evaluation text, and fully utilizing the knowledge of different medical experts to make the generated evaluation more professional and reliable. The optimization of the above total loss function helps the system to better converge during training, improves the quality and stability of the model, and thus improves the overall effect of the system in generating operation evaluation text, providing better service to users.

[0064] In summary, the present application can dynamically reproduce the operation process of the endoscope, including key links such as entering the mirror, rotating, and checking the lesion, by accurate data collection of the hardware part and real-time simulation of the software part, providing students with an intuitive and realistic operation training environment. The hardware simulates the real operation behavior of the endoscope and provides physical feedback, while the software part generates a virtual scene based on real-time data, presenting the endoscope front-end view and the digestive tract structure, allowing students to familiarize themselves with the technical points in actual operation, effectively solving the problems of high risk, high cost, and insufficient opportunities in traditional training, and significantly improving the students' skill level through repeatable, low-risk simulation operations; In addition, the system supports case creation and editing, provides diverse training scenarios, and can simulate the changes in the patient's condition, providing a flexible and efficient solution for medical teaching and in-service training in hospitals.

[0065] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various changes and modifications to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A virtual simulation training system for digestive endoscopy, characterized in that, include: The system includes a control terminal, a handle, an attitude sensor, a housing, and a back-end control terminal. The back-end control terminal is equipped with a visual interaction platform and a user assessment system. The attitude sensor is located inside the housing. The handle is used to simulate key operations of the endoscope. The visual interaction platform is equipped with a virtual digestive tract modeling scene. Both the handle and the attitude sensor are connected to the control terminal, which in turn is connected to the backend control terminal. The visualization and interaction platform reconstructs the operational trajectory of the endoscope in the virtual digestive tract modeling scene based on the received endoscope posture data and the virtual digestive tract modeling scene. Endoscopic video is captured by a camera on the handle, and the user assessment system evaluates the operation of the digestive endoscopy based on the operation behavior trajectory, endoscopic video, and the user's diagnostic report.

2. The virtual simulation training system for digestive endoscopy as described in claim 1, characterized in that, Key operations include: lens orientation control, water spraying, air blowing, taking pictures, air inhalation, marking, and force feedback.

3. The virtual simulation training system for digestive endoscopy as described in claim 1, characterized in that, It also includes a pedal that communicates with the control terminal, and when the pedal is pressed, it performs a set operation. It also includes an X-ray module that communicates with the control terminal, which is used to guide X-ray images during endoscope insertion.

4. The virtual simulation training system for digestive endoscopy as described in any one of claims 1-3, characterized in that, The visualization and interactive platform recreates the operational trajectory of the endoscope in a virtual digestive tract modeling scenario, and then generates a global navigation view to dynamically display the real-time depth and orientation of the endoscope and insertion tube.

5. The virtual simulation training system for digestive endoscopy as described in any one of claims 1-3, characterized in that, The visualization and interactive platform, combined with the angle adjustment of the handle, simulates the field of view of the endoscope's front-end camera, dynamically presenting the details and lesion characteristics of the digestive tract wall. The central area of ​​the visualization interface displays the image captured by the camera in real time, allowing operators to observe the details of different parts of the digestive tract to obtain information about lesions by advancing the endoscope and rotating the lens 360 degrees.

6. The virtual simulation training system for digestive endoscopy as described in any one of claims 1-3, characterized in that, The visual interactive platform integrates a pain value simulation unit, providing real-time feedback on virtual pain sensations.

7. The virtual simulation training system for digestive endoscopy as described in any one of claims 1-3, characterized in that, The back-end control terminal is also equipped with an editor, which allows users to set the location, size, and shape of lesions in the virtual digestive tract modeling scene.

8. A method for evaluating digestive endoscopy procedures, characterized in that, The virtual simulation training system for digestive endoscopy according to any one of claims 1-7 includes the following process: Acquire operational behavior trajectories, endoscopic videos, and user diagnostic reports; An operation encoder is used to encode the operation behavior trajectory, a video encoder is used to encode the endoscopic video, and a text encoder is used to encode the diagnostic report; The results of the operation behavior trajectory encoding, the endoscopic video encoding, and the diagnostic report encoding are processed in a unified dimension and then concatenated to obtain the concatenated features. Multi-head self-attention processing is applied to the spliced ​​features, and the results of multi-head self-attention processing are processed through a sparse gating hybrid expert mechanism to obtain multimodal fusion features; Based on the multimodal fusion features and the pre-trained decoder, the final operation evaluation result is obtained.

9. The method for evaluating digestive endoscopy procedures as described in claim 8, characterized in that, The concatenated features are subjected to multi-head self-attention processing. The results of the multi-head self-attention processing are then processed through a sparse-gated hybrid expert mechanism to obtain multimodal fused features, including: , , , , , ,in, Indicates the features after splicing. The feature representation after multi-head self-attention processing. The feature representation after multi-head self-attention processing of the input is relative to the routing coefficients of each expert model. This means taking the largest value. The routing coefficients after the given values, where FFN represents the feedforward neural network. Indicates the first The output of an expert model This represents the multimodal fusion feature.

10. The method for evaluating digestive endoscopy procedures as described in claim 8 or 9, characterized in that, The total loss function is: ; in, , , For in position The prediction results at the location, For the true value at position The encoding result at that location, This represents the total text length. For the first Total load of expert models This represents the total number of expert models.

Images