Gastrointestinal endoscope training system based on multi-dimensional data fusion

Abstract

The invention relates to the technical field of medical instruments, and discloses a multidimensional data fusion gastrointestinal endoscope training system, which comprises a data acquisition module, a multidimensional fusion processing module, a virtual simulation module, a tactile feedback module, an interaction terminal module and an edge computing architecture, the data acquisition module is used for acquiring multi-modal medical image data, physiological index data and operation track data; and the multi-dimensional fusion processing module is connected with the data acquisition module, the virtual simulation module and the tactile feedback module, and comprises a time sequence fusion unit, a space mapping unit and a generative adversarial network unit. According to the gastrointestinal endoscope training system, the data acquisition module, the multi-dimensional fusion processing module, the virtual simulation module, the tactile feedback module, the interaction terminal module and the edge computing architecture are integrated into a unified system, so that the mode that most of traditional gastrointestinal endoscope training systems adopt single-modal data or module independent operation is improved; therefore, the problem of single operation evaluation dimension is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical devices, in particular to a multi-dimensional data fusion gastrointestinal endoscope training system. BACKGROUND

[0002] Gastrointestinal endoscopy is a medical optical instrument for the diagnosis and treatment of digestive system diseases, which is composed of an elongated flexible tubular structure, a front-end camera and a light source, and can directly observe the mucosa of esophagus, stomach, intestinal tract, etc. Among them, gastroscopy enters the upper digestive tract through the mouth or nose, and can check the esophagus, stomach and duodenum; colonoscopy is inserted through the anus to observe the colon and rectum. It not only can find lesions such as ulcers, polyps and tumors, but also can obtain tissue samples through biopsy forceps to determine the pathological diagnosis, and has treatment functions such as polyp removal, hemostasis and stent placement. Modern gastrointestinal endoscopy technology is constantly developing, and painless endoscopy has been widely used, with the characteristics of intuitive precision, small trauma and high safety, and is an important means for the diagnosis and treatment of digestive tract diseases.

[0003] Traditional gastrointestinal endoscope training systems mostly use single modal data or module independent operation mode, which lacks data fusion and cross-module collaboration, resulting in insufficient training scene authenticity and single operation evaluation dimension. SUMMARY

[0004] In order to make up for the above shortcomings, the present application provides a multi-dimensional data fusion gastrointestinal endoscope training system, which aims to improve the problem that traditional gastrointestinal endoscope training systems mostly use single modal data or module independent operation mode, which lacks data fusion and cross-module collaboration, resulting in insufficient training scene authenticity and single operation evaluation dimension.

[0005] In the first aspect, the present application provides the following technical scheme, a multi-dimensional data fusion gastrointestinal endoscope training system, comprising a data acquisition module, a multi-dimensional fusion processing module, a virtual simulation module, a tactile feedback module, an interactive terminal module and an edge computing architecture;

[0006] The data acquisition module is used to acquire multi-modal medical image data, physiological index data and operation trajectory data.

[0007] The multi-dimensional fusion processing module is connected to the data acquisition module, the virtual simulation module and the tactile feedback module respectively, and comprises a time sequence fusion unit, a spatial mapping unit and a generative adversarial network unit.

[0008] The virtual simulation module is connected to the multi-dimensional fusion processing module and comprises a scene generation submodule and a real-time rendering submodule.

[0009] The tactile feedback module is connected to the multi-dimensional fusion processing module and comprises a six-dimensional force feedback handle and a segmented pneumatic simulation physiological digestive tract model.

[0010] The interaction terminal module is connected with the virtual simulation module and the tactile feedback module, and includes a VR helmet integrated with an eye movement tracking device, a tactile glove, and a pressure sensing foot pedal.

[0011] The edge computing architecture is connected with the multi-dimensional fusion processing module, the tactile feedback module, and the interaction terminal module, and is used for realizing low-delay processing and remote collaboration of data.

[0012] By adopting the above technical solutions, the data acquisition module, the multi-dimensional fusion processing module, the virtual simulation module, the tactile feedback module, the interaction terminal module, and the edge computing architecture are integrated into a unified system, and then the whole-process closed loop of multi-modal data acquisition, fusion processing, virtual simulation, tactile feedback, and remote collaboration is realized, so that the problem of insufficient authenticity of a training scene and single dimension of operation evaluation caused by lack of data fusion and cross-module collaboration in a traditional gastrointestinal endoscope training system, in which a single modal data or module is independently operated, is improved.

[0013] Preferably, the generative adversarial network unit adopts a WGAN-GP architecture, and includes:

[0014] A generator network G includes five residual blocks, each of which is composed of a convolutional layer, a batch normalization layer, and an RLU activation layer, and is used for mapping a random noise vector z into virtual lesion data G(z);

[0015] A discriminator network D includes four convolutional layers, each of which has a step size of 2 and a convolution kernel size of 4*4, and is used for judging whether input data is a real sample x or a generated sample G(z); and a loss function of the WGAN-GP is:

[0016]

[0017] wherein L(G, D) is a loss function, G is a generator, D is a discriminator, x is a real medical image sample, and z is a random noise vector, is a linear interpolation sample of the real sample and the generated sample, is a gradient vector of the discriminator to the interpolation sample, is an L2 norm of the gradient vector, and p data (x) is a real data distribution, p z (z) is a noise distribution, is an interpolation distribution.

[0018] Preferably, the multi-dimensional fusion processing module further includes a decision tree analysis model, and the decision tree analysis model adopts a random forest algorithm, and includes: a forest composed of 50 decision trees, a maximum depth of each decision tree is 8, and a minimum sample split number of a node is 2.

[0019] The splitting attribute of each decision node is determined by calculating the information gain ratio, and the formula for calculating the information gain ratio is:

[0020]

[0021] Where GainRatio(D,a) is the information gain ratio, D is the training dataset, a is the attribute to be split, Gain(D,a) is the information gain of attribute a with respect to dataset D, and IV(a) is the intrinsic value of attribute a: the intrinsic value IV(a) is calculated using the following formula:

[0022] Where V is the number of values ​​for attribute a, and D v Let D be a subset of samples where attribute a takes the value v.

[0023] Preferably, the force feedback algorithm of the six-dimensional force feedback handle is as follows:

[0024]

[0025] in, Let K be the feedback force vector at time t. d Here is the damping coefficient matrix. K is the velocity vector. p This is the stiffness coefficient matrix. Let K be the displacement vector. i This is the integral coefficient matrix;

[0026] The damping coefficient matrix K d It is a diagonal matrix, and the range of values ​​for the diagonal elements is:

[0027] [0.5 N·s / m, 8 N·s / m], dynamically adjusted according to the elastic modulus of the virtual tissue.

[0028] Preferably, the edge computing architecture adopts a fog computing layered model, including:

[0029] The device layer includes the data acquisition module, haptic feedback module, and interactive terminal module, which are connected via a USB 3.2 interface;

[0030] The edge layer includes local edge servers, equipped with ARM architecture processors, and running the QNX real-time operating system;

[0031] The cloud layer includes a cluster of cloud-based servers and uses the Hadoop distributed computing framework.

[0032] The edge layer and the cloud layer are isolated by bandwidth through 5G network slicing technology, and real-time haptic feedback data uses dedicated network slices.

[0033] Preferably, the segmented pneumatic physiologically mimetic digestive tract model comprises:

[0034] An esophagus segment made of thermoplastic elastomer material with a shape memory alloy wire inside;

[0035] A stomach segment made of silicone material with a 16x16 array of micro-pneumatic sensors inside, with a sampling frequency of 100 Hz;

[0036] An intestine segment divided into ascending colon, transverse colon and descending colon segments, each containing three independent pneumatic mimetic muscle units controlled by PWM signals with a PWM frequency of 20 kHz;

[0037] The segments are connected by magnetic quick-release interfaces with O-shaped waterproof sealing rings at the interfaces.

[0038] Preferably, the data acquisition module comprises: a medical image acquisition unit using a mutual information-based registration algorithm with the formula MI(A,B) = H(A) + H(B) - H(A,B); where MI(A,B) is mutual information, A and B are medical images of different modalities, H(A) and H(B) are their respective entropies, and H(A,B) is the joint entropy; a physiological indicator acquisition unit containing a single-lead ECG sensor, a piezoresistive blood pressure sensor and an optical fiber respiratory motion sensor simulating the human body model; an operation trajectory acquisition unit using an optical tracking system and a nine-axis inertial measurement unit to fuse positioning data through an extended Kalman filter algorithm.

[0039] In a second aspect, the present application provides the following technical solution, a training method of a multi-dimensional data fusion gastrointestinal endoscopy training system, taking a multi-dimensional fusion processing module as the execution subject, comprising the following steps:

[0040] S1, acquiring multi-modal medical image data, physiological indicator data and operation trajectory data output by the data acquisition module;

[0041] S2, time series aligning the physiological indicator data and the operation trajectory data to generate time series fusion data;

[0042] S3, three-dimensional space mapping the medical image data to generate space mapping data;

[0043] S4, controlling a generative adversarial network unit to process the space mapping data to synthesize virtual lesions containing heterogeneous pathological features;

[0044] S5, controlling a decision tree analysis model to analyze the time series fusion data to generate operation evaluation data;

[0045] S6, trigger the scene generation sub-module of the virtual simulation module, construct a virtual digestive tract scene based on the fused medical image data and virtual lesions;

[0046] S7, trigger the real-time rendering sub-module of the virtual simulation module, use a GPU acceleration unit and a gaze point rendering technology to perform real-time rendering on the scene;

[0047] S8, send operation evaluation data to the six-dimensional force feedback handle and the segmented pneumatic simulation physical digestive tract model, and trigger corresponding force feedback and digestive tract dynamic tension simulation;

[0048] S9, receive eye movement tracking data and operation instructions of the interactive terminal module, and adjust the rendering precision and interaction logic of the virtual digestive tract scene.

[0049] In a third aspect, the present application provides the following technical solution: a computer device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to realize the training method of the multi-dimensional data fusion gastrointestinal endoscopy training system described above.

[0050] In a fourth aspect, the present application provides the following technical solution: a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to realize the training method of the multi-dimensional data fusion gastrointestinal endoscopy training system described above.

[0051] The present application has the following beneficial effects:

[0052] 1、In the present application, the data acquisition module, the multi-dimensional fusion processing module, the virtual simulation module, the tactile feedback module, the interactive terminal module and the edge computing architecture are integrated into a unified system, thereby realizing the whole-process closed loop of multi-modal data acquisition, fusion processing, virtual simulation, tactile feedback and remote collaboration, and thereby improving the problem that the traditional gastrointestinal endoscopy training system mostly adopts a single modal data or module independent operation mode, and due to the lack of data fusion and cross-module collaboration, the training scene reality is insufficient and the operation evaluation dimension is single.

[0053] 2、In the present application, the WGAN-GP architecture is adopted in the generative adversarial network unit to synthesize virtual lesions containing heterogeneous pathological features, thereby breaking through the limitation of traditional simulation relying on real case data, and thereby improving the problem that the traditional endoscopy training system is mostly limited by the scarcity of medical image data, and due to the lack of real rare lesion samples, the training scene diversity is poor and students are difficult to access complex pathology.

[0054] 3、The six-dimensional force feedback handle force feedback algorithm in the application is combined with the segmented pneumatic simulation biological physical digestive tract model, and then the physical characteristics of the organization in the virtual simulation are converted into the six-dimensional force feedback and the dynamic tension of the digestive tract that can be perceived, so that the defects that most of the traditional virtual training systems lack physical interaction are improved, and the problems of insufficient hand-eye coordination training and force control skill training effect caused by the operation feeling depending on the preset fixed parameters are solved. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A system architecture diagram of a multi-dimensional data fusion gastrointestinal endoscope training system is provided for the application;

[0056] Figure 2 A training method step schematic diagram of a multi-dimensional data fusion gastrointestinal endoscope training system is provided for the application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the application will be described clearly and completely in conjunction with the drawings. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0058] Embodiment one

[0059] Reference Figure 1 In the first embodiment of the application, the application provides a multi-dimensional data fusion gastrointestinal endoscope training system, which comprises a data acquisition module, a multi-dimensional fusion processing module, a virtual simulation module, a tactile feedback module, an interactive terminal module and an edge computing architecture.

[0060] The data acquisition module is used to acquire multi-modal medical image data, physiological index data and operation trajectory data.

[0061] The multi-dimensional fusion processing module is connected with the data acquisition module, the virtual simulation module and the tactile feedback module respectively, and comprises a time sequence fusion unit, a space mapping unit and a generative adversarial network unit.

[0062] The virtual simulation module is connected with the multi-dimensional fusion processing module, and comprises a scene generation sub-module and a real-time rendering sub-module.

[0063] The tactile feedback module is connected with the multi-dimensional fusion processing module, and comprises a six-dimensional force feedback handle and a segmented pneumatic simulation biological physical digestive tract model.

[0064] The interactive terminal module is connected with the virtual simulation module and the tactile feedback module, and comprises a VR helmet integrated with an eye tracking device, a tactile glove and a pressure sensing foot pedal.

[0065] The edge computing architecture connects a multi-dimensional fusion processing module, a haptic feedback module, and an interactive terminal module to achieve low-latency data processing and remote collaboration.

[0066] Specifically, the data acquisition module can acquire multimodal data such as CT, MRI, and endoscopic white light images to construct high-precision virtual digestive tract scenes and lesion models, restoring real anatomical structures and pathological features; it can also collect data such as heart rate, blood pressure, and respiratory rate from simulated human models to achieve linkage simulation of operational stimuli and physiological responses, enhancing the clinical realism of the training scenario; and it can record operational parameters such as endoscope insertion depth, lens rotation angle, and instrument operation time, providing a data foundation for the operation evaluation model and supporting quantitative analysis of trainees' operational standardization and decision-making logic; the multidimensional fusion processing module connects the data acquisition module, virtual simulation module, and haptic feedback module, integrating temporal fusion units, spatial mapping units, and generative adversarial networks. The network unit can align physiological indicator data and operation trajectory data in time series, eliminate time deviations in multi-source data, generate time-series fused data, and provide a unified benchmark in the time dimension for operation evaluation; it maps multimodal medical imaging data to a three-dimensional virtual space, completing spatial registration and reconstruction of image data, and providing a high-precision three-dimensional anatomical structure model for the virtual simulation module; based on medical imaging data, it synthesizes virtual lesion data through a generator network, and combines it with a discriminator network to optimize the realism of the lesion model, realizing dynamic simulation of pathological features such as polyps and ulcers; it outputs fused image and lesion data to the virtual simulation module to drive the construction of virtual scenes; and it transmits operation evaluation data to the haptic feedback module, triggering force feedback and gastrointestinal tension. Force simulation enables cross-module linkage of "data fusion - scene generation - physical feedback"; the virtual simulation module constructs a virtual digestive tract scene containing the dynamic disease evolution process (such as the malignant transformation stage of polyps) based on the temporal fusion data and spatial mapping data output by the multi-dimensional fusion processing module, restoring the anatomical structure and pathological feature distribution in the real clinical scene; adopting foveated rendering technology, the scene rendering accuracy is dynamically adjusted according to the user's gaze focus, while ensuring high-resolution display of key areas (such as the center of the lens's field of view) and optimizing the allocation of system computing resources to achieve a smooth rendering effect of more than 60fps; it receives virtual disease data from the multi-dimensional fusion processing module and drives the dynamic update of the lesion morphology and position in the scene; and outputs data to the interactive terminal module. It outputs real-time scene data, supports immersive display and operation command response of VR headsets, and forms a simulation closed loop of "data-driven - scene generation - real-time interaction"; the six-dimensional force feedback handle in the haptic feedback module generates six-dimensional force feedback (three-dimensional translational force and three-dimensional rotational force) in real time through magnetorheological fluid damper based on the operation evaluation data output by the multi-dimensional fusion processing module, simulates the resistance and vibration feedback of operations such as contacting mucosa and grasping tissue in endoscopy, and improves the realism of the operation feel; the pneumatic bionic muscle unit in the segmented pneumatic bionic physical digestive tract model: controls the contraction and relaxation of pneumatic muscles in each intestinal segment through air pressure regulating solenoid valves, simulates the peristalsis and tension changes of the digestive tract, such as the elastic resistance of the stomach wall and the pushing resistance at the bend of the intestine;Pressure sensor array: Real-time acquisition of pressure data from various parts of the model during operation, feeding it back to the multi-dimensional fusion processing module to form a closed loop of "operational force input - physical feedback - data feedback," enhancing the immersive force perception during training; Haptic feedback module synchronizes the mechanical feedback of the physical model with the visual scene of the virtual simulation module, allowing the operator to perceive real operational force through the controller and model while observing the virtual digestive tract scene, achieving multi-modal collaborative feedback of vision and force perception, and strengthening the development of hand-eye coordination during training; VR headset (including eye tracking) receives the three-dimensional scene data output by the virtual simulation module, presenting the virtual digestive tract environment from an immersive perspective, supporting 6DOF (six degrees of freedom) head tracking, realizing the operator's perspective. Real-time synchronization; eye-tracking device captures the focal point of the gaze, triggering the real-time rendering submodule to adjust the image precision of the gaze area (e.g., magnifying lesion details), optimizing human-computer interaction efficiency; tactile gloves receive force feedback signals from the tactile feedback module, simulating the tactile perception of instruments contacting tissues during endoscopic operations (e.g., friction, sudden changes in resistance) through built-in shape memory alloy wires or vibration motors; integrated pressure sensors collect finger grip force data in real time, transmitting it to the multi-dimensional fusion processing module for operational force assessment; pressure-sensing foot pedals provide auxiliary function control during two-handed endoscopic operation via a three-position function switch (e.g., inhalation / exhalation, air / water injection, instrument switching), conforming to clinical operating habits; pressure sensors detect in real time... The force applied by the foot pedal is converted into an electrical signal that drives the virtual simulation module to update the scene state (such as the degree of expansion of the virtual digestive tract when adjusting the amount of gas injected); the interactive terminal module can collaborate across modules, synchronizing the virtual scene (VR headset), force feedback (haptic gloves), and pedal control signals to form an integrated interactive link of "visual guidance - hand operation - foot pedal assistance"; it records and stores operation data such as eye movement trajectory, hand movements, and foot pedal frequency, providing multi-dimensional behavioral feature parameters for training effect evaluation; the edge computing architecture connects the multi-dimensional fusion processing module, haptic feedback module, and interactive terminal module to achieve low-latency data processing and remote collaboration; remote collaborative training can be synchronized across multiple devices: supporting multiple training devices through edge layer aggregation. The system enables virtual scene synchronization, such as real-time mapping of teacher operations to student VR headsets for remote teaching; cloud model updates: the edge layer periodically pulls the latest medical image datasets and AI evaluation models (such as lesion recognition algorithms) from the cloud, and local updates improve the pathological diversity and evaluation accuracy of the training scenario; cross-regional collaboration: through the hybrid architecture of the edge layer and the cloud layer, real-time interaction between students and instructors in different locations is achieved (such as remote guidance on polyp removal path planning), breaking through physical space limitations; the edge computing architecture can achieve data security and energy efficiency optimization, and sensitive data localization: physiological indicator data and operation trajectory data are anonymized at the edge layer, and only anonymized statistical data is transmitted to the cloud, reducing the risk of privacy leakage;Computational resource scheduling: Rendering tasks (such as non-focal region compression rendering) are dynamically allocated to local GPUs and cloud servers through an edge-layer load balancing algorithm, reducing overall system power consumption by more than 30%. By integrating the data acquisition module, multi-dimensional fusion processing module, virtual simulation module, haptic feedback module, interactive terminal module, and edge computing architecture into a unified system, a closed-loop process of multimodal data acquisition, fusion processing, virtual simulation, haptic feedback, and remote collaboration is achieved. This improves upon the traditional gastrointestinal endoscopy training systems, which mostly adopt a single-modal data or independent module operation mode. Due to the lack of data fusion and cross-module collaboration, these systems suffer from insufficient realism in training scenarios and a single dimension of operational evaluation.

[0067] The generative adversarial network unit adopts the WGAN-GP architecture, including:

[0068] The generator network G contains 5 residual blocks, each consisting of a convolutional layer, a batch normalization layer, and an RLU activation layer, used to map the random noise vector Z into virtual lesion data G(z);

[0069] The discriminator network D contains four convolutional layers, each with a stride of 2 and a kernel size of 4×4, used to determine whether the input data is a real sample x or a generated sample G(z); the loss function of WGAN-GP is:

[0070]

[0071] Where L(G,D) is the loss function, G is the generator, D is the discriminator, x is the real medical image sample, and z is the random noise vector. The sample is a linear interpolation of the real sample and the generated sample. This is the gradient vector of the discriminator for the interpolated samples. Let p be the L2 norm of the gradient vector. data (x) represents the true data distribution, p z (z) represents the noise distribution. This is an interpolated distribution.

[0072] Specifically, the generator network G extracts deep features (such as mucosal texture and lesion morphology gradient) from real medical image samples (x) through 5 residual blocks (including convolutional layers, batch normalization layers, and ReLU activation layers), mapping the random noise vector (z) into virtual lesion data (G(z)) containing heterogeneous pathological features, covering multiple types of lesion morphologies such as polyps, ulcers, and early cancer; the discriminator network D extracts discriminative features from the input data layer by layer through 4 convolutional layers with a stride of 2 and a kernel size of 4×4, distinguishing between real samples (x) and generated samples (G(z)), forcing the generator to improve the realism of lesion synthesis; the gradient penalty term ( [...]) By constraining the discriminator gradient norm to be close to 1, mode collapse is avoided, ensuring the diversity of generated lesions (such as polyp models of different sizes, colors, and depths of invasion); based on the distribution of multimodal real data such as CT and endoscopic images (p data (x)), the generator can synthesize pathological samples rarely seen in traditional medical imaging (such as early submucosal lesions), solving the problem of insufficient training data; interpolation samples By linearly mixing real and generated data, the dynamic evolution of lesions (such as inflammation → adenoma → carcinogenesis) is simulated, providing a virtual scene of continuous pathological stages for the training system. The generated virtual lesion data (G(z)) is converted into a three-dimensional model by a spatial mapping unit and can be accurately embedded into the virtual digestive tract scene, supporting the simulation of skills such as lesion recognition and biopsy path planning in endoscopic operation training. Its realism is evaluated by a discriminator (D(x) approaches 1, D(G(z)) approaches 0), which is close to the clinical imaging standard.

[0073] The multi-dimensional fusion processing module also includes a decision tree analysis model, which uses the random forest algorithm and consists of a forest of 50 decision trees, each with a maximum depth of 8 and a minimum number of splits per node of 2.

[0074] The splitting attribute of each decision node is determined by calculating the information gain ratio. The formula for calculating the information gain ratio is:

[0075]

[0076] Where GainRatio(D,a) is the information gain ratio, D is the training dataset, a is the attribute to be split, Gain(D,a) is the information gain of attribute a with respect to dataset D, and IV(a) is the intrinsic value of attribute a: the intrinsic value IV(a) is calculated using the following formula:

[0077] Where V is the number of values ​​for attribute a, and D v Let D be a subset of samples where attribute a takes the value v.

[0078] Specifically, the random forest decision tree analysis model in the multidimensional fusion processing module integrates 50 decision trees with a maximum depth of 8 and a minimum number of splits per node. Based on the information gain ratio algorithm, it can perform multidimensional feature segmentation on the physiological index data (such as heart rate and respiratory rate) and operation trajectory data (such as endoscope movement speed and instrument operation time) output by the time-series fusion unit. By calculating the information gain (Gain(D,a)) and intrinsic value (IV(a)) of the attributes, it identifies key parameters that significantly affect the standardization of operation (such as the correlation between lens turning angle and mucosal injury risk), and generates quantitative operation evaluation data (such as operation fluency score and decision rationality index). Utilizing the nonlinear fitting capability of random forest, it mines complex correlations between multi-source data (such as the relationship between operational force and different lesion morphologies). The mapping relationship between tissue damage probability and the evaluation model overcomes the overfitting problem of a single decision tree and improves the generalization ability of the evaluation model. Through the voting mechanism of 50 decision trees, real-time operation data is analyzed in parallel, outputting operation deviation warnings (such as the polyp removal path deviating from the optimal trajectory), and the evaluation results are fed back to the virtual simulation module to dynamically adjust the scene difficulty (such as increasing the probability of complex lesions) or tactile feedback parameters (such as enhancing the resistance feedback when operation errors occur), realizing closed-loop optimization of "data acquisition - evaluation - feedback". The hierarchical structure of a single decision tree (maximum depth 8) can intuitively display the operation evaluation logic chain (such as "if the endoscope insertion speed is >5mm / s and the lens rotation angle is >60°, then the operation risk level is increased"), providing trainees with visual operation improvement guidance, which meets the interpretability requirements of medical training.

[0079] The force feedback algorithm for the six-dimensional force feedback handle is as follows:

[0080]

[0081] in, Let K be the feedback force vector at time t. d Here is the damping coefficient matrix. K is the velocity vector. p This is the stiffness coefficient matrix. Let K be the displacement vector. i This is the integral coefficient matrix;

[0082] Damping coefficient matrix K d It is a diagonal matrix, and the range of values ​​for the diagonal elements is:

[0083] [0.5 N·s / m, 8 N·s / m]; dynamically adjusted according to the elastic modulus of the virtual tissue.

[0084] Specifically, through damping force Based on the operating velocity vector The damping coefficient matrix K generates resistance in the opposite direction of motion (such as the viscous resistance when an endoscope is advanced through a bend in the digestive tract). d As a diagonal matrix, its diagonal elements are dynamically adjusted according to the elastic modulus of the virtual tissue (0.5-8 N·s / m) to simulate the frictional characteristics of different tissues (such as mucosa and muscle layer); elastic force Based on the displacement vector The stiffness coefficient matrix K generates elastic restoring force (such as tissue tension when biopsy forceps grasp a polyp). p Corresponding to different lesion hardness (such as the difference in elasticity between polyps and normal mucosa);

[0085] Inertial force: By generating cumulative inertial effects (such as inertial drag during rapid retraction of the endoscope) through integral displacement vectors, the realism of the tactile feedback is enhanced; K is adjusted in real time based on tissue properties output by the virtual simulation module (such as ulcer base stiffness and tumor invasion depth). d Matrix parameters enable the handle feedback force to dynamically match lesion characteristics (such as the high damping resistance of invasive cancer tissue), solving the problem that traditional fixed-parameter feedback cannot simulate tissue heterogeneity; by setting the force feedback threshold (e.g., when...), Vibration warnings are triggered when the safety threshold is exceeded, prompting operators to avoid excessive force that could cause mucosal damage. This helps establish a "force-displacement" operational intuition and reduces the risk of clinical operations. The algorithm independently calculates three-dimensional translational force (along the x / y / z axis) and three-dimensional rotational force (around the x / y / z axis), achieving precise decoupling of the handle's movement direction and force feedback (e.g., only torque feedback is generated when rotating the lens, and only translational force is generated when pushing axially), improving the spatial positioning accuracy of the operating feel.

[0086] The edge computing architecture adopts a layered fog computing model, including:

[0087] The device layer includes a data acquisition module, a haptic feedback module, and an interactive terminal module, which are connected via a USB 3.2 interface;

[0088] The edge layer includes local edge servers, equipped with ARM architecture processors, and running the QNX real-time operating system;

[0089] The cloud layer includes a cluster of cloud-based servers and uses the Hadoop distributed computing framework.

[0090] Bandwidth isolation between the edge layer and the cloud layer is achieved through 5G network slicing technology, and real-time haptic feedback data uses dedicated network slices.

[0091] Specifically, at the device layer: eye-tracking data from the interactive terminal module, haptic glove operation signals, and pressure sensor data from the haptic feedback module are collected in real time via a USB 3.2 interface. This data is pre-processed locally before being transmitted to the edge layer, reducing data transmission latency in the cloud. At the edge layer: an ARM architecture processor and QNX real-time operating system are used to perform real-time filtering and feature extraction (such as endoscopic motion speed and force feedback threshold calculation) on operation trajectory data and physiological indicator data, with a processing latency of ≤10ms, meeting the real-time requirements of haptic feedback. At the cloud layer: historical training data is stored and updated using the Hadoop distributed framework. The virtual lesion model parameters are isolated from the cloud layer via 5G network slicing, ensuring that the transmission latency of real-time data (such as force feedback signals) is ≤50ms. Bandwidth isolation between the edge layer and the cloud layer is achieved through 5G network slicing technology. Furthermore, dedicated network slices are used for real-time haptic feedback data (such as force signals from a six-dimensional force feedback handle and pressure data from a segmented pneumatic model), ensuring that the transmission latency of this type of data is ≤50ms and the packet loss rate is ≤0.1%, meeting the real-time requirements of haptic feedback and avoiding force feedback delays or interruptions due to network congestion, thus maintaining training stability. Immersive experience; Distinguishing between real-time services (haptic feedback, VR scene rendering) and non-real-time services (training data storage, model updates) network resources, dedicated slices guarantee bandwidth priority for real-time data through QoS (Quality of Service) mechanisms (e.g., allocating ≥100Mbps dedicated bandwidth) to prevent cloud-based batch data downloads and other operations from preempting real-time data transmission channels; Real-time haptic feedback data involves sensitive information such as operational details and physiological indicators, dedicated slices use network layer isolation technologies (e.g., VPN tunnels, traffic identification filtering) to prohibit unauthorized devices from accessing, reducing the risk of data leakage and complying with medical data security standards; Supporting remote training scenarios (e.g., remote expert guidance), real-time haptic data is directly transmitted to the edge layer cluster through dedicated slices, combined with the localized processing capabilities of edge nodes, reducing cross-regional transmission latency (e.g., latency ≤100ms between Beijing and Shanghai nodes), improving remote operation collaboration efficiency; The hard isolation characteristics of network slices (e.g., independent wireless resources, core network nodes) can avoid interference from sudden traffic surges in other services (e.g., sudden bandwidth increases in cloud AI model training) on ​​the training system, ensuring consistent service quality during multi-user concurrent training, supporting large-scale training scenarios with ≥200 devices online simultaneously.

[0092] The segmented pneumatic biomimetic digestive tract model includes:

[0093] The esophageal segment is made of thermoplastic elastomer material with built-in shape memory alloy wires;

[0094] The gastric segment is made of silicone material and has a built-in 16×16 array of micro-pressure sensors with a sampling frequency of 100Hz;

[0095] The intestinal segment is divided into the ascending colon, transverse colon and descending colon. Each segment contains 3 independent pneumatic bionic muscle units, which are controlled by PWM signals with a frequency of 20kHz.

[0096] The sections are connected by magnetic quick-release interfaces, and O-rings are installed at the interfaces for waterproof sealing.

[0097] Specifically, in the esophagus segment: thermoplastic elastomer (TPE) material simulates the flexibility of the esophageal mucosa, and the built-in shape memory alloy wire can be heated by electric current to restore a preset bending shape, simulating swallowing or esophageal peristalsis; in the stomach segment: silicone material replicates the elasticity of the stomach wall, and a 16×16 array of micro-pressure sensors (100Hz sampling frequency) monitors the distribution of operating forces in real time, such as local pressure changes when the endoscope contacts the gastric mucosa, and the data is fed back to the multi-dimensional fusion processing module; in the intestinal segment: the ascending / transverse / descending colon segments realize peristaltic waveform simulation through independent pneumatic bionic muscle units (3 per segment, 20kHz PWM signal control), such as the periodic contraction and relaxation of the haustra, enhancing the realism of the operating resistance; the pneumatic muscle units generate axial contraction force (0.5-5N) and radial tension (0.1-1.5N / cm) through air pressure regulation (0-100kPa). 2 This system simulates the physiological tension of different parts of the digestive tract (such as the peristaltic differences between the rectal ampulla and the sigmoid colon); a magnetic quick-release interface combined with an O-ring seal enables rapid replacement of different intestinal segments (disassembly time ≤ 10 seconds), supports modular assembly of different pathological models (such as esophageal cancer and colitis), and improves the flexibility of training scenarios; a gastric segment pressure sensor array generates operational force heat maps in real time, locating weak operational areas (such as the pressure concentration point during gastric angle biopsy), providing force data support for operational evaluation; and shape memory alloy wires and pneumatic muscle motion parameters (…). The bending angle and contraction frequency can be programmed to reproduce gastrointestinal deformations under specific pathological conditions (such as intestinal stenosis caused by tumor compression); the modular design is compatible with standard endoscopic instruments (such as gastroscopes / colonoscopes with a diameter of 5-12mm), the magnetic interface ensures airtightness (leakage rate ≤0.01L / min), and supports simulation of air / water injection functions; by replacing different pathological modules (such as gastric segments with built-in silicone polyps), various surgical procedures such as polyp removal and stenosis dilation can be trained, meeting the full-process training needs from basic operations to complex surgeries.

[0098] The data acquisition module includes: a medical image acquisition unit, which uses a registration algorithm based on mutual information, calculated as: MI(A,B)=H(A)+H(B)-H(A,B); where MI(A,B) is the mutual information, A and B are medical images of different modalities, H(A) and H(B) are their respective entropies, and H(A,B) is the joint entropy; a physiological indicator acquisition unit, which includes a single-lead ECG sensor simulating a human body model, a piezoresistive blood pressure sensor, and a fiber optic respiratory motion sensor; and an operation trajectory acquisition unit, which uses an optical tracking system and a nine-axis inertial measurement unit to fuse positioning data through an extended Kalman filter algorithm.

[0099] Specifically, based on the mutual information (MI) registration algorithm, the anatomical structure images are aligned with real-time endoscopic images by calculating the entropy (H(A), H(B)) and joint entropy (H(A,B)) of different modal images (such as CT and endoscopic white light images). The registration error is ≤0.5mm, ensuring the spatial consistency between the lesion location and the anatomical structure in the virtual scene, providing a high-precision image foundation for virtual simulation. A single-lead ECG sensor (accuracy ±0.05mV), a piezoresistive blood pressure sensor (error ±2mmHg), and a fiber optic respiratory sensor (resolution ±0.1mm chest diameter change) synchronously collect heart rate, blood pressure, and respiratory rate data of the simulated human body at a sampling frequency of 100Hz. This data is used to drive the dynamic display of physiological parameters in the virtual scene (such as increased heart rate caused by operational stimuli), enhancing the training realism. The system employs a real-time optical tracking system (accuracy ≤ 0.1 mm) and a nine-axis IMU (angular velocity error ≤ 0.5° / s) to capture the endoscope position, lens angle, and instrument movement trajectory. Multi-source data is fused using an extended Kalman filter algorithm to output a smooth six-degree-of-freedom motion trajectory (position error ≤ 0.3 mm, angle error ≤ 1°), providing accurate motion sequence data for the operational evaluation model (e.g., instrument path during polyp biopsy). Each unit achieves synchronized data acquisition through a hardware triggering mechanism (time deviation ≤ 10 ms), ensuring consistency between the timestamps of medical images, physiological indicators, and operational trajectories. This provides spatiotemporally aligned multi-source data for subsequent temporal fusion units, supporting correlation analysis of "operation-physiological response-image changes" (e.g., the correlation between blood pressure fluctuations and mucosal damage during rapid endoscope insertion).

[0100] Example 2:

[0101] Reference Figure 2 In a second embodiment of the present invention, a training method for a gastrointestinal endoscopy training system based on multidimensional data fusion is provided, wherein a multidimensional fusion processing module serves as the execution entity, and the method includes the following steps:

[0102] S1. Acquire multimodal medical imaging data, physiological index data, and operation trajectory data output by the data acquisition module;

[0103] S2. Align the physiological indicator data and the operation trajectory data in time series to generate time series fusion data;

[0104] S3. Perform three-dimensional spatial mapping on medical image data to generate spatial mapping data;

[0105] S4. Control the generative adversarial network units to process the spatial mapping data and synthesize virtual lesions containing heterogeneous pathological features;

[0106] S5. The control decision tree analysis model analyzes time-series fusion data to generate operational evaluation data.

[0107] S6, Trigger the scene generation submodule of the virtual simulation module to construct a virtual digestive tract scene based on the fused medical image data and virtual lesions;

[0108] S7. Trigger the real-time rendering submodule of the virtual simulation module to render the scene in real time using GPU acceleration unit and foveated rendering technology.

[0109] S8. Send operation evaluation data to the six-dimensional force feedback handle and the segmented pneumatic biomimetic digestive tract model to trigger stress feedback and dynamic tension simulation of the digestive tract.

[0110] S9. Receive eye-tracking data and operation instructions from the interactive terminal module, and adjust the rendering accuracy and interaction logic of the virtual digestive tract scene.

[0111] Specifically, S1 acquires multimodal medical images, physiological indicators, and operational trajectory data to provide raw information for the system; S2 eliminates the time discrepancy between physiological and operational data through time series alignment, generating time-series fusion data; S3 maps medical images to three-dimensional space, laying the spatiotemporal foundation for subsequent processing; S4 uses generative adversarial network units to synthesize virtual lesions containing heterogeneous pathological features, enriching the training scenario; S5 analyzes the time-series fusion data through a decision tree analysis model to generate quantitative operational evaluation data for judging operational compliance; S6 constructs a virtual digestive tract scene based on fused images and virtual lesions; S7 utilizes GPU acceleration and foveated rendering technology to achieve real-time and efficient scene rendering, ensuring the smoothness and realism of visual presentation; S8 sends the operational evaluation data to hardware devices, triggering a six-dimensional force feedback handle and a segmented pneumatic model to generate stress feedback and digestive tract tension simulation; S9 receives eye tracking and operational commands, dynamically adjusts scene rendering accuracy and interaction logic, realizes real-time human-computer interaction, and forms a complete training process of "data acquisition—processing and analysis—scene presentation—feedback interaction," improving the realism and effectiveness of endoscopic operation training.

[0112] Example 3

[0113] In the third embodiment of the present invention, based on the same inventive concept, the present invention proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a training method for a multidimensional data fusion gastrointestinal endoscopy training system as described in the above embodiments.

[0114] Example 4

[0115] In the fourth embodiment of the present invention, based on the same inventive concept, a computer device is proposed, comprising: a processor and a memory; the processor and the memory communicate with each other; the memory is used to store instructions; the processor is used to execute the instructions in the memory to execute the training method of the multidimensional data fusion gastrointestinal endoscopy training system of the above embodiment.

[0116] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0117] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-dimensional data fusion gastrointestinal endoscopy training system, comprising a data acquisition module, a multi-dimensional fusion processing module, a virtual simulation module, a haptic feedback module, an interactive terminal module, and an edge computing architecture, characterized in that... ； The data acquisition module is used to acquire multimodal medical image data, physiological indicator data, and operation trajectory data; The multi-dimensional fusion processing module is connected to the data acquisition module, the virtual simulation module and the haptic feedback module respectively, and includes a temporal fusion unit, a spatial mapping unit and a generative adversarial network unit. The virtual simulation module is connected to the multi-dimensional fusion processing module and includes a scene generation submodule and a real-time rendering submodule. The tactile feedback module is connected to the multi-dimensional fusion processing module and includes a six-dimensional force feedback handle and a segmented pneumatic biomimetic digestive tract model. The interactive terminal module connects the virtual simulation module and the haptic feedback module, and includes a VR helmet with an integrated eye-tracking device, haptic gloves, and a pressure-sensing foot pedal. The edge computing architecture connects the multi-dimensional fusion processing module, the haptic feedback module, and the interactive terminal module to achieve low-latency data processing and remote collaboration.

2. The gastrointestinal endoscopy training system with multidimensional data fusion according to claim 1, characterized in that: The generative adversarial network unit adopts the WGAN-GP architecture and includes: The generator network G contains 5 residual blocks, each consisting of a convolutional layer, a batch normalization layer, and an RLU activation layer, used to map a random noise vector z to virtual lesion data G(z); The discriminator network D contains four convolutional layers, each with a stride of 2 and a kernel size of 4×4, used to determine whether the input data is a real sample x or a generated sample G(z); the loss function of the WGAN-GP is: Where L(G,D) is the loss function, G is the generator, D is the discriminator, x is the real medical image sample, and z is the random noise vector. The sample is a linear interpolation of the real sample and the generated sample. This is the gradient vector of the discriminator for the interpolated samples. Let p be the L2 norm of the gradient vector. data (x) represents the true data distribution, p z (z) represents the noise distribution. This is an interpolated distribution.

3. The gastrointestinal endoscopy training system with multidimensional data fusion according to claim 1, characterized in that: The multidimensional fusion processing module also includes a decision tree analysis model, which employs a random forest algorithm and comprises a forest of 50 decision trees, each with a maximum depth of 8 and a minimum number of splits per node of 2. The splitting attribute of each decision node is determined by calculating the information gain ratio, and the formula for calculating the information gain ratio is: Where GainRatio(D,a) is the information gain ratio, D is the training dataset, a is the attribute to be split, Gain(D,a) is the information gain of attribute a with respect to dataset D, and IV(a) is the intrinsic value of attribute a: the intrinsic value IV(a) is calculated using the following formula: Where V is the number of values ​​for attribute a, and D v Let D be a subset of samples where attribute a takes the value v.

4. The gastrointestinal endoscopy training system with multidimensional data fusion according to claim 1, characterized in that: The force feedback algorithm of the six-dimensional force feedback handle is as follows: in, Let K be the feedback force vector at time t. d Here is the damping coefficient matrix. K is the velocity vector. p This is the stiffness coefficient matrix. Let K be the displacement vector. i This is the integral coefficient matrix; The damping coefficient matrix K d It is a diagonal matrix, and the range of values ​​for the diagonal elements is: [0.5 N·s / m, 8 N·s / m], dynamically adjusted according to the elastic modulus of the virtual tissue.

5. The gastrointestinal endoscopy training system with multidimensional data fusion according to claim 1, characterized in that: The edge computing architecture adopts a fog computing layered model, including: The device layer includes the data acquisition module, haptic feedback module, and interactive terminal module, which are connected via a USB 3.2 interface; The edge layer includes local edge servers, equipped with ARM architecture processors, and running the QNX real-time operating system; The cloud layer includes a cluster of cloud-based servers and uses the Hadoop distributed computing framework. The edge layer and the cloud layer are isolated by bandwidth through 5G network slicing technology, and real-time haptic feedback data uses dedicated network slices.

6. The gastrointestinal endoscopy training system with multidimensional data fusion according to claim 1, characterized in that: The segmented pneumatic biomimetic digestive tract model includes: The esophageal segment is made of thermoplastic elastomer material with built-in shape memory alloy wires; The gastric segment is made of silicone material and has a built-in 16×16 array of micro-pressure sensors with a sampling frequency of 100Hz; The intestinal segment is divided into the ascending colon, transverse colon and descending colon. Each segment contains 3 independent pneumatic bionic muscle units, which are controlled by PWM signals with a frequency of 20kHz. The segments are connected by a magnetic quick-release interface, and an O-ring waterproof seal is provided at the interface.

7. The gastrointestinal endoscopy training system with multidimensional data fusion according to claim 1, characterized in that: The data acquisition module includes a medical image acquisition unit, which employs a mutual information-based registration algorithm, calculated using the following formula: MI(A,B)=H(A)+H(B)-H(A,B); Wherein, MI(A,B) represents mutual information, A and B represent medical images of different modalities, H(A) and H(B) represent their respective entropies, and H(A,B) represents the joint entropy; the physiological indicator acquisition unit includes a single-lead ECG sensor simulating a human body model, a piezoresistive blood pressure sensor, and a fiber optic respiratory motion sensor; the operation trajectory acquisition unit uses an optical tracking system and a nine-axis inertial measurement unit, and fuses positioning data through an extended Kalman filter algorithm.

8. A training method for a multidimensional data fusion gastrointestinal endoscopy training system, characterized in that, The gastrointestinal endoscopy training system using the multidimensional data fusion method according to any one of claims 1-7, with the multidimensional fusion processing module as the execution entity, includes the following steps: S1. Acquire multimodal medical imaging data, physiological index data, and operation trajectory data output by the data acquisition module; S2. Align the physiological indicator data and the operation trajectory data in time series to generate time series fusion data; S3. Perform three-dimensional spatial mapping on the medical image data to generate spatial mapping data; S4. Control the generative adversarial network unit to process the spatial mapping data and synthesize virtual lesions containing heterogeneous pathological features; S5. The control decision tree analysis model analyzes the time-series fusion data to generate operational evaluation data. S6, Trigger the scene generation submodule of the virtual simulation module to construct a virtual digestive tract scene based on the fused medical image data and virtual lesions; S7. Trigger the real-time rendering submodule of the virtual simulation module to render the scene in real time using the GPU acceleration unit and foveated rendering technology; S8. Send operation evaluation data to the six-dimensional force feedback handle and the segmented pneumatic biomimetic digestive tract model to trigger stress feedback and dynamic tension simulation of the digestive tract. S9. Receive eye-tracking data and operation instructions from the interactive terminal module, and adjust the rendering accuracy and interaction logic of the virtual digestive tract scene.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method of the gastrointestinal endoscopy training system with multidimensional data fusion as described in claim 8.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements the training method of a gastrointestinal endoscopy training system with multidimensional data fusion as described in claim 8.

Images