Real-time positioning and operation alarm system and method for digestive endoscope based on voice navigation

Abstract

The application provides a gastrointestinal endoscope real-time positioning and operation alarm system and method based on voice navigation, and belongs to the technical field of medical endoscopes, and comprises: an image processing module, which is used for anatomical structure recognition, lesion detection and image feature output of endoscope real-time images based on a deep neural network model; a posture detection module, which is used for acquiring posture features and mechanical features and quantifying a snaring risk; a multi-modal prediction module, which is used for unified time sequence modeling of the image features, the posture features and the mechanical features by using a multi-modal fusion framework, so that a predicted trajectory and a comprehensive risk index at a future time are obtained; and a voice navigation and alarm module, which is used for reminding an operator to make adjustments by voice guidance and screen visual prompts, and triggering an alarm response. The application can only synchronously broadcast key anatomical structures, and can also send operation suggestions to the operator in real time according to future trajectory prediction.

Description

Technical Field

[0001] This invention belongs to the field of medical endoscopy technology, and particularly relates to a real-time positioning and operation alarm system and method for digestive endoscopy based on voice navigation. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Digestive endoscopy is a core tool for diagnosing digestive tract diseases and plays a vital role in modern medical diagnosis and treatment. Doctors use endoscopic imaging to observe lesions in the esophagus and colon mucosa. However, current endoscopic techniques have several problems that urgently need to be addressed. Traditional endoscopic procedures rely heavily on the doctor's personal experience and are limited by blind spots, organ peristalsis, and interference from secretions, which can easily lead to missed lesions, lens deviation causing perforation or bleeding risks, and increase patient discomfort and procedure time.

[0004] There are also significant shortcomings in navigation systems for endoscopic procedures. Most existing navigation systems rely on image annotations, requiring doctors to be distracted by observing the screen during the procedure, making it difficult to maintain full concentration. Furthermore, current navigation systems cannot provide real-time warnings for potential abnormalities in the endoscope, such as looping or excessive traction, which undoubtedly increases the risk of complications.

[0005] From a technical perspective, existing endoscopic technologies also have many limitations. They primarily focus on image localization but lack integration with voice interaction and dynamic alarm functions. Traditional endoscopes also lack automated posture detection and intelligent voice navigation capabilities, forcing doctors to rely solely on their senses and experience during procedures, lacking effective auxiliary tools. Furthermore, most existing alarm systems are based on single sensors. This single detection method has significant limitations, lacking multimodal data fusion and hierarchical response mechanisms. It cannot accurately and comprehensively assess and classify various complex situations, failing to meet the high safety and accuracy requirements of actual clinical operations. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention proposes a voice-guided real-time positioning and operation alarm system and method for digestive endoscopy. Applicable to gastroscopy, colonoscopy, and other digestive endoscopy examinations and treatments, this system utilizes image visual analysis, posture detection, endoscope biomechanical sensing, and multimodal fusion technology based on artificial intelligence to achieve precise positioning, trajectory prediction, anomaly identification, and intelligent voice guidance of the digestive endoscope. This invention breaks through the limitations of traditional "image-based single-modal navigation" by proposing an "image-posture-mechanical three-modal fusion prediction framework," which can identify potentially high-risk actions in advance during actual clinical operations, such as looping, excessive bending, and blind-zone advancement, thereby significantly improving the safety and reliability of endoscopic operations.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention discloses a voice-guided real-time positioning and operation alarm system for gastrointestinal endoscopy, comprising: The image processing module is used to perform anatomical structure recognition, lesion detection, and output image features based on a deep neural network model in real-time endoscopic images. The attitude detection module is used to acquire attitude and mechanical characteristics and quantify the risk of buckling through the built-in IMU, magnetic positioner and bending force sensor; The multimodal prediction module is used to perform unified temporal modeling of image features, pose features and mechanical features using a multimodal fusion framework to obtain the predicted trajectory and comprehensive risk index for future moments; The voice navigation and alarm module is used to remind the operator to make adjustments based on the predicted trajectory through voice guidance and on-screen visual prompts, and to trigger an alarm response based on the looping risk and comprehensive risk index.

[0008] Secondly, this invention discloses a method for real-time positioning and operation alarm of a digestive endoscope based on voice navigation, comprising: Based on a deep neural network model, anatomical structure recognition and lesion detection are performed on real-time endoscopic images, and image features are output. The system acquires attitude and mechanical characteristics and quantifies the risk of buckling by using a built-in IMU, magnetic locator, and bending force sensor. A multimodal fusion framework is adopted to perform unified temporal modeling of image features, pose features and mechanical features to obtain the predicted trajectory and comprehensive risk index for future moments; Based on the predicted trajectory, the operator is prompted to make adjustments via voice guidance and on-screen visual prompts, and an alarm response is triggered based on the looping risk and comprehensive risk index.

[0009] Thirdly, the present invention discloses an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps of the above-mentioned voice-guided real-time positioning and operation alarm method for gastrointestinal endoscopy.

[0010] Fourthly, the present invention discloses a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above-mentioned voice-guided real-time positioning and operation alarm method for gastrointestinal endoscopy.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: The active voice navigation proposed in this invention can not only broadcast key anatomical structures in real time, but also provide operators with operation suggestions such as "please make a slight adjustment to the left" and "be careful of excessive bending angle" based on future trajectory prediction, forming an AI co-pilot-style intelligent collaboration mode.

[0012] This invention can record the entire operation process and generate a real-time operation quality score based on indicators such as trajectory stability, number of risky actions, and proportion of ineffective advancements, which can be used for postoperative teaching and quality assessment.

[0013] This invention has significant advantages such as strong real-time performance, high positioning accuracy, outstanding predictive ability, perfect alarm mechanism, and high degree of intelligent voice navigation. It is particularly suitable for complex operation paths such as gastroscopy and colonoscopy, and can significantly reduce the incidence of high-risk events such as perforation and bleeding, improve the operation ability of novice doctors, and improve examination efficiency and diagnostic accuracy.

[0014] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0016] Figure 1 This is a structural diagram of the voice-guided real-time positioning and operation alarm system for digestive endoscopy as described in Embodiment 1 of the present invention.

[0017] Figure 2 This is a logic block diagram of the multimodal prediction module described in Embodiment 1 of the present invention. Detailed Implementation

[0018] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0020] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0021] Example 1 In one or more embodiments, a voice-guided real-time positioning and operation alarm system for gastrointestinal endoscopy is disclosed, such as... Figure 1 As shown, it includes: The image processing module is used to perform anatomical structure recognition, lesion detection, and output image features based on a deep neural network model in real-time endoscopic images. The attitude detection module is used to acquire attitude and mechanical characteristics and quantify the risk of buckling through the built-in IMU, magnetic positioner and bending force sensor; The multimodal prediction module is used to perform unified temporal modeling of image features, pose features and mechanical features using a multimodal fusion framework to obtain the predicted trajectory and comprehensive risk index for future moments; The voice navigation and alarm module is used to remind the operator to make adjustments based on the predicted trajectory through voice guidance and on-screen visual prompts, and to trigger an alarm response based on the looping risk and comprehensive risk index.

[0022] The image processing module includes an acquisition submodule and a calculation submodule. The acquisition submodule is used to acquire image sequences in real time at 25–60 fps through the endoscope front-end camera, and performs automatic exposure, white balance calibration, edge sharpening and noise reduction on the acquired image sequences to ensure stable image quality even under gastric acid reflection, mucus occlusion and fast movement scenarios. The calculation submodule is used to use a pre-trained deep neural network to perform anatomical structure recognition, lesion detection, image feature extraction and calculation of image feature stability index.

[0023] The computational submodule specifically employs a pre-trained deep neural network (CNN, Transformer, or a combination thereof) to perform the following tasks: 1. Identify anatomical structures, including the cardia, pylorus, antrum, ileocecal valve, etc.

[0024] 2. Detect lesions, including polyps, ulcers, erosions, etc.

[0025] 3. Extract image features to generate feature vectors. In this embodiment, feature vectors are obtained from the mid-to-high-level features used for feature representation in the network.

[0026] 4. Calculate the image feature stability index.

[0027] To quantify image stability, this invention proposes the following innovative indicators:

[0028] Where N is the number of feature points. This represents the feature vector of the image sequence at time t. This is the feature vector corresponding to the previous time step. This is an image feature stability index. When an image is jittery, occluded, or blurred, the cosine similarity decreases, thus increasing the stability index.

[0029] Preferably, the lens motion trajectory is generated based on the image sequence, providing basic visual features for the multimodal prediction module. The lens motion trajectory consists of image features or positional information corresponding to endoscopic images at consecutive time points, used to describe the movement state of the endoscope lens within the digestive tract over time. Specifically, the lens motion trajectory can be represented as a temporal data sequence composed of multiple time steps, each time step corresponding to a lens position, orientation, or motion state parameters inferred from image features, thus forming motion trajectory information reflecting the endoscope's advancement, retraction, and posture changes. This motion trajectory maintains consistency with the image features in the temporal dimension and can be used as temporal input for subsequent multimodal fusion and predictive analysis.

[0030] The attitude detection module acquires attitude and mechanical characteristics and quantifies the risk of buckling by using a built-in IMU, magnetic locator and bending force sensor; the attitude and mechanical characteristics include the three-dimensional attitude of the mirror body, bending angle, rotation state and force conditions.

[0031] This embodiment uses an IMU, electromagnetic positioning and bending force sensors to achieve full-dimensional monitoring of the endoscope's posture.

[0032] Specifically, a three-axis accelerometer and a three-axis gyroscope are used to calculate the real-time acceleration, angular velocity, and rotational attitude of the endoscope. An electromagnetic positioning unit, with a miniature electromagnetic coil embedded in the endoscope tip, achieves spatial positioning error <2mm through an external magnetic field positioning system. A bending force sensor is used to acquire bending force and its rate of change, determining the stress trend on the endoscope.

[0033] Furthermore, this embodiment proposes a loophole trend factor model to quantify the loophole risk, expressed as follows:

[0034] in, To mitigate risk indicators, For real-time bending angle, The rate of change of bending force. Angular velocity of rotation , , These are the weight coefficients obtained through training and optimization. When... When the set threshold is exceeded, the system enters a warning or danger state.

[0035] This module outputs attitude feature vectors, providing crucial data for subsequent risk prediction.

[0036] like Figure 2 As shown, the multimodal prediction module uses a multimodal fusion framework to perform unified temporal modeling of image features, pose features, and mechanical features to obtain the predicted trajectory and comprehensive risk index for future moments; wherein, the multimodal fusion framework is a Transformer or Kalman filter fusion framework.

[0037] This embodiment employs an image-attitude-mechanical three-modal fusion method and establishes a comprehensive risk index (R) prediction model, specifically: The multimodal fusion framework receives input data including image feature vectors. Attitude feature vector (Provided by the attitude detection module), mechanical feature vector (Provided by a bending force sensor).

[0038] Cross-modal fusion is performed using the Transformer multi-head attention mechanism or a temporal Kalman filter model.

[0039] The multimodal prediction model (i.e., the multimodal fusion framework) outputs the predicted position of the lens at a future time point τ, and the trajectory offset is defined as:

[0040] in, Predict the position of the camera. This represents the normal statistical trajectory of the lumen center. When the deviation exceeds a threshold, the system predicts a potential risk of entering a blind lumen, fold, or bending.

[0041] Real-time risk classification is achieved using a comprehensive risk index, which is:

[0042] in, For image stability index, For the curved angle of the mirror body, The rate of change of bending force. This represents the trajectory offset in the future τ seconds (τ = 0.5–2 s). , , , The weights are obtained through training with data. The activation function is Sigmoid. The system uses the R-value to achieve real-time risk classification. Its training process primarily involves parameter learning for the multimodal fusion prediction model to obtain model parameters for shot position prediction and risk assessment. The weight parameters in the comprehensive risk index can be set based on training data or empirical rules to weightedly fuse multiple risk-related factors. The voice navigation and alarm module reminds the operator to make adjustments based on the predicted trajectory through proactive voice guidance and on-screen visual prompts, and triggers an alarm response based on the comprehensive risk index.

[0043] The risk prediction results are transformed into real-time instructions to achieve predictive proactive navigation. The risk prediction results include the risk of looping and the comprehensive risk index.

[0044] Specifically, the tiered alarm mechanism includes: Level L1: Mild risk, indicating anatomical structures and directional suggestions; Warning Level L2: The risk of a loophole or the comprehensive risk index reaches the warning threshold; High-risk level L3: Risk of perforation or severe bending exists, triggering a red light flashing and a high-risk voice prompt. Specifically, the triggering of high-risk level L3 is based on a combined judgment of the comprehensive risk index, the risk of looping, and its key components. When the comprehensive risk index exceeds the high-risk threshold, or the risk of looping exceeds the high-risk threshold, or when the combined conditions such as "increased trajectory deviation and a sharp increase in bending angle / force" are met simultaneously, a high risk of bending injury is determined, and an L3 alarm is triggered. The threshold can be set based on historical data statistics and the doctor's clinical experience.

[0045] Based on the comprehensive risk index, trajectory deviation, and real-time curvature angle, navigation commands are generated. This embodiment proposes the following decision model for achieving intelligent voice navigation:

[0046] in, Indicates left-hand, neutral, and right-hand keys. The direction probability is calculated based on risk parameters, and f(·) represents the output natural language instruction, such as "fine-tune to the left," "maintain direction," or "beware of rightward deviation." Predictive voice prompts can alert the operator 0.5–2 seconds in advance, significantly reducing the risk of endoscopic injury.

[0047] Next, we will use gastroscopy as an example to illustrate the actual use of this invention: 1. Initialization phase; IMU automatic calibration and electromagnetic positioning activation; the system enters tracking mode.

[0048] 2. Enters the esophagus; The image module identifies the esophageal structure and prompts "Entering the esophageal segment".

[0049] 3. Enters the stomach body and antrum; The fusion module calculates the risk index R in real time.

[0050] If the image shifts significantly or the pose changes abnormally, navigation instructions will be automatically generated.

[0051] 4. Increased bending force (predicted risk). when or Upon entering L2 warning, a voice prompt reads, "The bending angle has increased; please reduce thrust." 5. Potential for tangling (high risk); when: or The system prematurely enters L3 alert mode: "Possible blockage, please stop advancing immediately." 6. When a lesion is detected, the image module will display the message: "Suspected polyp detected. Please zoom in for further observation." 7. Inspection complete.

[0052] The system outputs an operation score, and the scoring model is as follows:

[0053] in, Number of dangerous actions The peak bending force is used to evaluate operational smoothness and safety. This represents the cumulative offset of the lens trajectory within a predetermined time window, reflecting the overall stability during endoscopic procedures. k1, k2, and k3 are weighting coefficients used to weight different scoring factors. k1 is the deduction coefficient for the number of dangerous actions; for each dangerous action, k1 points are deducted. k2 is the deduction coefficient for the cumulative trajectory offset. For every additional unit (e.g., mm or pixel normalization unit), deduct k2 points; k3 is the deduction coefficient for peak bending force. The larger the peak bending force, the "harder" the operation, and the more points are deducted.

[0054] This invention constructs an intelligent navigation model by fusing multimodal data (image localization + posture sensing), integrating dynamic voice guidance and graded alarms for abnormal states. It overcomes the limitations of traditional single-image navigation, enabling "eye-hand-ear" collaborative operation and significantly improving examination safety and operational efficiency. It assists doctors in using digestive endoscopes more safely and conveniently, achieving precise endoscopic navigation and reducing the rate of missed lesion detection and operational risks.

[0055] Example 2 In one or more embodiments, a method for real-time positioning and operation alarm of a digestive endoscope based on voice navigation is disclosed, specifically including: Based on a deep neural network model, anatomical structure recognition and lesion detection are performed on real-time endoscopic images, and image features are output. The system acquires attitude and mechanical characteristics and quantifies the risk of buckling by using a built-in IMU, magnetic locator, and bending force sensor. A multimodal fusion framework is adopted to perform unified temporal modeling of image features, pose features and mechanical features to obtain the predicted trajectory and comprehensive risk index for future moments; Based on the predicted trajectory, the operator is prompted to make adjustments via voice guidance and on-screen visual prompts, and an alarm response is triggered based on the looping risk and comprehensive risk index.

[0056] Example 3 This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps of the above-mentioned method for real-time positioning and operation alarm of digestive endoscopy based on voice navigation.

[0057] Example 4 This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they complete the steps of the above-described method for real-time positioning and operation alarm of digestive endoscopy based on voice navigation.

[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A voice-guided real-time positioning and operation alarm system for gastrointestinal endoscopy, characterized in that, include: The image processing module is used to perform anatomical structure recognition, lesion detection, and output image features based on a deep neural network model in real-time endoscopic images. The attitude detection module is used to acquire attitude and mechanical characteristics and quantify the risk of buckling through the built-in IMU, magnetic positioner and bending force sensor; The multimodal prediction module is used to perform unified temporal modeling of image features, pose features and mechanical features using a multimodal fusion framework to obtain the predicted trajectory and comprehensive risk index for future moments; The voice navigation and alarm module is used to remind the operator to make adjustments based on the predicted trajectory through voice guidance and on-screen visual prompts, and to trigger an alarm response based on the looping risk and comprehensive risk index.

2. The voice-guided real-time positioning and operation alarm system for gastrointestinal endoscopy as described in claim 1, characterized in that, A pre-trained deep neural network is used to extract image features and generate feature vectors. The image feature stability index is then calculated, expressed as: Where N is the number of feature points. This represents the feature vector of the image sequence at time t. This is the feature vector corresponding to the previous time step. This is an index for image feature stability.

3. The voice-guided real-time positioning and operation alarm system for gastrointestinal endoscopy as described in claim 1, characterized in that, The quantitative risk of the pullback is expressed as follows: in, To mitigate risk indicators, For real-time bending angle, The rate of change of bending force. Angular velocity of rotation , , These are the weight coefficients obtained through training and optimization.

4. The voice-guided real-time positioning and operation alarm system for gastrointestinal endoscopy as described in claim 1, characterized in that, The multimodal prediction module outputs the predicted position of the lens at a future time point τ, and the trajectory offset is defined as: in, Predict the position of the camera. This represents the normal statistical trajectory of the lumen center.

5. The voice-guided real-time positioning and operation alarm system for gastrointestinal endoscopy as described in claim 1, characterized in that, The comprehensive risk index is: in, For image stability index, For the curved angle of the mirror body, The rate of change of bending force. This represents the trajectory offset in the next τ seconds. , , , The weights are obtained through training with data. This is the Sigmoid activation function.

6. The voice-guided real-time positioning and operation alarm system for gastrointestinal endoscopy as described in claim 1, characterized in that, Based on the predicted trajectory, the operator is prompted to make adjustments via voice guidance and on-screen visual prompts. The decision-making model is as follows: in, Indicates left-hand, neutral, and right-hand keys. f(·) represents the directional probability calculated based on the risk parameters, and f(·) represents the output natural language instruction.

7. The voice-guided real-time positioning and operation alarm system for gastrointestinal endoscopy as described in claim 1, characterized in that, The alarm response triggered based on the looping risk and comprehensive risk index, the tiered alarm mechanism includes: Level L1: Mild risk, indicating anatomical structures and directional suggestions; Warning Level L2: The risk of a loophole or the comprehensive risk index reaches the warning threshold; High-risk level L3: There is a risk of perforation or severe bending, triggering red light flashing and a high-risk voice message.

8. A method for real-time positioning and operation alarm of gastrointestinal endoscopy based on voice navigation, characterized in that, include: Based on a deep neural network model, anatomical structure recognition and lesion detection are performed on real-time endoscopic images, and image features are output. The system acquires attitude and mechanical characteristics and quantifies the risk of buckling by using a built-in IMU, magnetic locator, and bending force sensor. A multimodal fusion framework is adopted to perform unified temporal modeling of image features, pose features and mechanical features to obtain the predicted trajectory and comprehensive risk index for future moments; Based on the predicted trajectory, the operator is prompted to make adjustments via voice guidance and on-screen visual prompts, and an alarm response is triggered based on the looping risk and comprehensive risk index.

9. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the voice-guided real-time positioning and operation alarm method for gastrointestinal endoscopy as described in claim 8.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by the processor, complete the real-time positioning and operation alarm method for gastrointestinal endoscopy based on voice navigation as described in claim 8.

Images