System and method for automatic stomach screening

Abstract

A system for imaging one or more particular structures in the stomach within a workspace is provided. In one embodiment, the system comprises: a) an active capsule endoscope positionable in the stomach, comprising a camera, an inertial measurement unit, and a permanent magnet; the driving unit comprises a Hall sensor array located at the bottom of the working space and a plurality of magnetic field generators used for generating expected magnetic fields; and c) a control unit.

Description

TECHNICAL FIELD

[0001] The present application relates to a system and method for automatic gastric screening, in particular to a system comprising an active capsule endoscope and a magnetic drive unit, capable of active screening inside the stomach, and a method of controlling the active capsule endoscope to perform automatic gastric screening, which sequentially observes six main areas of the stomach with the aid of an image artificial intelligence model and a closed-loop motion controller. BACKGROUND

[0002] Gastric cancer is one of the most serious health challenges, with high incidence and mortality rates. In 2020, more than one million cases were reported worldwide, and 760,000 people died. To date, regular screening and early diagnosis remain the most effective means of preventing gastric cancer. However, due to the shortage of endoscopy physicians, especially in developing countries, large-scale screening programs for gastric cancer are still limited. Wireless capsule endoscopy provides a non-invasive diagnostic method for the gastrointestinal tract, especially the stomach.

[0003] Currently, wireless capsule endoscopes can be classified into two categories according to their locomotion characteristics: (1) passive capsule endoscopes (PCEs) and (2) active capsule endoscopes (ACEs). There have been many PCEs developed by researchers and commercial companies, such as US Patent No. US7244229B2, titled “Capsule endoscope”, filed on August 10, 2005, and US Patent No. US7316647B2, titled “Capsule endoscope and a capsule endoscope system”, filed on April 23, 2004. PCEs rely on the peristaltic waves of the gastrointestinal tract to move passively and transmit the collected images to the physician for subsequent diagnosis. However, this passive movement limits the region of interest (ROI) that can be diagnosed, thus reducing the accuracy and reliability of the diagnosis. In contrast, ACEs are able to achieve active locomotion in the gastrointestinal tract through various locomotion mechanisms, such as legged propulsion (US Patent No. US8066632B2, titled “Teleoperated endoscopic capsule equipped with active locomotion system”, filed on February 17, 2005), inchworm propulsion (US Patent No. US7365509B2, titled “Capsule type micro-robot moving system”, filed on May 10, 2006), and magnetic actuation (US Patent No. US9125557B2, titled “Magnetic maneuvering system for capsule endoscope”, filed on May 30, 2012). Among these locomotion methods, magnetic actuation has shown promising prospects in clinical applications, as it is able to transfer functional torque and force to the capsule endoscope, thus enabling locomotion control through magnetic fields, rather than through other bulky onboard mechanisms.In view of the advantages of magnetic actuation, some MACEs with different functions have also been developed, such as biopsy function (US Patent No. US9125557B2, titled “Magnetically actuated capsule endoscope, magnetic field generating and sensing apparatus and method of actuating a magnetically actuated capsule endoscope”, application date May 23, 2018) and drug delivery function (US Patent No. US9445711B2, titled “System and method to magnetically actuate a capsule endoscopic robot for diagnosis and treatment”, application date May 9, 2013), thereby improving its applicability in clinical practice.

[0004] Generally, MACEs have higher maneuverability, can achieve more comprehensive diagnostic results and shorten the time of examination procedures, so that the energy of the embedded battery is more used to provide high-resolution real-time image transmission rather than to maintain long-time endurance under low-quality, low-frame-rate images. The maneuverability of the developed MACEs depends on their driving systems, which can be divided into two major categories: (1) permanent magnet based actuation (PMBA) systems; (2) electromagnetic actuation (EMA) systems. For PMBA systems, it is usually required to use a mechanical arm to carry a permanent magnet to generate the expected magnetic field, thereby achieving multi-degree-of-freedom (DOF) motion control of MACEs. For example, Mahoney et al. (Mahoney et al.,“Five-degree-of-freedom manipulation of an untethered magnetic device in fluid using a single permanent magnet with application in stomach capsule endoscopy.” The International Journal of Robotics Research 35.1-3 (2016): 129-147.) used a single permanent magnet mounted on a mechanical arm to achieve high-accuracy five-DOF closed-loop control (i.e., three-DOF position control and two-DOF orientation control) of a capsule robot in a liquid environment. However, the controllability of this method is limited due to the single-directional attractive force from the large external permanent magnet, resulting in insufficient maneuverability, non-ideal motion response time, and complex maneuvering process.

[0005] Compared with PMBA systems, EMA systems are more easily to achieve multi-DOF motion control of MACEs by changing the current applied to multiple electromagnets. Hoang et al. (Hoang et al.,“Electromagnetic actuation of a magnetic capsule endoscope.” IEEE Transactions on Robotics 27.6 (2011): 1026-1034.) used a pair of electromagnets to achieve two-DOF motion control of a MACE in a liquid environment. However, the controllability of this method is limited due to the single-directional attractive force from the large external permanent magnet, resulting in insufficient maneuverability, non-ideal motion response time, and complex maneuvering process. “Independent electromagnetic field control for practical approach toactively locomotive wireless capsule endoscope.”IEEE Transactions on Systems,Man,and Cybernetics:Systems 51.5(2019):3040-3052.) and Lee et al. (Lee et al.,“Active locomotive intestinal capsule endoscope(ALICE)system:A prospective feasibility study.”IEEE / ASME Transactions on Mechatronics 20.5(2014):2067-2074.) achieved five degrees of freedom open-loop locomotion control of MACEs by combining uniform and gradient magnetic fields generated by multiple pairs of air-core electromagnetic coils. With further development, Song et al. (Song et al.,“Motion Control of Capsule Robot Based on Adaptive Magnetic Levitation Using Electromagnetic Coil.”IEEE Transactions on Automation Science and Engineering(2022).) achieved closed-loop control of the capsule robot’s suspension and direction. Their method utilized three pairs of air-core Helmholtz coils to generate uniform magnetic fields and a three-degree-of-freedom movable electromagnetic coil to generate gradient magnetic fields, thus achieving more flexible motion control of MACEs. Compared to the air-core electromagnetic coils used in the above-mentioned EMA systems, the combination of a core and a coil can generate a magnetic field with higher intensity than an air-core electromagnet, thereby reducing energy consumption and heat generation. For example, Kummer et al. (Kummer et al.,“OctoMag:An electromagnetic system for 5-DOF wireless micromanipulation.”IEEE Transactions on Robotics 26.6(2010):1006-1017.) reported a well-designed EMA system based on a core that can generate a non-uniform magnetic field and achieve five degrees of freedom motion control of magnetic objects.However, most current EMA systems lack full 6D pose feedback of the MACEs or rely on external cameras to identify the pose of the capsule, which is not practical for closed-loop control in a clinical environment. To achieve closed-loop control of MACEs with multiple degrees of freedom, a feasible and accurate in-vivo localization scheme needs to be integrated into these systems. Some related magnetic localization studies are listed as follows: a. Son et al.: "A 5-D localization method for a magnetically manipulated untethered robot using a 2-D array of Hall-effect sensors." IEEE / ASME transactions on mechatronics 21.2 (2015): 708-716. b. Dai et al.: "A novel 6-D tracking method by fusion of 5-D magnetic tracking and 3-D inertial sensing." IEEE Sensors Journal 18.23 (2018): 9640-9648. c. Zhang et al.: "6-D spatial localization of wireless magnetically actuated capsule endoscopes based on the fusion of hall sensor array and IMU." IEEE Sensors Journal 22.13 (2022): 13424-13433. d. Placidi et al.: "Review on patents about magnetic localisation systems for in vivo catheterizations." Recent Patents on Biomedical Engineering (Discontinued) 2.1 (2009): 58-64.

[0006] With the development of various high-maneuverable MACEs, improving their level of autonomy has become the natural direction of future development to shorten the time of gastric screening, reduce the learning curve of doctors, and reduce the workload of users, so that endoscopic doctors can focus more on the diagnosis of abnormal conditions in the stomach. Currently, most capsule endoscopy research focuses on using artificial intelligence (AI) technology for abnormal condition diagnosis to reduce the burden on doctors (see: Wu et al., “Real-time artificial intelligence for detecting focal lesions and diagnosing neoplasms of the stomach by white-light endoscopy (with videos).” Gastrointestinal Endoscopy 95.2 (2022): 269-280.). However, AI-assisted diagnosis usually relies on capsule endoscopy to collect a large number of high-quality gastric images. Yao et al. (Yao K. “The endoscopic diagnosis of early gastric cancer”. Ann Gastroenterol. 2013; 26(1): 11-22.) proposed a systematic gastric screening program, pointing out that the entire stomach atlas construction and diagnosis can be completed by 22 endoscopic images from different gastric regions. Xiao et al. (Xiao et al. “Fully automated magnetically controlled capsule endoscopy for examination of the stomach and small bowel: a prospective, feasibility, two-centre study”. The Lancet Gastroenterology & Hepatology 6.11 (2021): 914-921.) reported an automated gastric examination process that uses a MACE to detect and record six main regions of the stomach, including the cardia, fundus, body, antrum, angularis, and pylorus. However, this system lacks closed-loop control of the capsule endoscope, resulting in deficiencies in the efficiency and reliability of the screening procedure. Therefore, if advanced control strategies and artificial intelligence technology are combined, automated control of MACEs can be achieved to automatically detect the above-mentioned gastric regions and collect images from the optimal viewing angle, thereby achieving automated and comprehensive gastric screening.

[0007] So far, there are very few capsule endoscopes capable of automatic gastric screening based on artificial intelligence for gastric region detection and active motion control. If a capsule endoscope with active driving and closed-loop motion control capability is introduced to realize such an automatic gastric screening process, it will have a significant advantage, which can greatly shorten the screening time and effectively reduce the workload of doctors. SUMMARY

[0008] The present application proposes an automatic gastric screening method using a magnetically driven capsule endoscope, which can shorten the screening time and reduce the workload of doctors while ensuring the accuracy and reliability of the diagnostic results. The method disclosed in the present application combines artificial intelligence-based detection technology and advanced closed-loop motion controllers.

[0009] In the first aspect of the present application, the automatic screening method consists of a carefully designed workflow that controls the capsule endoscope to observe the six main regions of the stomach in sequence to realize the mapping of the entire stomach. The method disclosed in the present application uses a capsule endoscope to collect images in the stomach and uses an electromagnetic drive system composed of multiple coils and cores to drive the capsule endoscope. In an embodiment of the present application, a fuzzy proportional-integral-derivative (PID) controller is used to drive the capsule endoscope to move to different regions of the stomach, and a fast region convolutional neural network (Faster-RCNN), a widely used learning-based object detection benchmark model, is used to detect regions of the stomach with specific features. In order to make the capsule endoscope collect images of the detected stomach region from the optimal angle, a proportional sliding mode visual servoing controller is used to adjust the direction of the capsule. The present application also uses a six-dimensional magnetic positioning strategy that combines the magnetic field data measured by the Hall sensor array and the direction data output from the capsule inertial measurement unit (IMU) to estimate the six-dimensional pose of the capsule, thereby realizing closed-loop motion control of the capsule. The artificial intelligence detection technology, motion controller, and positioning algorithm used in the present application can also be replaced by other methods with similar performance.

[0010] In an embodiment of the present application, the capsule endoscope includes a ring-shaped permanent magnet, multiple printed circuit boards (PCBs), and a housing for accommodating the above components. More specifically, the PCBs include an image acquisition unit, an illumination unit, a data transmission unit, and an inertial measurement unit (IMU) for collecting three-dimensional directional information of the capsule. In an embodiment of the present application, the capsule endoscope can wirelessly transmit video data and IMU data to an external control unit, and the control unit can send instructions to the capsule endoscope to adjust the data transmission frequency and the illumination intensity.

[0011] In one embodiment of the present application, the magnetic drive unit is composed of a plurality of electromagnets, which are distributed above and below to surround a working space for accommodating the subject's body. Each electromagnet includes a copper coil and a core. The configuration of the electromagnet array can be adjusted according to different application scenarios, surrounding a working space sufficient to accommodate the subject's body.

[0012] In one embodiment of the present application, by applying a specific current to the electromagnets, an expected magnetic field can be generated at the location of the capsule endoscope, thereby generating a magnetic force and a magnetic moment applied to the capsule, achieving the steering and translation of the capsule endoscope.

[0013] In another embodiment of the present application, a Hall sensor array is provided at the bottom of the working space to achieve five-dimensional positioning of the capsule endoscope, thereby achieving closed-loop motion control of the capsule. By fusing the IMU data and the magnetic field data measured by the Hall sensor array at the bottom of the working space, the six-dimensional pose of the capsule endoscope is obtained.

[0014] In the second aspect of the present application, the workflow of the present application is described as follows. First, the subject drinks a large amount of water before the screening procedure begins to fill the stomach and provide buoyancy for the capsule endoscope. Then, the subject swallows the capsule endoscope and lies on the bed in the drive system. The drive system generates a magnetic field to control the translation and direction of the capsule endoscope, image acquisition of the six main regions of the stomach, and closed-loop control using the six-dimensional pose information of the capsule. The six main regions of the stomach are detected in turn by artificial intelligence technology, including: cardia, gastric fundus, gastric body, gastric antrum, gastric angle and pylorus. More than three images are collected for each gastric region, and the direction of the capsule is adjusted to the optimal viewing angle through visual servoing control. Then, after enough images are collected to achieve the mapping of the entire stomach, the doctor can manually control the capsule endoscope through the joystick on the control unit to perform detailed diagnosis on the target region. Finally, the capsule endoscope is naturally expelled from the body under the action of the gastrointestinal peristaltic wave.

[0015] In the third aspect of the present application, a magnetic drive model of the drive system is established to reveal the relationship between the current applied to the electromagnets and the magnetic field (magnetic force or magnetic moment) at a certain position. The kinematic model of the capsule endoscope is analyzed to achieve visual servoing control based on the detected features of the stomach for adjusting the direction of the capsule. In addition, a positioning model is constructed to obtain the six-dimensional pose of the capsule endoscope, which is based on the fusion of IMU data and magnetic field data measured by the Hall sensor array.

[0016] The above and other features, advantages, and important significance of the present application in technology and industry will be better understood by reading the following detailed description of the presently preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A system flow chart showing the procedure for automated gastric screening using a magnetically driven capsule endoscope assisted by artificial intelligence detection technology and an advanced closed-loop motion controller.

[0018] Figure 2 The present invention provides a system overview of a magnetically driven capsule endoscopy device for automatic gastric screening, comprising: 1. a capsule endoscope; 2. an upper electromagnet module; 3. a drive system base; 4. a subject's bed; 5. a lower electromagnet module; 6. a Hall sensor array; 7. Doctor's chair; 8. Workbench; 9. First display screen, used to display the stomach image detected by artificial intelligence; 10. Stomach image detected by artificial intelligence; 11. Second display screen, used to display the original stomach image; 12. Main control computer.

[0019] Figure 3 The front view of the capsule endoscope proposed according to the features of the present invention is shown, including: 1-1: Bluetooth (BLE) module; 1-2: Inertial measurement unit (IMU); 1-3: Annular permanent magnet; 1-4: Housing; 1-5: Camera module; 1-6: Light emitting diode; 1-7: Analog video (AV) transmitter; 1-8: AV antenna; 1-9: Power processing module; 1-10: Battery.

[0020] Figure 4 The driving system base in the magnetically driven capsule endoscopy device proposed according to the characteristics of the present invention is shown, which mainly includes: 3-1: an upper driving installation space for installing an upper electromagnet module; 3-2: a workspace for placing a subject's bed and the subject; 3-3: a lower installation space for installing a lower electromagnet module and a Hall sensor array.

[0021] Figure 5 (a)-(d) illustrate partial cross-sectional views of an upper or lower electromagnet module of a magnetically driven capsule endoscopy device according to features of the present invention. The electromagnet module comprises a plurality of electromagnets arranged in various configurations. In this embodiment, an upper or lower electromagnet module comprises three to six electromagnets. Each electromagnet comprises: 5-1: a copper coil; 5-2: a pure iron core.

[0022] Figure 6 A schematic diagram showing the positioning of a magnetically driven capsule endoscopy device according to the present invention mainly includes: 6-1: a printed circuit board with a Hall sensor installed; 6-2: a three-axis Hall sensor; 13: a stomach environment.

[0023] Figure 7A block diagram of the capsule pose estimation algorithm, which solves an optimization problem by fusing the magnetic field data measured by the Hall sensor array and the orientation information measured by the Inertial Measurement Unit (IMU) of the capsule endoscope.

[0024] Figure 8 A block diagram of an embodiment of a fuzzy PID position controller with six-dimensional capsule pose feedback.

[0025] Figure 9 A block diagram showing the kinematic analysis process of the capsule endoscope: (a) the mapping from the world coordinate frame to the capsule coordinate frame, the magnetic moment vector m c and the displacement vector r c represent the pose of the capsule. m c can be represented by two angles and in two orthogonal planes. (b) the coordinate frame of the capsule is established using the Denavit-Hartenberg (DH) modeling method, and two virtual orthogonal revolute joints are configured inside the capsule.

[0026] Figure 10 A pinhole camera model used for visual servoing control is shown. A point r(x, y, z) in space is projected to a point p(u, v) on the image plane.

[0027] Figure 11 A block diagram showing the process of adjusting the orientation of the capsule endoscope using visual servoing control to capture images from the optimal viewing angle.

[0028] Figure 12 An example of controlling the capsule endoscope to move from one position to another target position while adjusting the orientation using magnetic field and magnetic force is shown. This process includes four operations: I: the starting position in the stomach; II: the second position in the stomach; III: the third position in the stomach; IV: the end position in the stomach.

[0029] Figure 13 Examples of the six main regions of the stomach detected by artificial intelligence detection technology are shown, including: (a): the cardia of the stomach; (b): the fundus of the stomach; (c): the body of the stomach; (d): the antrum of the stomach; (e): the angle of the stomach; (f): the pylorus of the stomach.

[0030] Figure 14(a)-(f) demonstrate the workflow of performing automatic gastric screening using the magnetic driven capsule endoscope device of the present application, which covers six main regions of the stomach: 13-1: cardia of the stomach; 13-2: fundus of the stomach; 13-3: body of the stomach; 13-4: antrum of the stomach; 13-5: angularis of the stomach; 13-6: pylorus of the stomach. The capsule endoscope can rotate and translate to a specific region of the stomach according to the features of the present application, perform the screening task, and transmit the images to the information processing module.

[0031] Figure 15 An experimental setup of driving a capsule endoscope using nine electromagnets is demonstrated. (a): experimental setup of automatic gastric detection; (b): visual servoing control process of the capsule; (c): self-loop trajectory running process of the capsule.

[0032] Figure 16 Experimental results of visual servoing control are demonstrated. (a): pixel trajectory of the target; (b): corresponding pixel error.

[0033] Figure 17 Experimental setup and process of automatic gastric screening experiment are demonstrated. (a): top view of the experimental setup; (b): side view of the experimental setup; (c): views of the four markers; (d): four detected target features.

[0034] Figure 18 Experimental results of automatic gastric screening bench experiment are demonstrated. (a): capsule position control results; (b): pixel error of the four tracked target features. DETAILED DESCRIPTION

[0035] The exemplary embodiments of the capsule endoscope of the present application will be described in detail below with reference to the accompanying drawings. The present application is not limited to the embodiments described.

[0036] The present invention provides a system for imaging one or more specific structures in a stomach within a workspace. In one embodiment, the system comprises: a) an active capsule endoscope disposable in the stomach, comprising a camera, an inertial measurement unit, and a permanent magnet; b) a driving unit comprising a Hall sensor array located at the bottom of the workspace, and a plurality of magnetic field generators for generating a desired magnetic field; c) a control unit comprising an image module, a closed-loop position controller, and a vision-based servo controller; the control unit acquires images from the camera and orientation data from the inertial measurement unit; the control unit acquires magnetic field data of the desired magnetic field from the Hall sensor array; the control unit generates control inputs to the driving unit to drive the active capsule endoscope to move along a predetermined trajectory in the stomach; the control unit calculates the six-dimensional pose χ of the active capsule endoscope in real time using the orientation data and the magnetic field data; the six-dimensional pose χ is fed back to the closed-loop position controller to control the plurality of magnetic field generators to adjust the translation and orientation, so that the active capsule endoscope remains on the predetermined trajectory; the vision-based servo controller controls the plurality of magnetic field generators based on the images and the six-dimensional pose χ to drive the active capsule endoscope to image the one or more specific structures, and the image module matches the images with a database containing gastric markers and general gastric three-dimensional structures to locate and image the one or more specific structures.

[0037] In one embodiment, the image module matches the images by artificial intelligence.

[0038] In one embodiment, the system further comprises a dynamic trajectory generator for guiding the active capsule endoscope to move along the predetermined trajectory.

[0039] In one embodiment, the active capsule endoscope further comprises one or more illumination units or data transmission units.

[0040] In one embodiment, the Hall sensor array comprises one or more three-axis Hall sensors, which are distributed in the three-dimensional space of the workspace according to the desired magnetic field.

[0041] In one embodiment, each of the one or more three-axis Hall sensors has a different direction relative to the orthogonal coordinate system (X, Y, Z) of the desired magnetic field.

[0042] In one embodiment, the control unit comprises an algorithm for computing the six-dimensional pose χ of the active capsule endoscope; the algorithm comprises the following steps: a) measuring the magnetic field B generated by the permanent magnets using the array of Hall sensors cap and representing the magnetic field as a stacked matrix B H : b) modeling the magnetic field of each Hall sensor in the array of Hall sensors as a stacked matrix B c (χ, P ij ) with respect to the six-dimensional pose χ and position P ij : c) measuring the orientation θ I , γ I of the active capsule endoscope with respect to a world coordinate system by a filtering algorithm; d) computing the six-dimensional pose χ by solving an optimization function θ≡θ I , γ≡γ I constructed under the condition that

[0043] In one embodiment, the B c (χ, P ij ) is selected from one of the following models: a magnetic dipole model, a magnetic multipole model, or a magnetic field fitting model in a mathematical paradigm.

[0044] In one embodiment, the filtering algorithm in step (c) is selected from one of the following: a Kalman filter algorithm, an extended Kalman filter algorithm, or a complementary filter algorithm.

[0045] In one embodiment, each of the plurality of magnetic field generators is controlled by an input current to generate the expected magnetic field, at a given position p in the expected magnetic field B, the magnetic force F exerted on the active capsule endoscope based on the input current c applied to the active capsule endoscope is determined according to the following formula: where m is the magnetic moment of the permanent magnet.

[0046] In one embodiment, the closed-loop position controller or the vision-based servo controller is one of the following: a proportional-integral-derivative controller, a fuzzy controller, an adaptive controller, a sliding mode controller, or a model predictive controller.

[0047] In one embodiment, the control input is constructed based on a kinematic analysis of the active capsule endoscope as follows: ​ wherein, wherein, Q capsule is the Jacobian matrix of the capsule; Q system is the Jacobian matrix of the system; and are the heading angles of the active capsule endoscope in the horizontal and vertical planes, respectively; R is a 2x2 rotation matrix related to the rotation angle of the heading of the active capsule endoscope; is a 3x3 rotation matrix of a rotation matrix from the world coordinate system to the camera coordinate system.

[0048] In one embodiment, the gastric landmarks include one or more of the cardia, the fundus, the body, the antrum, the angularis, and the pylorus.

[0049] In one embodiment, the pre-determined trajectory includes sequentially passing through the cardia, the fundus, the body, the antrum, the angularis, and the pylorus.

[0050] The present disclosure also provides a method of imaging one or more specific structures in the stomach using the system of the present disclosure. In one embodiment, the method includes the steps of: a) introducing the active capsule endoscope into the stomach; b) sending instructions from the control unit to the drive unit to generate the intended magnetic field to cause the active capsule endoscope to move along the pre-determined trajectory; and c) acquiring one or more images as the active capsule endoscope moves along the pre-determined trajectory in the stomach.

[0051] In one embodiment, the instructions in step (b) are generated by a method comprising the steps of: i) identifying a first position of the active capsule endoscope by analyzing one or more images acquired by the active capsule endoscope using the image module; ii) mapping the one or more images to the gastric landmarks; iii) obtaining the current six-dimensional pose of the active capsule endoscope; iv) calculating the deviation between the current six-dimensional pose and the pre-determined trajectory; and v) generating instructions for generating an intended magnetic field to cause the active capsule endoscope to move towards the pre-determined trajectory.

[0052] In one embodiment, the stomach is filled with liquid.

[0053] In one embodiment, the pre-determined trajectory sequentially passes through the cardia, the fundus, the body, the antrum, the angularis, and the pylorus.

[0054] In one embodiment, the method further includes, after step (c), expelling the active capsule endoscope from the subject’s body by the natural peristalsis of the gastrointestinal tract.

[0055] In one embodiment, the method further comprises, after step (c), performing an examination of the target region by manually controlling the active capsule endoscope.

[0056] An automatic gastric screening method is disclosed, which comprises a capsule endoscope that is swallowed by a subject and performs screening inside the stomach by a closed-loop magnetic driving method. Through the proposed screening process, the device can automatically achieve magnetic levitation and move to a specific position with the assistance of advanced position controllers and artificial intelligence technology, detect gastric regions with different characteristics, and then collect sufficient images of these gastric regions from the optimal viewing angle under the control of a visual servo controller. Throughout the screening procedure, the six-dimensional pose of the capsule endoscope is obtained by a positioning method that fuses the directional data output by the embedded IMU and the magnetic field data measured by the Hall sensor array, thereby achieving closed-loop motion control of the capsule endoscope and ensuring the accuracy and reliability of the gastric screening results. The automatic screening process aims to shorten the time of the gastric screening procedure. In addition, the main task of the doctor will focus on monitoring and diagnosing gastric abnormalities, thereby effectively reducing the workload of the doctor.

[0057] The present application adopts a magnetically driven capsule endoscope for automatic gastric screening, which has the following main advantages: (1) The automatic process can shorten the time of the gastric screening procedure, reduce the workload of the doctor, and ensure the accuracy and reliability of the screening results; (2) The introduction of artificial intelligence technology enables the capsule endoscope to detect different gastric regions, realize the construction of the entire gastric atlas, and assist the doctor in diagnosing gastric abnormalities; (3) Six-dimensional pose feedback achieves closed-loop motion control of the capsule endoscope, improving the efficiency of the screening procedure and the accuracy of the screening results; (4) Advanced controllers can adjust the position and direction of the capsule, enabling the capsule endoscope to efficiently and stably screen specific regions of the stomach.

[0058] Figure 1A system flowchart for automatic gastric screening using a capsule endoscope under closed-loop magnetic actuation is shown. The workflow of the present invention is as follows: first, the subject drinks a large amount of water before the screening procedure to fill the stomach and provide buoyancy for the capsule endoscope. Then, the subject swallows the capsule endoscope and lies on the bed in the actuation system. After starting the automatic screening process, the six-dimensional pose of the capsule endoscope is obtained through data fusion, which includes magnetic field data measured by a Hall sensor array and directional information obtained from the capsule inertial measurement unit (IMU). The actuation system generates a magnetic field to control the translation and orientation of the capsule endoscope, so that the capsule sequentially acquires images of the six main regions of the stomach detected by artificial intelligence detection technology. The six main regions of the stomach for sequential image acquisition include the cardia, fundus, body, antrum, angle, and pylorus. More than three images are acquired for each gastric region, and the direction of the capsule is adjusted to the optimal viewing angle through visual servoing control. Subsequently, after sufficient images are acquired to achieve the construction of the entire stomach atlas, the physician can manually control the capsule endoscope through the joystick on the control unit to perform detailed diagnosis on the target area. Finally, the capsule endoscope is naturally expelled from the body under the action of the gastrointestinal tract peristaltic wave.

[0059] Figure 2 A preferred embodiment of the magnetic actuation capsule endoscope device proposed by the present invention is shown. The device includes three subassemblies: a control system, an actuation system, and a capsule endoscope 1. The control unit includes a main control computer 12, two display screens 9 and 11 for displaying internal gastric detection images 10, a physician's chair 7, and a workbench 8. The actuation unit consists of a base 3, an upper electromagnet module 2, a subject bed 4, a lower electromagnet module, and a Hall sensor array 6. The control unit is used to process data (including images and IMU data from the capsule endoscope, magnetic field data from the Hall sensor array) and run related algorithms (including magnetic actuation algorithms, positioning algorithms, and artificial intelligence techniques), while the actuation unit is used to generate a magnetic field to control the motion of the capsule endoscope and collect data that can provide feedback information on the capsule's pose.

[0060] As Figure 3 shown, the capsule endoscope can perform real-time screening and diagnosis in the stomach under magnetic actuation. The capsule endoscope is mainly composed of a ring-shaped permanent magnet, multiple PCBs, and a shell for accommodating the above components. Typically, the PCBs include a camera module, an illumination module, a video transmission module, and a battery module. An IMU module is embedded inside the capsule to obtain directional information of the capsule endoscope and transmit directional data to external devices through Bluetooth technology.

[0061] Figure 4The magnet distribution of the drive system is shown. The drive system includes a cubic workspace 3-2 surrounded by upper magnet modules 3-1 and lower magnet modules 3-3, for accommodating the subject's body and performing screening under the action of the generated magnetic field.

[0062] Figure 5 (a)-(d) show several configuration examples of the upper or lower magnet modules. The configuration can be adjusted according to the actual application requirements. Each magnet is a cylindrical structure and is composed of a copper coil and a pure iron core. In a given static configuration, each magnet can generate a pre-calculated magnetic field in the entire workspace. At any given position p, the magnetic field generated by a given magnet can be represented as: c e is the current flowing through the magnet. Alternatively, the magnetic field generated by the given magnet can also be calculated using a magnetic dipole model or a multipole model. The superimposed magnetic field generated by multiple magnets at a given position p can be calculated by linear superposition and can be represented as: The magnetic force applied to the capsule endoscope is related to the magnetic field gradient and can be represented as: m is the magnetic moment of the embedded permanent magnet of the capsule endoscope. It can be more specifically represented as: Assuming that the capsule endoscope is not constrained in the liquid-filled stomach, the capsule direction (which can also be represented by the magnetic moment m) can be considered to always align with the magnetic field direction B(p). Therefore, by re-expressing the above equation, the magnetic field and magnetic force on the capsule endoscope can be represented as follows: For a certain expected magnetic field and magnetic force, the current required to be applied to each magnet can be solved by the following pseudo-inverse: Therefore, by adjusting the current applied to the magnets, the magnetic field and magnetic force applied to the capsule endoscope can be adjusted, and the position and direction of the capsule endoscope can be actively controlled.

[0063] Figure 6 is a positioning schematic diagram of the magnetic drive capsule endoscope device of the present application. The coordinate system is defined as follows: an orthogonal coordinate system (X, Y, Z) is defined on the printed circuit board on which the Hall sensor is installed, with the origin located on the printed circuit board, referred to as the H coordinate system; an orthogonal coordinate system (x, y, z) is defined on the capsule endoscope, with the origin set at the geometric center of the capsule endoscope, referred to as the C coordinate system; an orthogonal coordinate system (X ij ,Y ij ,Z ij), the origin is set at the geometric center of the ij Hall sensor, i and j represent the Hall sensor array The rows and columns in .

[0064] Figure 7 An example of a method for six-dimensional capsule pose estimation by constructing and solving an optimization problem is presented. According to one aspect of the present invention, the positioning system is capable of measuring the six degrees of freedom of the capsule endoscope (i.e., the position x, y, z and direction relative to the H coordinate system). θ,γ). For simplicity, the following definitions are derived: According to another aspect of the present invention, when the capsule endoscope works in the stomach environment, the permanent magnet inside the capsule endoscope generates a magnetic field B cap , the portion of the magnetic field that can be measured by the Hall sensor. The measurement result can be represented as a stacked matrix: in, Represents the magnetic field measured in the ij Hall sensor. On the other hand, the magnetic field modeled in each Hall sensor can be expressed as: Among them, P ij is the position of the Hall sensor relative to the H coordinate system, is the real magnetic field generated by the capsule endoscope and acting on the ij sensor, that is, It should be noted that B c (χ,P ij ) is related to the position and orientation of the capsule endoscope, as well as the relative position between the capsule endoscope and the ij Hall sensor. It is worth pointing out that B c (χ,P ij ) can be represented based on different models, including magnetic dipole model, magnetic multipole model and magnetic field fitting model in mathematical paradigm. In addition, the capsule endoscope contains an IMU, which can measure the direction of the capsule endoscope relative to the H coordinate system in real time through filtering algorithms such as Kalman filtering algorithm, extended Kalman filtering algorithm or complementary filtering algorithm. θ I ,γ I According to another aspect of the present invention, the position and orientation of the capsule endoscope can be calculated by constructing and solving the following optimization function: in, θ≡θ I ,γ≡γ I Thus, the position and direction of the capsule endoscope can be obtained.

[0065] Figure 8 Figure 1 is a block diagram showing an example of a capsule position controller using fuzzy logic. As an embodiment of the present invention, a fuzzy proportional method is designed to control the translation and levitation of a capsule endoscope. First, the position error is defined as the difference between the desired position and the current position: e = p p - p des . Then, e is transformed into discrete units to obtain the input quantity E of the fuzzy controller. Finally, E is converted into the correction value Δk c of the proportional controller parameter according to the fuzzy rules defined by the system. p The fuzzy sets of each fuzzy variable (E and Δk p ) are defined by triangular membership functions. Thus, the magnetic force can be expressed as: F m = K p e p where K p = K p + Δk p It is assumed that the orientation of the capsule endoscope is aligned with the direction of the applied magnetic field B m during the translation control process. Therefore, the current applied to the electromagnet can be calculated according to the desired magnetic field and magnetic force.

[0066] Figure 9 (a) shows the mapping relationship from the world coordinate system to the capsule coordinate system. The magnetic moment vector m c and the displacement vector r c represent the pose of the capsule endoscope. m c can be represented by two angles and in the two orthogonal planes.

[0067] Figure 9 (b) shows the Denavit-Hartenberg (DH) modeling method of the capsule coordinate system. By regarding the capsule as having two orthogonal rotational joints, the kinematics of the capsule endoscope can be analyzed using the three coordinate systems defined. The relevant transformation matrices are as follows: and where, and Therefore, the pose of the camera coordinate system relative to the world coordinate system can be represented by the following transformation matrix: where where, and are the rotation matrix and position vector, respectively. If the linear and angular velocities of the camera coordinate system are defined in the world coordinate system, the kinematic equation of the velocity level can be naturally obtained: wherein, and are the linear and angular velocities of the camera coordinate system, respectively; J capsule is the Jacobian matrix of the capsule.

[0068] Figure 10 A pinhole camera model for visual servoing control is shown. A point r(x, y, z) in space is projected to a point v(u, v) on the image plane. Since the magnetic driving system cannot control the rotation along the main axis of the capsule (i.e. the magnetization direction of the embedded magnet), but this rotation can occur in the process of controlling the direction of the capsule, the rotation angle γ needs to be considered in the visual servoing modeling process. The mapping relationship between the velocity of the point p(u, v) on the image plane and the instantaneous linear velocity and the angular velocity in the camera coordinate system is as follows: wherein, γ is the rotation angle along the main axis of the capsule, measured by the IMU embedded in the capsule, and the image Jacobian matrix is represented as: wherein, and Here, f and p are the focal length of the camera and the pixel width of each square pixel in the image, respectively. Due to the inherent error tolerance of visual servoing control, the depth z can be regarded as a constant. It should be noted that, and are defined in the camera coordinate system, so the following equation can be further derived: wherein, J system is the Jacobian matrix of the entire capsule endoscope system.

[0069] Figure 11 The block diagram of shows the process of adjusting the direction of the capsule endoscope using visual servoing control in order to collect images of the stomach from the optimal viewing angle. As an embodiment of the present application, a sliding mode controller is used to control the capsule endoscope based on the detected gastric markers. First, the sliding surface is defined as the pixel position error on the image plane: wherein, (u, v) and (u des ,v desare the current and desired positions of the target feature on the image plane, respectively. Then, their derivatives are calculated: Using the exponential approach law, the control input can be expressed as: where, is the pseudo-inverse of the system Jacobian matrix J system . E = diag(ε u , ε v ) and K = diag(κ u , κ v ) are two diagonal positive definite matrices used to adjust the convergence speed. By integrating the velocity and , the desired magnetic field and force can be calculated: where b is the magnetic field strength and f c is the magnetic force used to compensate the gravity of the capsule endoscope. Therefore, based on the desired magnetic field and force, the control current applied to the electromagnet can be calculated.

[0070] Figure 12 An example of controlling the capsule endoscope to move from a current position to a target position under the action of magnetic field and force is shown. During the movement, the desired magnetic field and force are calculated in real time based on the six-dimensional pose feedback of the capsule, enabling the capsule to perform translation and pose adjustment.

[0071] Figure 13 (a)-(f) show some examples of detecting the six main regions of the stomach using artificial intelligence detection technology. As an embodiment of the present application, the well-known learning-based target detection benchmark model Faster-RCNN is used to identify the stomach regions with specific features. The detection model can also be replaced by other detection technologies with similar performance. As an embodiment of the present application, the well-known learning-based target detection benchmark model Faster-RCNN is used to detect the stomach regions with specific features. The detection model can also be replaced by other detection technologies with similar performance.

[0072] Figure 14(a)-(f) show the process of automatic gastric screening using a magnetically actuated capsule endoscope. The screening covers six main regions of the stomach, and a map of the entire stomach is constructed, with the screening sequence being: cardia (a) -> fundus (b) -> corpus (c) -> antrum (d) -> angularis (e) -> pylorus (f). The present invention uses artificial intelligence to detect the above-mentioned gastric landmark regions and gastric abnormalities (such as polyps and ulcers). The present invention uses an advanced position controller to control the translation of the capsule endoscope, so that the capsule can move to different preset positions for diagnosis. The present invention also uses visual servoing control to adjust the direction of the capsule endoscope, so that the capsule can collect images of gastric landmarks from the optimal viewing angle. At least four images are collected for each gastric landmark to complete the mapping of the entire stomach.

[0073] Figure 15 (a)-(c) show the experimental setup of an embodiment of the present invention, which uses nine electromagnets to drive a capsule endoscope prototype for automatic gastric screening, including (a) an overall view of the system, including an electromagnet array, a capsule endoscope, and other related devices; (b) a close-up view of the capsule endoscope in the stomach phantom under the action of magnetic driving; (c) a camera view of the capsule endoscope detecting the target landmark.

[0074] Figure 16 (a)-(b) show the experimental results of visual servoing control using the prototype and controller of the above embodiment, which shows that the capsule endoscope can track the target feature to the expected position on the image plane. (a) Pixel trajectory of the target. (b) Corresponding pixel error.

[0075] Figure 17 (a)-(d) show some images collected during the automatic gastric screening test, which uses a position controller and a visual servoing controller to control the capsule endoscope to move to multiple expected positions to detect specific features and collect images from the optimal viewing angle. (a) Top view of the experimental setup. (b) Side view of the experimental setup. (c) Camera view of the capsule endoscope. (d) Four detected target features.

[0076] Figure 18 Show the experimental results of the automatic gastric screening bench test. (a) Capsule position control results. (b) Pixel error of four tracking target features.

Claims

1. A system for imaging one or more specific structures in the stomach within a working space, the system comprising: a. An active capsule endoscope that can be placed in the stomach, comprising a camera, an inertial measurement unit, and a permanent magnet; b. a drive unit comprising a Hall sensor array located at the bottom of the workspace and a plurality of magnetic field generators for generating a desired magnetic field; c. a control unit comprising an imaging module, a closed-loop position controller, and a vision-based servo controller; the control unit acquires images from the camera and orientation data from the inertial measurement unit; the control unit acquires magnetic field data of the expected magnetic field from the Hall sensor array; the control unit generates a control input to the drive unit to drive the active capsule endoscope to move along a preset trajectory in the stomach; The invention is characterized in that: the control unit uses the direction data and the magnetic field data to calculate the six-dimensional posture χ of the active capsule endoscope in real time; the six-dimensional posture χ is fed back to the closed-loop position controller, which controls the multiple magnetic field generators to perform translation and direction adjustment, thereby keeping the active capsule endoscope on the preset trajectory; The vision-based servo controller controls the multiple magnetic field generators based on the image and the six-dimensional posture χ to drive the active capsule endoscope to image the one or more specific structures, and the imaging module matches the image with a database containing gastric landmarks and general gastric three-dimensional structure to locate and image the one or more specific structures.

2. The system according to claim 1, wherein: The image module matches the images through artificial intelligence.

3. The system according to claim 1, wherein: The system further includes a dynamic trajectory generator for guiding the active capsule endoscope to move along the preset trajectory.

4. The system according to claim 1, wherein: The active capsule endoscope further includes one or more lighting units or data transmission units.

5. The system according to claim 1, wherein: The Hall sensor array includes one or more three-axis Hall sensors, and the three-axis Hall sensors are distributed in the three-dimensional space of the working space according to the expected magnetic field.

6. The system according to claim 5, characterized in that: Each of the one or more three-axis Hall sensors has a different orientation relative to an orthogonal coordinate system (X, Y, Z) of the expected magnetic field.

7. The system according to claim 1, wherein: The control unit includes an algorithm for calculating the six-dimensional posture χ of the active capsule endoscope; the algorithm includes the following steps: a. Use the Hall sensor array to measure the magnetic field B generated by the permanent magnet cap , and represent the magnetic field as a stacking matrix B H : b. Model the magnetic field of each Hall sensor in the Hall sensor array into a stacked matrix B c (χ,P ij ), the six-dimensional posture χ and position P relative to the ij Hall sensor ij : c. Measuring the direction of the active capsule endoscope relative to a world coordinate system through a filtering algorithm θ I ,γ I ; d. By solving θ≡θ I ,γ≡γ I The optimization function constructed under the conditions Calculate the six-dimensional pose χ.

8. The system according to claim 7, characterized in that: The B c (χ,P ij ) is selected from one of the following models: magnetic dipole model, magnetic multipole model or magnetic field fitting model in mathematical paradigm.

9. The system according to claim 7, characterized in that: The filtering algorithm in step (c) is selected from a Kalman filtering algorithm, an extended Kalman filtering algorithm or a complementary filtering algorithm.

10. The system according to claim 1, wherein: Each of the plurality of magnetic field generators is controlled by an input current to generate the desired magnetic field. At a given position p in the desired magnetic field B, the magnetic force F applied to the active capsule endoscope based on the input current c is determined according to the following formula: Wherein, m is the magnetic moment of the permanent magnet.

11. The system according to claim 1, wherein: The closed-loop position controller or the vision-based servo controller is one of the following controllers: a proportional-integral-derivative controller, a fuzzy controller, an adaptive controller, a sliding mode controller, or a model predictive controller.

12. The system according to claim 1, wherein: The control input is constructed based on the kinematic analysis of the active capsule endoscope as follows: in, Among them, Q capsule is the Jacobian matrix of the capsule; Q system is the Jacobian matrix of the system; and are the heading angles of the active capsule endoscope in the horizontal and vertical planes, respectively; R is a 2×2 rotation matrix related to the rotation angle of the heading of the active capsule endoscope; A 3×3 rotation matrix that transforms the world coordinate system into the camera coordinate system.

13. The system according to claim 1, wherein: The gastric markers include one or more of the cardia, gastric fundus, gastric body, gastric antrum, gastric angle and pylorus.

14. The system according to claim 1, wherein: The preset trajectory includes sequentially passing through the cardia, gastric fundus, gastric body, gastric antrum, gastric angle and pylorus.

15. A method for imaging one or more specific structures in the stomach using the system of claim 1, the method comprising the steps of: a. introducing the active capsule endoscope into the stomach; b. Sending an instruction from the control unit to the drive unit to generate the desired magnetic field so that the active capsule endoscope moves according to the preset trajectory; as well as c. When the active capsule endoscope moves along the preset trajectory in the stomach, one or more images are collected.

16. The method according to claim 15, characterized in that: The instructions in step (b) are generated by a method comprising the following steps: i. analyzing one or more images collected by the active capsule endoscope using the image module to identify a first position of the active capsule endoscope; ii. mapping the one or more images to the stomach landmarks; iii. Obtaining the current six-dimensional posture of the active capsule endoscope; iv. Calculating the deviation between the current six-dimensional posture and the preset trajectory; as well as v. generating an instruction for generating a desired magnetic field to move the active capsule endoscope toward the preset trajectory.

17. The method according to claim 15, characterized in that: The stomach is filled with fluid.

18. The method according to claim 15, wherein: The preset trajectory passes through the cardia, gastric fundus, gastric body, gastric antrum, gastric angle and pylorus in sequence.

19. The method according to claim 15, wherein: The method further comprises, after step (c), expelling the active capsule endoscope from the subject's body through natural peristalsis of the gastrointestinal tract.

20. The method according to claim 15, wherein: The method further includes inspecting a target area by manually controlling the active capsule endoscope after step (c).

Images