Automatic navigation method, device and navigation system of capsule endoscope
Through the combination of capsule depth estimation network and attitude data, the movement direction of the capsule endoscope is automatically adjusted, which solves the problem of difficult direction control of the capsule endoscope during movement, and improves diagnostic efficiency and control accuracy.
Patent Information
- Application Number
- CN202410924184.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
During the exercise, existing capsule endoscopes are difficult for doctors to accurately control their movement direction due to their own rotation during exercise, resulting in low diagnostic efficiency and poor control effect.
The capsule depth estimation network is used to process the real-time image data, generate a depth map, and determine the driving strategy based on the attitude data, and automatically adjust the movement direction of the capsule endoscope through an in vitro magnetic source.
Automatic navigation of capsule endoscope is realized, which reduces operation difficulty, improves movement efficiency, ensures consistency between the capsule orientation and the desired direction, and reduces the impact of spin on the travel process.
Smart Images

Figure CN118948184A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application belong to the field of computer-assisted medical technology, and in particular, relate to an automatic navigation method, device and navigation system for a capsule endoscope. Background Art
[0002] Capsule endoscopes can be swallowed into the human digestive tract, collect images of the digestive tract and send them to external devices for doctors to diagnose diseases.
[0003] Capsule endoscopes mainly include passive and active types. Among them, the passive capsule endoscope moves with the peristalsis of the human stomach and intestines, and takes images of the inside of the digestive tract during the movement for doctors to diagnose. This diagnosis and treatment method of randomly taking images of the digestive tract is prone to missing lesions, and because the movement of the capsule endoscope is uncontrolled, doctors cannot make detailed observations of the areas of interest inside the digestive tract.
[0004] Active capsule endoscopes are controlled by doctors using operating devices such as external handles. Doctors use operating devices to control the movement and rotation of the external magnetic source, thereby controlling the capsule endoscope to move in different directions, capture images of different locations inside the digestive tract, and achieve observation and diagnosis of specific locations. However, the capsule endoscope itself rotates during movement. In the process of controlling the movement of the capsule endoscope by manual operation in the existing technology, due to the existence of the capsule's self-spin, it is difficult for doctors to determine the exact direction of travel based on the captured images, and it is difficult to control the capsule endoscope to move in the direction desired by the doctor, resulting in poor control effect and low efficiency, which requires certain operating experience to complete. Summary of the invention
[0005] In view of this, an embodiment of the present application provides an automatic navigation method, device and navigation system for a capsule endoscope, which are used to automatically determine the accurate direction of travel according to the images captured by the capsule endoscope, thereby realizing automatic navigation of the movement of the capsule endoscope.
[0006] A first aspect of an embodiment of the present application provides an automatic navigation method for a capsule endoscope, comprising:
[0007] receiving real-time data transmitted by a capsule endoscope, wherein the real-time data includes image data captured by the capsule endoscope and posture data of the capsule endoscope;
[0008] Processing the image data using a capsule depth estimation network to output a capsule depth map corresponding to the image data;
[0009] Determining a current driving strategy of the capsule endoscope according to the capsule depth map and the posture data;
[0010] The capsule endoscope is driven to move automatically according to the driving strategy.
[0011] A second aspect of an embodiment of the present application provides an automatic navigation device for a capsule endoscope, comprising:
[0012] A receiving module, used for receiving real-time data transmitted by a capsule endoscope, wherein the real-time data includes image data captured by the capsule endoscope and posture data of the capsule endoscope;
[0013] A processing module, used to process the image data using a capsule depth estimation network and output a capsule depth map corresponding to the image data;
[0014] A determination module, used for determining a current driving strategy of the capsule endoscope according to the capsule depth map and the posture data;
[0015] A driving module is used to drive the capsule endoscope to move automatically according to the driving strategy.
[0016] A third aspect of the embodiments of the present application provides a navigation system for a capsule endoscope, including a display device, an external magnetic source, and a navigation unit; wherein:
[0017] a display device, configured to receive image data transmitted by the capsule endoscope and display the image data;
[0018] A navigation unit is used to receive the posture data transmitted by the capsule endoscope; process the image data using a capsule depth estimation network to output a capsule depth map corresponding to the image data; determine the current driving strategy of the capsule endoscope according to the capsule depth map and the posture data, and control an external magnetic source according to the driving strategy to drive the capsule endoscope to move automatically.
[0019] A fourth aspect of an embodiment of the present application provides an automatic navigation device for a capsule endoscope, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the automatic navigation method for the capsule endoscope as described in the first aspect above is implemented.
[0020] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the automatic navigation method of a capsule endoscope as described in the first aspect above is implemented.
[0021] A sixth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a computer, the computer executes the automatic navigation method of the capsule endoscope described in the first aspect.
[0022] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0023] In the embodiment of the present application, the image data and posture data collected by the capsule endoscope can be transmitted to the body in real time. The external device such as the software host computer or the navigation unit can process the image data and output the capsule depth map corresponding to the image data. In combination with the capsule depth map and the real-time posture data, the external device can determine the current orientation of the capsule and the difference between the orientation and the desired direction of travel, and then determine the driving strategy that needs to improve the difference. In the actual working process of the capsule, due to its own rotation, the actual orientation may be inconsistent with the desired direction at certain moments or for a long time. The embodiment of the present application can accurately determine the difference between the actual orientation and the desired direction by processing the real-time image data and posture data, and then adjust the corresponding driving strategy so that the capsule orientation is as consistent as possible with the desired direction of travel, thereby reducing the impact of the capsule spin on the travel process, achieving the purpose of automatic navigation of the capsule, reducing the difficulty of the operator manually controlling the capsule movement, and improving the efficiency of capsule movement and inspection. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0025] Figure 1 is a schematic diagram of a capsule endoscope navigation system provided in an embodiment of the present application;
[0026] Figure 2 is a schematic diagram of an automatic navigation method for a capsule endoscope provided in an embodiment of the present application;
[0027] Figure 3 It is a schematic diagram of a possible implementation of S203 in an automatic navigation method for a capsule endoscope provided in an embodiment of the present application;
[0028] Figure 4 It is a schematic diagram of the overall process of an automatic navigation method for a capsule endoscope provided in an embodiment of the present application;
[0029] Figure 5 It is a schematic diagram of a magnetic source driving process in an automatic navigation method of a capsule endoscope provided in an embodiment of the present application;
[0030] Figure 6is a schematic diagram of an automatic navigation device for a capsule endoscope provided in an embodiment of the present application;
[0031] Figure 7 It is a schematic diagram of a navigation device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.
[0033] The technical solution of the present application is described below through specific embodiments.
[0034] Reference Figure 1 , showing a schematic diagram of a capsule endoscope navigation system provided in an embodiment of the present application, the system may include Figure 1 The system can be used to locate the capsule 101, the capsule positioning module 102, the external magnetic source 104 and the navigation unit (not shown in the figure). Figure 1 The capsule endoscope 103 in the apparatus is navigated, and the navigation process may include the navigation of the capsule endoscope 103 under the manual operation of an operator such as a doctor, and the automatic navigation of the capsule endoscope 103 based on a corresponding algorithm.
[0035] Specifically, the display device 101 is used to receive image data transmitted by the capsule endoscope transmission 103 and display the image data.
[0036] The navigation unit is used to receive the posture data transmitted by the capsule endoscope 103; process the image data using the capsule depth estimation network and output a capsule depth map corresponding to the image data; determine the driving strategy of the capsule endoscope 103 according to the capsule depth map and the current posture data; and control the extracorporeal magnetic source 104 according to the driving strategy to drive the capsule endoscope 103 to move automatically.
[0037] In a possible implementation of the embodiment of the present application, the display device 101 and the navigation unit can be integrated into the same device. Exemplarily, the display device 101 and the navigation unit can be components of an integrated computer device, and the display screen of the computer device can be used as the display device 101 to display the image captured by the capsule endoscope 103; the navigation unit can be a control unit in the computer device or other units or modules that can implement corresponding functions. The navigation unit can be used to implement various algorithms, such as image data processing, vector calculation, output of control instructions, etc. In addition, in addition to the navigation unit integrated in the computer device, the above-mentioned navigation system may also include other auxiliary operating components, such as for controlling Figure 1 The mechanical arm of the extracorporeal magnetic source 104 (not shown in the figure), etc., is not limited to this embodiment of the present application.
[0038] like Figure 1 As shown, the capsule endoscope 103 may include a wireless module 1031, a capsule magnet 1032, an inertial measurement module 1033 and a data acquisition module 1034. Among them, the wireless module 1031 can be used to send the data collected by the data acquisition module 1034 to the control unit, and the data collected by the data acquisition module 1034 may include the captured image, its own orientation data, etc. The capsule magnet 1032 can generate a force to drive the capsule endoscope 103 to move under the action of the external magnetic source 104, thereby driving the capsule endoscope 103 to rotate or move. The inertial measurement module 1033 can be used to measure the movement direction, posture and other parameters of the capsule endoscope 103 in the human digestive tract, and transmit them to the external navigation unit through the wireless module 1031.
[0039] In a possible implementation of the embodiment of the present application, the data acquisition module 1034 may be an image acquisition module. The image acquisition module in the capsule endoscope 103 may capture images of the human digestive tract during the capsule operation to obtain image data in the digestive tract. The image data and posture data required to implement the method of the embodiment of the present application may be acquired by the image acquisition module and Figure 1 The inertial measurement module in the apparatus collects the information and then transmits it to the navigation unit outside the body through the wireless module 1031.
[0040] In another possible implementation of the embodiment of the present application, the data acquisition module 1034 may integrate image acquisition and posture data acquisition functions. That is, Figure 1 The data acquisition module 1034 in the embodiment can realize image acquisition, and Figure 1 In this implementation, the data acquisition module 1034 and the inertial measurement module 1033 can be integrated into one.
[0041] like Figure 1 As shown, the image data of the human digestive tract collected by the data acquisition module 1034 can be sent to the outside of the body through the wireless module 1031 and displayed on the display device 101 outside the body. In addition, the posture data of the capsule collected by the inertial measurement module 1033 can also be sent to the outside of the body through the wireless module 1031 and displayed on the display device 101. Among them, the posture data of the capsule can be superimposed on the image data for display. That is, after displaying the image data captured by the capsule, the display device 101 superimposes the posture data of the capsule on the display screen, and the above-mentioned posture data can be displayed by indicating arrows, directions, etc., to indicate the current posture of the capsule.
[0042] In the embodiments of the present application, Figure 1 The navigation system shown can realize automatic navigation of the capsule endoscope 103 in the automatic navigation mode. That is, the navigation unit can process the image data collected by the data acquisition module 1034 and output the corresponding capsule depth map. On this basis, the navigation unit can combine the capsule depth map and the posture data collected by the inertial measurement module 1033, and use the corresponding algorithm to determine the driving strategy required for realizing automatic navigation of the capsule. The navigation unit can generate corresponding instructions and transmit them to the external magnetic source 104 according to the control method required by the driving strategy, so as to drive the movement of the capsule endoscope 103 in the body by controlling the rotation or movement of the external magnetic source 104, and realize the automatic movement of the capsule endoscope 103 in the human digestive tract in the automatic navigation mode. The use of the above-mentioned navigation system to drive the capsule endoscope 103 to move automatically can reduce the impact of the capsule's own rotation on its movement process, reduce the operation intensity of operators such as doctors, and improve the efficiency of capsule control.
[0043] In the embodiment of the present application, the position information of the capsule endoscope 103 required for the aforementioned automatic navigation can be obtained by Figure 1 The capsule positioning module 102 in the apparatus can be obtained, and the capsule positioning module 102 can be a magnetic field array installed under the examination bed. When the patient swallows the capsule endoscope 102 and lies on the examination bed, the capsule positioning module 102 can locate the position of the capsule endoscope 103 in the digestive tract. The position information obtained in the above positioning process can also be used to calibrate the inertial measurement module 1033 to improve the accuracy of the posture data subsequently collected by the inertial measurement module 1033.
[0044] The automatic navigation method of the capsule endoscope provided in the embodiment of the present application can be Figure 1 The method is implemented in the navigation system shown in the figure. The following is a detailed introduction of the method in combination with the detailed steps. Unless otherwise specified, the capsule, endoscope, and capsule endoscope referred to in the embodiments of the present application all refer to the following. Figure 1 A capsule endoscope 103 is shown.
[0045] Reference Figure 2 , shows a schematic diagram of an automatic navigation method for a capsule endoscope provided in an embodiment of the present application, which may specifically include the following steps:
[0046] S201. Receive real-time data transmitted by a capsule endoscope, where the real-time data includes image data captured by the capsule endoscope and posture data of the capsule endoscope.
[0047] It should be noted that this method can be applied to Figure 1 Specifically, each step of the method can be performed by a navigation unit in the above navigation system. That is, Figure 1 The navigation unit in the navigation system shown can realize automatic navigation of the capsule endoscope by executing each step of the method.
[0048] In a possible implementation of the embodiment of the present application, the navigation unit may exist in the form of a software host computer. That is, the software host computer in the navigation system may serve as the navigation unit to implement each step in the method.
[0049] In the embodiment of the present application, the capsule endoscope can move in a working environment and take images of the environment during operation. The working environment can be an internal environment of a human body to be examined using the capsule endoscope, such as the human digestive tract. For example, a doctor can use the capsule endoscope to examine the human digestive tract. During this process, the capsule endoscope can Figure 1 The capsule endoscope moves in the digestive tract under the control of the navigation system shown, and takes pictures to obtain image data in the digestive tract. The real-time data transmitted by the capsule endoscope to the external software host computer through the wireless module may include the above image data.
[0050] In addition, the inertial measurement module in the capsule endoscope can collect the current posture data of the capsule in real time, and the posture data can also be transmitted to the software host computer through the wireless module inside the capsule. The software host computer can process the image data and posture data according to this method to realize automatic navigation of the capsule working process.
[0051] In a possible implementation of the embodiment of the present application, before the capsule endoscope works, an in vitro positioning module may be used, such as Figure 1 The positioning information of the capsule positioning module 102 shown in is used to calibrate the inertial measurement module. After calibration, the inertial measurement module can collect the posture data of the capsule and transmit it to an external software host computer.
[0052] In an embodiment of the present application, the calibration of the inertial measurement module inside the capsule endoscope can be performed when the navigation system is started. Specifically, when the navigation system is started, the capsule that has entered the patient's body can be positioned by the capsule positioning module outside the body to obtain corresponding posture information. The posture information can include not only the current direction vector of the capsule endoscope itself, but also the posture information of the inertial measurement module in the capsule endoscope. Among them, the direction vector of the capsule endoscope can be used to indicate the current direction of the capsule endoscope, and the posture information of the inertial measurement module can be the quaternion posture of the inertial measurement module represented by quaternion.
[0053] Then, the capsule endoscope can rotate the unit vector corresponding to the current direction vector of the capsule endoscope based on the quaternion attitude of the inertial measurement module. For example, the unit vector corresponding to the current direction vector of the capsule endoscope can be expressed as (0, 0, 1). During the calibration process, the unit vector can be rotated using the quaternion representation.
[0054] After rotating the unit vector represented by quaternion, the angle between the rotated unit vector and the current direction vector of the capsule endoscope is calculated. This angle is the calibration value, which can be used to compensate the data in subsequent coordinate conversion and other processing processes to achieve calibration of the inertial measurement module.
[0055] After the calibration of the inertial measurement module is completed, the data acquisition module in the capsule endoscope can perform real-time image acquisition, and the inertial measurement module can also collect the capsule's posture data in real time. The acquired image data and posture data can be transmitted to the software host computer through the wireless module and displayed in real time on the display device outside the body.
[0056] In a possible implementation of the embodiment of the present application, the posture data may represent the posture of the capsule endoscope in the body. When the software host computer displays the image data captured by the capsule through a display device, the posture may be synchronously displayed on the image, so that doctors and other operators can intuitively understand the current posture of the capsule.
[0057] As an example of an embodiment of the present application, when displaying the image data captured by the capsule, the software host computer can simulate the environment in which the capsule is located in the body, and the above environment simulated by the software host computer can be simply expressed as the orientation of the capsule during its travel. For example, it can be presented in the display device in the form of a coordinate axis or an orientation arrow. Then, the software host computer can adjust the orientation of the image data in the simulated environment (or orientation) according to the relative positional relationship between the captured image and the environment (or the orientation presented in the display device), thereby achieving the effect of displaying the posture data on the captured image data. The embodiment of the present application is not limited to this.
[0058] S202: Process the image data using a capsule depth estimation network, and output a capsule depth map corresponding to the image data.
[0059] In the embodiment of the present application, the software host computer can use the trained capsule depth estimation network to process the image data and output a capsule depth map, which can be used to reflect the depth of each pixel in the image captured by the capsule. For example, a depth value can be used to represent the depth of each pixel.
[0060] In a possible implementation manner of an embodiment of the present application, the capsule depth estimation network can be implemented using a network model including a teacher model and a student model.
[0061] The capsule depth estimation network in the embodiment of the present application may include an encoder and a decoder, using a visual transformer (ViT) or convolution as the basic computational building block of the encoder, and the decoder can aggregate features from the encoder and convert them into final depth predictions. Visual Transformer is a self-attention-based neural network architecture that can be used to process computer vision tasks and can achieve better task results in tasks such as image classification, object detection, and image segmentation.
[0062] The capsule depth estimation network in the embodiment of the present application can use capsule endoscope sample images labeled with depth information and those not labeled with depth information for model training, and use pre-trained ViT or convolutional model weights to initialize the encoder to generate a network model that can ultimately be used for depth estimation.
[0063] Specifically, in the model training stage, a teacher model can be trained using capsule endoscope sample images annotated with depth information, and then the teacher model can be used to assign pseudo-depth annotations to capsule endoscope sample images that are not annotated with depth information. In this way, the data coverage can be significantly expanded. On this basis, the student model can be jointly trained using capsule endoscope sample images annotated with depth information and pseudo-depth annotation sets. The trained student model can be used as a capsule depth estimation network to perform depth prediction on the image data captured by the capsule endoscope in the subsequent processing process to obtain the depth value of each pixel in the image data.
[0064] That is, after receiving the image data transmitted by the capsule through the wireless module, the software host computer can use the trained student model to predict the depth value of each pixel in the image data and output the capsule depth map corresponding to the image data. The depth map can record the depth value of each pixel in the image.
[0065] S203: Determine a current driving strategy of the capsule endoscope according to the capsule depth map and the posture data.
[0066] In the embodiment of the present application, the software host computer can determine a driving strategy based on the processed capsule depth map and the capsule attitude data collected by the inertial measurement module. The driving strategy can be a strategy for driving the capsule to move automatically in the automatic navigation mode based on the current working state of the capsule.
[0067] In an embodiment of the present application, the driving strategy can be a kind of instruction information, and the external magnetic source can rotate or move accordingly under the instruction of the instruction information, so as to generate a magnetic field to act on the capsule endoscope in the body, so that the capsule endoscope can move in a manner compatible with the driving strategy. In one example, the above process of determining the driving strategy based on the capsule depth map and posture data can be implemented by a software host computer and sent to a magnetic source driving module, and the magnetic source driving module drives the external magnetic source according to the received driving strategy. In another example, the above process of determining the driving strategy can also be directly implemented by a magnetic source driving module, and the magnetic source driving module can process according to the corresponding algorithm based on the capsule depth map and posture data, determine the driving strategy for the capsule, and drive the external magnetic source according to the driving strategy. The above magnetic source driving module can be configured in the software host computer.
[0068] In a possible implementation of the embodiment of the present application, as Figure 3 As shown, in S203, the current driving strategy of the capsule endoscope is determined according to the capsule depth map and the posture data, which may specifically include the following steps S2031-S2034:
[0069] S2031 . Convert the capsule depth map into a binary image according to the depth value of each pixel in the capsule depth map.
[0070] In the embodiment of the present application, the binarization processing of the capsule depth map by the software host computer may refer to processing the depth value of each pixel in the capsule depth map so that the depth value of each pixel after processing is converted into a fixed first value or a second value.
[0071] In a possible implementation of the embodiment of the present application, before converting the capsule depth map into a binary image, the software host computer may normalize the depth value of each pixel in the capsule depth map, thereby processing the depth value of each pixel to a preset value range. For example, the depth value of each pixel is processed to between [0, 255].
[0072] Then, the software host computer can convert the depth value of each pixel into a first value or a second value based on the size relationship between the depth value of each pixel after processing and the preset threshold value, and obtain a binary image corresponding to the capsule depth map. The above-mentioned first value and second value can be the upper and lower limits of the interval in the above-mentioned normalization process. For example, the first value can be the lower limit of the interval, that is, the first value is 0; the second value can be the upper limit of the interval, that is, the second value is 255; or, in the above-mentioned example, the first value can be 255 and the second value can be 0, which is not limited in the embodiment of the present application.
[0073] In a possible implementation of the embodiment of the present application, the software host computer may use the following formula to normalize the capsule depth map:
[0074] depth=(depth-depth.min()) / (depth.max()-depth.min())*255.0
[0075] In the above formula, depth represents the depth value, depth.min represents the minimum value of the depth values of each pixel in the depth map, and depth.max represents the maximum value of the depth values of each pixel in the depth map.
[0076] After completing the normalization process, the software host computer can compare the depth value of each pixel with the threshold. Then, the pixel value greater than the threshold is converted to the first value, and the pixel value less than or equal to the threshold is converted to the second value. For example, the preset threshold can be 5, and among the depth values of each pixel after the normalization process, the pixel value less than or equal to 5 can be reset to 255, and the pixel value greater than 5 can be reset to 0.
[0077] S2032: performing contour search in the binary image, and determining the center coordinates of the advancing direction of the capsule endoscope according to the found contour.
[0078] In the embodiment of the present application, the contour in the binary image may be an image contour formed by pixels having a pixel value of 255. The software host computer may use the cv2.findContours function in the cross-platform computer vision library OpenCV to find the contour in the above binary image.
[0079] On this basis, the software host computer can determine the center coordinates of the capsule endoscope's advancing direction according to the found contour.
[0080] Specifically, the software host can calculate the moment of the contour and determine the center coordinates of the capsule's forward direction through the moment wallpaper. The moment of the contour represents the global characteristics of a contour, an image, or a set of points. The moment information contains different types of geometric features of the corresponding object, such as size, position, angle, shape, etc.
[0081] In the field of image processing and computer vision, the OpenCV library provides the cv2.moments() function to calculate the moments of an image or contour, and then analyze and describe various properties of the object. Therefore, the software host computer can use the cv2.moments() function to calculate the moments of the contour, that is: M = cv2.moments(contours[0]).
[0082] After the moment is calculated, the software host computer can determine the center coordinates (x, y) of the capsule endoscope's advancing direction through the moment. The center coordinates can be the coordinates of the contour centroid, that is, the coordinates of the contour geometric center point.
[0083] S2033. Determine the current orientation of the capsule endoscope and the forward direction based on the center coordinates of the forward direction and the posture data.
[0084] After determining the center coordinates of the capsule's forward direction, the software host computer can transform the center coordinates of the forward direction in combination with the posture data collected by the inertial measurement module to obtain two vectors of the XY plane in the fixed coordinate system. These two vectors can respectively represent the current orientation and forward direction of the capsule endoscope. The fixed coordinate system can be a magnetic positioning coordinate system, that is, a coordinate system established based on an external positioning module, and the origin of the coordinate system is the center point of the magnetic positioning module.
[0085] In a possible implementation of the embodiment of the present application, after calculating the current orientation and the forward direction of the capsule, the image data displayed in the software host computer can synchronously display these two vectors to indicate the difference between the actual orientation and the expected direction of the capsule during its current movement.
[0086] S2034: Determine a current driving strategy of the capsule endoscope according to the orientation and the forward direction.
[0087] The two vectors calculated in the above steps can be used to represent the difference between the actual direction and the expected direction of the capsule during its current movement. The purpose of the embodiment of the present application is to make the current direction of the capsule as consistent as possible with the expected direction through the control of the driving strategy, thereby achieving the effect of automatic navigation.
[0088] Therefore, the software host computer can determine the current driving strategy of the capsule endoscope according to the orientation and the forward direction.
[0089] In a possible implementation of an embodiment of the present application, when determining the current driving strategy of the capsule endoscope according to the orientation and the forward direction, the software host computer can perform a dot product operation on the two vectors to obtain the angle between the two vectors. If the angle is less than or equal to the angle threshold, the software host computer can determine that the current driving strategy of the capsule endoscope is forward driving. The above-mentioned angle can represent the difference between the capsule orientation and the forward direction. If the angle is less than the angle threshold, it can be said that the difference between the capsule orientation and the forward direction is small, and the capsule can move according to the current orientation, so the corresponding driving strategy can be forward driving. If the angle is greater than the above-mentioned angle threshold, it can be said that the difference between the capsule orientation and the forward direction is large, and the capsule orientation needs to be adjusted to make it as consistent as possible with the desired forward direction. At this time, the software host computer can perform a cross product operation on the two vectors, and determine the current driving strategy of the capsule endoscope according to the result of the cross product operation.
[0090] Specifically, if the result of the cross product operation is greater than zero, it can be determined that the current driving strategy of the capsule endoscope is rightward driving; if the result of the cross product operation is less than or equal to zero, it can be determined that the current driving strategy of the capsule endoscope is leftward driving. The above rightward driving and leftward driving can be determined based on the current orientation of the capsule, that is, rightward driving is to drive the capsule to move to the right of the direction indicated by the vector corresponding to the current orientation, and leftward driving is to drive the capsule to move to the left of the direction indicated by the vector corresponding to the current orientation.
[0091] S204: Drive the capsule endoscope to move automatically according to the driving strategy.
[0092] In the embodiment of the present application, the driving strategy can be executed by a device or module capable of driving an external magnetic source. For example, the software host computer or the magnetic source driving module in the software host computer can adjust the driving mode of the external magnetic source according to the determined driving strategy, so that the magnetic field generated by the external magnetic source can drive the capsule endoscope to automatically move in the body.
[0093] In an embodiment of the present application, the image data and posture data collected by the capsule endoscope can be transmitted to the body in real time. An external device such as a software host computer or a navigation unit can process the image data and output a capsule depth map corresponding to the image data. In combination with the capsule depth map and real-time posture data, the external device can determine the current orientation of the capsule and the difference between the orientation and the desired direction of travel, and then determine the driving strategy that needs to improve the difference. In the actual working process of the capsule, due to its own rotation, the actual orientation may be inconsistent with the desired direction at certain moments or for a long time. By processing the real-time image data and posture data, the embodiment of the present application can accurately determine the difference between the actual orientation and the desired direction, and then adjust the corresponding driving strategy so that the capsule orientation is as consistent as possible with the desired direction of travel, thereby reducing the impact of the capsule spin on the travel process, achieving the purpose of automatic navigation of the capsule, reducing the difficulty of the operator manually controlling the capsule movement, and improving the efficiency of capsule movement and inspection.
[0094] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0095] For ease of understanding, the automatic navigation method of the capsule endoscope provided in the embodiment of the present application is introduced below with reference to a specific example.
[0096] like Figure 4 FIG. 1 is a schematic diagram of the overall process of an automatic navigation method for a capsule endoscope provided in an embodiment of the present application. Figure 4 In the process shown, after starting the capsule, the inertial measurement module in the capsule can be calibrated first. How to calibrate the inertial measurement module can be referred to the introduction of S201 in the above method embodiment, which will not be repeated here.
[0097] After the calibration of the inertial measurement module is completed, the capsule endoscope can collect image data and send the posture data to the software host computer outside the body, which can display the image data on the display device. In addition, information reflecting the capsule posture data can also be displayed synchronously on the image data.
[0098] In this step, the image data can be acquired by the data acquisition module in the capsule endoscope, and the posture data can be acquired by the calibrated inertial measurement module. The image data and posture data can be transmitted to the software host computer outside the body through the wireless module in the capsule endoscope.
[0099] In a possible implementation of the embodiment of the present application, the automatic navigation of the capsule endoscope may be implemented when the capsule is in the automatic navigation mode. Figure 4 As shown, the software host computer can determine whether the current capsule is in the automatic navigation mode. If the capsule is not in the automatic navigation mode, the doctor or other operators can control the rotation and / or movement of the external magnetic source through manual operation according to the image data and information reflecting the posture data displayed in the display device, so as to drive the movement of the capsule in the body.
[0100] If the capsule is in automatic navigation mode, the software host computer can perform automatic navigation control based on the built-in relevant algorithms. Figure 4 As shown, the software host computer can use the trained capsule depth estimation network to process the captured image data to generate a capsule depth map. The process of training the capsule depth estimation network and how to use the network to generate a capsule depth map can be referred to the introduction of S202 in the above method embodiment, which will not be repeated here.
[0101] like Figure 4 As shown in FIG. 1 , after generating the capsule depth map, the software host computer can determine the current driving strategy by combining the depth map and the capsule posture data. The driving strategy can be used to instruct the magnetic source driving module to drive the external magnetic source in a manner that is compatible with the driving strategy, thereby driving the capsule to move automatically.
[0102] like Figure 5 , which is a schematic diagram of a magnetic source driving process in an automatic navigation method of a capsule endoscope provided in an embodiment of the present application. After generating a capsule depth map, the software host computer can normalize and binarize it to obtain a binary image. Then, combined with the binary image, the software host computer can calculate a vector representing the current orientation and forward direction of the capsule endoscope. By operating on the above two vectors, a specific driving strategy can be determined.
[0103] Specifically, see Figure 5 , after calculating the above two vectors, the software host computer can calculate the angle between the two vectors, and the angle can be obtained by performing a dot product on the two vectors. If the calculated angle is less than or equal to the angle threshold, the software host computer can determine that the current driving strategy of the capsule endoscope is forward driving, and the software host computer can automatically control the extracorporeal magnetic source to drive the capsule endoscope forward. If the calculated angle is greater than the angle threshold, the software host computer can perform a cross product operation on the two vectors to determine the size relationship between the vector cross product and 0, and thus determine the current driving strategy of the capsule endoscope based on the result of the cross product operation. Figure 5As shown, if the result of the cross product operation is greater than zero, it can be determined that the current driving strategy of the capsule endoscope is driving to the right, and the software host computer can automatically control the extracorporeal magnetic source to drive the capsule endoscope to move to the right; if the result of the cross product operation is less than or equal to zero, it can be determined that the current driving strategy of the capsule endoscope is driving to the left, and the software host computer can automatically control the extracorporeal magnetic source to drive the capsule endoscope to move to the left.
[0104] Reference Figure 6 , shows a schematic diagram of an automatic navigation device for a capsule endoscope provided in an embodiment of the present application, which may specifically include a receiving module 601, a processing module 602, a determining module 603 and a driving module 604, wherein:
[0105] A receiving module 601 is used to receive real-time data transmitted by a capsule endoscope, wherein the real-time data includes image data captured by the capsule endoscope and posture data of the capsule endoscope;
[0106] A processing module 602 is used to process the image data using a capsule depth estimation network and output a capsule depth map corresponding to the image data;
[0107] A determination module 603 is used to determine a current driving strategy of the capsule endoscope according to the capsule depth map and the posture data;
[0108] The driving module 604 is used to drive the capsule endoscope to move automatically according to the driving strategy.
[0109] In a possible implementation manner of an embodiment of the present application, the capsule depth estimation network may include a teacher model and a student model obtained through model training, the teacher model may be obtained by training capsule endoscope sample images annotated with depth information, the student model may be obtained by jointly training capsule endoscope sample images annotated with depth information and a pseudo-depth annotation set, the pseudo-depth annotation set may be obtained by using the trained teacher model to assign pseudo-depth annotation information to capsule endoscope sample images not annotated with depth information, and the processing module 602 may be specifically used to:
[0110] The trained student model is used to predict the depth value of each pixel in the image data, and a capsule depth map corresponding to the image data is output.
[0111] In a possible implementation of the embodiment of the present application, the determining module 603 may be specifically used to:
[0112] According to the depth value of each pixel in the capsule depth map, converting the capsule depth map into a binary image;
[0113] Performing contour search in the binary image, and determining the center coordinate of the advancing direction of the capsule endoscope according to the found contour;
[0114] Determining the current orientation of the capsule endoscope and the forward direction based on the center coordinates of the forward direction and the posture data;
[0115] A current driving strategy of the capsule endoscope is determined according to the orientation and the forward direction.
[0116] In a possible implementation of the embodiment of the present application, the determining module 603 may also be used to:
[0117] Normalizing the depth value of each pixel in the capsule depth map to process the depth value of each pixel to a preset value range;
[0118] Based on the magnitude relationship between the depth value of each pixel after processing and a preset threshold, the depth value of each pixel is converted into a first value or a second value to obtain a binary image corresponding to the capsule depth map.
[0119] In another possible implementation of the embodiment of the present application, the determining module 603 may also be used to:
[0120] Based on the posture data, the central coordinates of the forward direction are transformed to obtain two vectors in the XY plane in a fixed coordinate system, and the two vectors respectively represent the current orientation and the forward direction of the capsule endoscope.
[0121] In another possible implementation of the embodiment of the present application, the determining module 603 may also be used to:
[0122] Performing a dot product operation on the two vectors to obtain an angle between the two vectors;
[0123] If the angle is less than or equal to the angle threshold, determining that the current driving strategy of the capsule endoscope is forward driving;
[0124] If the included angle is greater than the angle threshold, a cross product operation is performed on the two vectors, and a current driving strategy of the capsule endoscope is determined according to a result of the cross product operation.
[0125] In another possible implementation of the embodiment of the present application, the determining module 603 may also be used to:
[0126] If the result of the cross product operation is greater than zero, determining that the current driving strategy of the capsule endoscope is rightward driving;
[0127] If the result of the cross product operation is less than or equal to zero, it is determined that the current driving strategy of the capsule endoscope is left driving.
[0128] An automatic navigation device for a capsule endoscope provided in an embodiment of the present application can be used to implement the various steps in the aforementioned method embodiments. In a possible implementation, the above-mentioned device can be the software host computer or navigation unit mentioned in the aforementioned system embodiment or method embodiment, and the software host computer or navigation unit can be carried by a computer device. In this way, through the relevant processing of the computer device, the various steps in the aforementioned method embodiment can be implemented to achieve the purpose of computer-assisted diagnosis and treatment.
[0129] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.
[0130] Reference Figure 7 , shows a schematic diagram of a navigation device provided by an embodiment of the present application. Figure 7 As shown, the navigation device 700 in the embodiment of the present application includes: a processor 710, a memory 720, and a computer program 721 stored in the memory 720 and executable on the processor 710. When the processor 710 executes the computer program 721, the steps in each embodiment of the automatic navigation method for capsule endoscope are implemented, such as Figure 2 Alternatively, when the processor 710 executes the computer program 721, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 6 Functions of modules 601 to 604 are shown.
[0131] Exemplarily, the computer program 721 may be divided into one or more modules / units, which are stored in the memory 720 and executed by the processor 710 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which may be used to describe the execution process of the computer program 721 in the navigation device 700. For example, the computer program 721 may be divided into a receiving module, a processing module, a determining module, and a driving module, and the specific functions of each module are as follows:
[0132] A receiving module, used for receiving real-time data transmitted by a capsule endoscope, wherein the real-time data includes image data captured by the capsule endoscope and posture data of the capsule endoscope;
[0133] A processing module, used to process the image data using a capsule depth estimation network and output a capsule depth map corresponding to the image data;
[0134] A determination module, used for determining a current driving strategy of the capsule endoscope according to the capsule depth map and the posture data;
[0135] A driving module is used to drive the capsule endoscope to move automatically according to the driving strategy.
[0136] The navigation device 700 may be an entity carrying a software host computer or a navigation unit in each of the above embodiments, and the navigation device may be a desktop computer, a cloud server, or other devices. The navigation device 700 may include, but is not limited to, a processor 710 and a memory 720. Those skilled in the art will appreciate that Figure 7 It is only an example of the navigation device 700 and does not constitute a limitation of the navigation device 700. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the navigation device 700 may also include input and output devices, network access devices, buses, etc.
[0137] The processor 710 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0138] The memory 720 may be an internal storage unit of the navigation device 700, such as a hard disk or memory of the navigation device 700. The memory 720 may also be an external storage device of the navigation device 700, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the navigation device 700. Further, the memory 720 may also include both an internal storage unit and an external storage device of the navigation device 700. The memory 720 is used to store the computer program 721 and other programs and data required by the navigation device 700. The memory 720 may also be used to temporarily store data that has been output or is to be output.
[0139] An embodiment of the present application also discloses a navigation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the automatic navigation method of the capsule endoscope as described in the aforementioned embodiments is implemented.
[0140] The embodiments of the present application further disclose a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the automatic navigation method of the capsule endoscope as described in the aforementioned embodiments is implemented.
[0141] The present application also discloses a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the automatic navigation method of the capsule endoscope described in the above-mentioned embodiments.
[0142] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.
Claims
1. An automatic navigation method for a capsule endoscope, characterized in that: include: receiving real-time data transmitted by a capsule endoscope, wherein the real-time data includes image data captured by the capsule endoscope and posture data of the capsule endoscope; Processing the image data using a capsule depth estimation network to output a capsule depth map corresponding to the image data; Determining a current driving strategy of the capsule endoscope according to the capsule depth map and the posture data; The capsule endoscope is driven to move automatically according to the driving strategy.
2. The method according to claim 1, characterized in that The capsule depth estimation network includes a teacher model and a student model obtained through model training, wherein the teacher model is obtained by training capsule endoscope sample images annotated with depth information, and the student model is obtained by jointly training capsule endoscope sample images annotated with depth information and a pseudo-depth annotation set, wherein the pseudo-depth annotation set is obtained by assigning pseudo-depth annotation information to capsule endoscope sample images not annotated with depth information using the trained teacher model, and the capsule depth estimation network is used to process the image data and output a capsule depth map corresponding to the image data, including: The trained student model is used to predict the depth value of each pixel in the image data, and a capsule depth map corresponding to the image data is output.
3. The method according to claim 1 or 2, characterized in that: Determining a current driving strategy of the capsule endoscope according to the capsule depth map and the posture data includes: According to the depth value of each pixel in the capsule depth map, converting the capsule depth map into a binary image; Performing contour search in the binary image, and determining the center coordinate of the advancing direction of the capsule endoscope according to the found contour; Determining the current orientation of the capsule endoscope and the forward direction based on the center coordinates of the forward direction and the posture data; A current driving strategy of the capsule endoscope is determined according to the orientation and the forward direction.
4. The method according to claim 3, characterized in that: The step of converting the capsule depth map into a binary image according to the depth value of each pixel in the capsule depth map comprises: Normalizing the depth value of each pixel in the capsule depth map to process the depth value of each pixel to a preset value range; Based on the magnitude relationship between the depth value of each pixel after processing and a preset threshold, the depth value of each pixel is converted into a first value or a second value to obtain a binary image corresponding to the capsule depth map.
5. The method according to claim 3, characterized in that: The determining the current orientation and the forward direction of the capsule endoscope based on the central coordinates of the forward direction and the posture data includes: Based on the posture data, the central coordinates of the forward direction are transformed to obtain two vectors in the XY plane in a fixed coordinate system, and the two vectors respectively represent the current orientation and the forward direction of the capsule endoscope.
6. The method according to claim 5, characterized in that The step of determining a current driving strategy of the capsule endoscope according to the orientation and the advancing direction includes: Performing a dot product operation on the two vectors to obtain an angle between the two vectors; If the included angle is less than or equal to the angle threshold, determining that the current driving strategy of the capsule endoscope is forward driving; If the included angle is greater than the angle threshold, a cross product operation is performed on the two vectors, and a current driving strategy of the capsule endoscope is determined according to a result of the cross product operation.
7. The method according to claim 6, characterized in that The step of determining the current driving strategy of the capsule endoscope according to the result of the cross product operation includes: If the result of the cross product operation is greater than zero, determining that the current driving strategy of the capsule endoscope is rightward driving; If the result of the cross product operation is less than or equal to zero, it is determined that the current driving strategy of the capsule endoscope is left driving.
8. A capsule endoscope navigation system, characterized in that: It includes a display device, an external magnetic source and a navigation unit; wherein: a display device, configured to receive image data transmitted by the capsule endoscope and display the image data; A navigation unit is used to receive the posture data transmitted by the capsule endoscope; process the image data using a capsule depth estimation network to output a capsule depth map corresponding to the image data; determine the current driving strategy of the capsule endoscope according to the capsule depth map and the posture data, and control an external magnetic source according to the driving strategy to drive the capsule endoscope to move automatically.
9. An automatic navigation device for a capsule endoscope, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the automatic navigation method of the capsule endoscope is implemented as described in any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the automatic navigation method of a capsule endoscope is implemented.
Citation Information
Patent Citations
Magnetic-control capsule-type endoscope examination system
CN109091098A
Endoscope image parallax prediction model establishment method based on bidirectional supervision and application
CN115619727A
Autonomous navigation method for tracheal intubation robot based on self-supervised monocular depth estimation
CN116966381A
Endoscope image detection auxiliary system and method, medium and electronic device
WO2023124876A1
KR20220160865A
Cited By
Jejunum nutrient canal indwelling auxiliary system
CN119303209A
A jejunal nutrition tube placement auxiliary system
CN119303209B
