Coloscope tail end pose control method, system and device

Abstract

The invention relates to the technical field of colonoscopy, and provides a colonoscope tail end pose control method, system and device, and the colonoscope tail end pose control method comprises the steps: obtaining a colonoscope initial image and the real-time pose of the colonoscope tail end in real time; inputting the initial image of the colonoscope into a preset colon model to obtain registration parameters of the tail end of the colonoscope; registering the actual pose of the colonoscope tail end with a virtual pose in a preset colon model based on the registration parameters; analyzing the initial image of the colonoscope by adopting a preset image analysis model to obtain correction parameters of the tail end of the colonoscope; and correcting the position and posture of the colonoscope tail end based on the correction parameters to obtain the optimal position and posture of the colonoscope tail end. According to the colonoscope tail end pose control method, the technical problem that position detection of the tail end and effective pose adjustment during colonoscope retreating in the prior art are lacked is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of colonoscopy, and in particular to a method and device for controlling the position and posture of a colonoscope terminal. Background Art

[0002] Colorectal cancer is one of the malignant tumors with the highest morbidity and mortality rates worldwide. To effectively prevent and treat colorectal cancer, colonoscopy has become widely popularized and applied. It has become an important component of modern intestinal disease prevention and treatment and a key means of effectively reducing colorectal cancer mortality.

[0003] Colonoscopy mainly involves two processes: inserting the colonoscope and withdrawing it. During the insertion process, the doctor inserts the colonoscope from the patient's anus and eventually reaches the ileocecal valve. The total length is approximately 100 cm. During the insertion process, the doctor needs to determine the colonoscopy insertion method to be adopted based on the length of the colonoscope that has entered the patient's body, such as "shortening method" or "right rotation and downward pull". This process mainly relies on the doctor's operating experience. During the withdrawal process, the colonoscope moves from the ileocecal valve to the outside of the body until it is withdrawn from the body. During this period, the camera at the end of the colonoscope will capture long-range and close-up images of suspected lesion structures. Based on the images, the doctor determines the occurrence of lesions in the colon.

[0004] However, the process of withdrawing a colonoscope presents numerous challenges. The complex internal anatomy of the colon makes it easy for the endoscope to encounter curved or folded sections of the intestine during examination. Large deviations in the colonoscope's motion path and inaccurate position adjustment of the endoscope tip in these sections can lead to severely low image quality, making it difficult for physicians to identify lesions within the colon, reducing both detection efficiency and lesion recognition rate. Furthermore, when withdrawing a colonoscope, the physician typically needs to retract it along the path it previously advanced. During this process, the endoscope's camera angle cannot be adjusted to the direction of withdrawal. Therefore, determining the colonoscope's position during withdrawal, as well as the accuracy and efficiency of the withdrawal process, primarily relies on the physician's experience and skill. For physicians with limited colonoscopy experience, colonoscopy procedures often take longer and are less efficient. Furthermore, in critical areas such as the hepatic and splenic flexure corners, physicians are unable to determine the colonoscope's position within the intestine, resulting in a 29% missed diagnosis rate for lesions on the dorsal side of the folds. This phenomenon is difficult to effectively address.

[0005] To address these issues, intelligent assisted navigation systems and image-guided technologies have gained increasing attention in the medical field in recent years, particularly in colonoscopy. By incorporating image registration, 3D reconstruction, and virtual reality technologies, physicians can gain a more comprehensive global perspective, improving accuracy and efficiency. For example, the Aer-O-Scope System, developed by GI-View, features real-time imaging, enabling physicians to clearly visualize the interior of the colon. It also adaptively adjusts the direction and position of the scope based on the colon's shape, assisting with endoscopic manipulation. However, most current colonoscopy-assisted navigation systems still primarily focus on the advancement phase of the examination, lacking proper position detection during retraction and effective adjustment of the colonoscope tip. This can lead to problems such as the colonoscope tip being too close or too far from the suspected lesion, as well as poor centering. According to the "Chinese Digestive Endoscopy Image Quality Standard," the optimal distance between the colonoscope tip and the lesion is 10-20 mm. Too close may cause image distortion and mucosal damage, while too far may reduce resolution and impair interpretation of the suspected lesion. Furthermore, the lesion should be located in the center of the image, with a deviation of no more than ±15% of the field of view. If the centrality deviation is too large, there will be edge distortion and uneven illumination problems, which will lead to an increase in the missed diagnosis rate of lesions.

[0006] Therefore, how to provide the accurate position of the colonoscope during the withdrawal process and adaptively adjust the position of the colonoscope terminal to improve the shooting distance and shooting angle of the medical endoscope camera, thereby reducing the lesion detection rate, is still an important issue facing current technology.

[0007] Therefore, the prior art needs to be further developed. Summary of the Invention

[0008] The purpose of the present invention is to overcome the above-mentioned technical deficiencies and provide a method and device for controlling the position of the end of a colonoscope, so as to solve the technical problem in the related art of lacking the position detection of the end when the colonoscope is withdrawn and the effective adjustment of the position.

[0009] In order to achieve the above technical objectives, the present invention adopts the following technical solutions: a method for controlling the posture of the end of a colonoscope is provided, including: acquiring the initial image of the colonoscope and the real-time posture of the end of the colonoscope in real time; inputting the initial image of the colonoscope into a preset colon model to obtain the alignment parameters of the end of the colonoscope; aligning the actual posture of the end of the colonoscope with the virtual posture in the preset colon model based on the alignment parameters; using a preset image analysis model to analyze the initial image of the colonoscope to obtain the correction parameters of the end of the colonoscope; and correcting the posture of the end of the colonoscope based on the correction parameters to obtain the optimal posture of the end of the colonoscope.

[0010] Furthermore, the method for obtaining the real-time posture of the end of the colonoscope includes: collecting the translation distance of the driving device along the x-axis, the rotation angle θy around the y-axis, and the rotation angle θz around the z-axis, wherein the x-axis, y-axis, and z-axis all belong to the colonoscope end coordinate system; obtaining the x-axis coordinate of the end of the colonoscope in the colonoscope end coordinate system based on the translation distance; calculating the rotation matrices Ry and Rz according to the rotation angle θy around the y-axis and the rotation angle θz around the z-axis of the driving device; obtaining the y-axis and z-axis coordinates of the end of the colonoscope in the colonoscope end coordinate system based on the rotation matrices Ry and Rz, performing coordinate posture conversion on the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the end of the colonoscope to obtain the real-time posture of the end of the colonoscope.

[0011] Furthermore, the method for coordinate posture conversion includes: aligning the origin of the colonoscope terminal coordinate system with the origin of the world coordinate system; aligning the x, y and z axes of the colonoscope terminal coordinate system with the x, y and z axes of the world coordinate system through rotation matrix transformation; selecting a preset scale to enlarge or reduce the colonoscope terminal coordinate system to obtain the real-time posture of the colonoscope terminal in the world coordinate system.

[0012] Furthermore, the method for obtaining the alignment parameters of the end of the colonoscope includes: converting the coordinate system of the initial image of the colonoscope and the coordinate system of the end of the colonoscope to the coordinate system of the preset colon model; obtaining the feature point pair of the converted initial image of the colonoscope; comparing the feature point pair with the preset colon model to determine whether the end of the colonoscope deviates; if so, obtaining the alignment parameters through homography matrix calculation.

[0013] Furthermore, the method for obtaining the registration parameters through homography matrix calculation includes: obtaining the homography matrix H based on the feature point pairs of the initial colonoscope image; obtaining the actual posture of the end of the colonoscope in the coordinate system of the preset colon model through the homography matrix H; and obtaining the registration parameters based on the actual posture and the real-time posture of the end of the colonoscope.

[0014] Furthermore, the method for obtaining the correction parameters of the end of the colonoscope includes: obtaining the straight-line distance between the endoscopic camera at the end of the colonoscope and the suspected lesion site and the center deviation value of the colonoscope correction image based on the colonoscope correction image; performing coordinate conversion on the straight-line distance and the center deviation value to obtain the true position of the end of the colonoscope in the coordinate system of the preset colon model; and obtaining the correction parameters based on the true position, the preset straight-line distance and the preset center deviation.

[0015] Furthermore, the preset image analysis model is specifically a neural network architecture based on a Pytorch-defined intelligent agent, wherein the preset image analysis model includes a feature reconstruction layer, which is used to capture boundary features of suspected lesion sites, and the feature reconstruction layer includes 3×3 convolution combined with 1×1 convolution.

[0016] Furthermore, the colonoscope terminal posture control method also includes: based on a preset colon model, obtaining a visualized virtual colonoscope motion trajectory diagram by real-time position sampling, visual rendering and motion path superposition of the colonoscope terminal to display the position of the colonoscope terminal in the colon.

[0017] A colonoscope terminal posture control system, the colonoscope terminal posture control system includes: a data acquisition unit, the data acquisition unit is used to obtain the initial image of the colonoscope and the real-time posture of the colonoscope terminal in real time; a first data processing unit, the first data processing unit is used to input the initial image of the colonoscope into a preset colon model to obtain the alignment parameters of the colonoscope terminal; the alignment unit, the alignment unit is used to align the actual posture of the colonoscope terminal with the virtual posture in the preset colon model based on the alignment parameters; the second data processing unit, the second data processing unit is used to analyze the initial image of the colonoscope using a preset image analysis model to obtain the correction parameters of the colonoscope terminal; the correction unit, the correction unit is used to correct the posture of the colonoscope terminal based on the correction parameters to obtain the optimal posture of the colonoscope terminal.

[0018] A colonoscope terminal position control device comprises: a first drive assembly, the first drive assembly is used to drive the colonoscope terminal to move horizontally along the x-axis; a second drive assembly, the second drive assembly is used to drive the colonoscope terminal to rotate along the y-axis; and a third drive assembly, the third drive assembly is used to drive the colonoscope terminal to rotate along the z-axis.

[0019] Beneficial effects:

[0020] 1. The colonoscope terminal posture control method of the present invention goes through four stages: image acquisition, image feature processing, network control, and execution. The neural network finally outputs the straight-line distance between the endoscopic camera at the end of the colonoscope and the suspected lesion site, and the center deviation value of the suspected lesion site in the image. The virtual simulation platform converts the distance and center deviation values ​​into the △x, △y, and △z values ​​required for adjustment of the virtual colonoscope in the virtual simulation platform world coordinate system, updates the virtual colonoscope coordinates, adjusts the virtual colonoscope terminal posture, and converts the final posture into x, y, and z-axis posture data in the real world coordinate system, and outputs it to the scope withdrawal auxiliary device. After receiving the data information, the scope withdrawal auxiliary device controls the movement of the closed-loop stepping motor to achieve precise control of the colonoscope terminal posture, ensuring that the colonoscope terminal camera is 10-20mm away from the lesion site and the offset of the lesion site in the image does not exceed 15%, providing doctors with high-quality medical endoscopic images that can read and identify lesions, and realizing the position detection of the terminal when withdrawing the colonoscope and the effective adjustment of the colonoscope terminal posture.

[0021] 2. The colonoscope terminal posture control method of the present invention builds a virtual colonoscope model based on the three-dimensional data of CT scan, and displays the posture of the colonoscope terminal in the model through coordinate transformation, obtaining a visualized virtual colonoscope motion trajectory diagram to determine the position of the colonoscope in the colon, providing guidance for the doctor's scope withdrawal operation, and reducing the operator's dependence on experience and operating skills.

[0022] 3. The position control device for the distal end of the colonoscope of the present invention controls the movement of the distal end of the colonoscope through three closed-loop stepper motors, two rotating wheels, and several gear sets. The two rotating wheels fit tightly against the two knobs on the operating part of the colonoscope. The two closed-loop stepper motors change and transmit power through the gear sets to rotate the rotating wheels, which in turn rotate the knobs to achieve y- and z-axis movement of the distal end of the colonoscope. Another closed-loop stepper motor acts on the body of the colonoscope through a friction wheel. The motor drives the friction wheel to rotate, thereby moving the body of the colonoscope and achieving x-axis movement of the distal end of the colonoscope. The real-time position of the distal end of the colonoscope can be accurately calculated by the rotation distance and rotation angle of the gears, solving the technical problem in the prior art of the difficulty in locating the position of the colonoscope in the colon during the withdrawal of the colonoscope. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the structure of the colonoscope terminal position control device used in an embodiment of the present invention;

[0024] Figure 2 is a flow chart of a colonoscope terminal position control method used in an embodiment of the present invention;

[0025] Figure 3 Schematic diagram of the structure of the colonoscope terminal posture control system used in an embodiment of the present invention.

[0026] The above drawings include the following reference numerals:

[0027] 1. First drive assembly; 2. Second drive assembly; 3. Third drive assembly. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] According to an embodiment of the present invention, a method for controlling the position and posture of a colonoscope terminal is provided. Figures 1 to 3 ,include:

[0030] S100 acquires the initial colonoscope image and the real-time position of the colonoscope tip in real time;

[0031] In the colonoscope terminal end posture control method of this embodiment, the method for obtaining the real-time posture of the colonoscope terminal includes:

[0032] S110 acquires the translation distance of the driving device along the x-axis, the rotation angle θy around the y-axis, and the rotation angle θz around the z-axis, wherein the x-axis, the y-axis, and the z-axis all belong to the colonoscope end coordinate system;

[0033] S120 obtains an x-axis coordinate of the colonoscope end in the colonoscope end coordinate system based on the translation distance;

[0034] S130 calculates rotation matrices Ry and Rz based on the rotation angle θy of the driving device around the y-axis and the rotation angle θz around the z-axis;

[0035] S140 obtains the y-axis and z-axis coordinates of the colonoscope tip in the colonoscope tip coordinate system based on the rotation matrices Ry and Rz;

[0036] S150 performs coordinate posture conversion on the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the end of the colonoscope to obtain the real-time posture of the end of the colonoscope.

[0037] In practice, two closed-loop stepper motors drive a gear train at the end of the colonoscope to change and transmit power. The operator controls the two knobs on the operating end of the colonoscope to drive the end of the colonoscope on the y and z axes. Another closed-loop stepper motor directly controls a friction wheel, which contacts the body of the colonoscope. When the motor rotates, the friction wheel rotates, driving the body of the colonoscope through friction at equal distances, thereby controlling the x-axis movement of the end of the colonoscope.

[0038] It should be noted that by calculating the number of rotations of the closed-loop stepper motor, the circumference of the friction wheel can be obtained, and then the x-axis translation data of the colonoscope can be calculated.

[0039] In actual practice, the three motors work together to ultimately control the x-, y-, and z-axis movements of the colonoscope's terminal position.

[0040] It should be noted that the two closed-loop stepper motors drive the wheels to rotate and then drive the knobs to rotate, so that the movement of the colonoscope terminal on the y and z axes is rotation rather than translation. Therefore, after collecting the rotation angles θy and θz, it is necessary to transform the position of the colon terminal on the y and z axes based on the matrix rotation transformation theory. The specific calculation process is as follows:

[0041] First, the rotation matrix Ry is constructed according to the rotation angle θy around the y-axis, and its expression is

[0042] Next, the rotation matrix Rz is constructed according to the rotation angle θz around the z axis, and its expression is

[0043]

[0044] Then, the two rotation matrices are multiplied to obtain the total rotation matrix R. The matrix multiplication order is R = Ry × Rz to realize the rotation transformation around the z axis first and then around the y axis. The initial position vector p of the colonoscope is defined as the unit matrix.

[0045] During the movement of the colonoscope, the initial position vector p is continuously multiplied by the rotation matrix R to obtain the colonoscope coordinate p' that changes at all times during the process, and then the y and z axis data of the end of the colonoscope after rotation are obtained.

[0046] In specific practice, the methods for coordinate pose conversion also include:

[0047] S151 aligns the origin of the colonoscope tip coordinate system with the origin of the world coordinate system;

[0048] S152 aligns the x, y, and z axes of the colonoscope tip coordinate system with the x, y, and z axes of the world coordinate system through a rotation matrix transformation;

[0049] S153 selects a preset scale to enlarge or reduce the colonoscope end coordinate system to obtain the real-time position and posture of the colonoscope end in the world coordinate system.

[0050] In actual practice, it is necessary to combine the x-axis translation data to perform the position coordinate conversion between different coordinate systems on the end of the colonoscope.

[0051] Specifically, when performing posture coordinate transformation, the colonoscope terminal coordinate system is first translated to align its origin with the world coordinate system; then, a 3x3 rotation matrix transformation is performed to make the x, y, and z axes of the colonoscope terminal coordinate system collinear with the x, y, and z axes of the world coordinate system; finally, the colonoscope terminal posture data in the real-world coordinate system is obtained by reducing the scale.

[0052] In practice, the encoder can be used to measure the rotation angle of the closed-loop stepper motor when the colonoscope is retracted. The rotation angle of the coaxial wheel with the motor shaft is equal to the rotation angle of the motor shaft. The wheel drives the knob to rotate synchronously. The movement angle of the two knobs is obtained, and the final movement distance of the colonoscope tip can be obtained. The coordinates of the colonoscope tip are subjected to matrix rotation transformation and posture coordinate transformation, and then the x, y, and z axis data of the colonoscope tip posture in the virtual simulation platform world coordinate system (i.e., the coordinate system where the preset colon model is located) can be obtained.

[0053] S200 inputs the initial colonoscope image into the preset colon model to obtain the registration parameters of the colonoscope end;

[0054] In the colonoscope terminal posture control method of this embodiment, the method for obtaining the registration parameters of the colonoscope terminal includes:

[0055] S210 converts the initial colonoscope image coordinate system and the coordinate system where the colonoscope end is located into the coordinate system where the preset colon model is located;

[0056] S220 obtains feature point pairs of the converted colonoscopy initial image;

[0057] S230 compares the feature point pair with a preset colon model to determine whether the end of the colonoscope deviates;

[0058] If yes, then the registration parameters are obtained by calculating the homography matrix.

[0059] Specifically, before a colonoscopy, a "virtual colonoscope" agent (including a preset colon model) is created using Unity software. During the colonoscopy, the colonoscope's distal end posture data and initial colonoscope image are imported into the preset colon model. The STM32F103 microcontroller transmits this data to the virtual simulation platform built using Unity via serial communication. The virtual colonoscope's distal end is then driven by this data and moves to the designated location. MJPEG streaming, which compresses a sequence of continuous JPEG-formatted static images for real-time streaming, is suitable for demanding real-time scenarios.

[0060] During this process, Unity continuously receives and reads 320x240 JPEG images (i.e., the initial colonoscope image) captured by the medical endoscope camera at the end of the colonoscope through MJPEG streaming. It then performs feature point matching between the initial colonoscope image and the virtual colonoscope field of view image on the virtual simulation platform. If the feature point matching between the two falls below 70%, it is determined that the motion matching between the virtual simulation platform and the actual colonoscope has deviated, and the agent's position and posture are then corrected based on the homography matrix.

[0061] Specifically, the method of obtaining the registration parameters by calculating the homography matrix includes:

[0062] S241 calculates a homography matrix H based on feature point pairs of the initial colonoscopy image;

[0063] S242 obtains the actual position of the colonoscope end in the coordinate system of the preset colon model through the homography matrix H;

[0064] S243 obtains registration parameters based on the actual posture and the real-time posture of the colonoscope tip.

[0065] Specifically, 4 pairs are randomly selected from the feature point pairs, and the coordinates of the feature points in the source image are set as (xi, yi) and the coordinates of the corresponding points in the target image are set as (xi′, yi′). The homogeneous coordinate equation [x i ′,y i ′,1] T＝ H·[x i ,y i ,1] T , the candidate homography matrix H is obtained by the least squares method; then the candidate matrix is ​​used to perform a projection transformation on all feature point pairs, the error between the transformed coordinates and the actual target point coordinates is calculated, and the correct matching points with an error less than the threshold and the incorrect matching points with an error greater than the threshold are distinguished according to the threshold; then the sampling, solving, and screening steps are repeated, and the candidate matrix with the largest number of inliers is recorded as the optimal solution. The iteration is terminated until the preset number of iterations is reached or the number of inliers no longer increases significantly. Finally, all inliers are used to recalculate the homography matrix H through the least squares method to obtain a high-precision homography matrix.

[0066] Preferably, the homography matrix H obtained through several iterations and modifications is used as a rotation transformation matrix and multiplied by the original virtual colonoscope terminal posture P to obtain the colonoscope terminal posture information P' (P'=H·P) under the virtual simulation platform.

[0067] S300 aligns the actual posture of the colonoscope tip with the virtual posture in the preset colon model based on the registration parameters;

[0068] In specific practice, the posture of the virtual simulation platform in the world coordinate system is transformed to obtain the posture of the colonoscope terminal in the real world coordinate system, and then the three closed-loop stepper motors are controlled to rotate to achieve the posture adjustment of the colonoscope terminal and complete the image matching.

[0069] The S400 uses a preset image analysis model to analyze the initial colonoscope image and obtain correction parameters for the end of the colonoscope;

[0070] In the colonoscope terminal posture control method of this embodiment, the method for obtaining the correction parameters of the colonoscope terminal includes:

[0071] S510 obtains a straight-line distance between the end endoscope camera of the colonoscope and the suspected lesion site and a center deviation value of the colonoscope correction image based on the colonoscope correction image;

[0072] S520 performs coordinate conversion on the straight-line distance and the center deviation value to obtain the true position of the colonoscope end in the coordinate system of the preset colon model;

[0073] S530 obtains a correction parameter based on the actual posture, the preset straight line distance, and the preset center deviation.

[0074] In practice, the deep learning of the preset image analysis model calculates the distance value and the center deviation value. The virtual simulation platform (i.e., the preset colon model) converts the distance and center offset into the offset of the x, y, and z axes in the virtual colonoscope coordinate system. Finally, the virtual colonoscope coordinate system is translated to the coordinate system of the virtual simulation platform world coordinate system, and the rotation matrix is ​​changed to achieve the collinearity of the x, y, and z axes in the two coordinate systems and the scale is magnified to obtain the true posture of the virtual colonoscope. Then, the virtual colonoscope coordinates are updated, and the end posture of the virtual colonoscope is driven to move to achieve a distance of 10-20mm from the suspected lesion site, and the center offset of the suspected lesion site does not exceed 15%.

[0075] Specifically, the neural network realizes depth estimation of monocular images based on deep learning. The core working principle is to map the input RGB image into a pixel-level depth map through the encoder-decoder architecture combined with transfer learning, and then convert it into actual distance; in terms of network architecture, the encoder uses the pre-trained DenseNet, and the decoder gradually restores the resolution through deconvolution and jump connections, fusing the semantic features of different levels of the encoder with the underlying detail information, and finally generating a high-resolution depth map.

[0076] The calculation method of the distance value obtained by deep learning includes:

[0077] After the encoder extracts deep features from the input image, the decoder gradually enlarges the size of the feature map through upsampling operations and convolutional layers without batch normalization. At the same time, it uses skip connections to directly pass the edge, texture and other detail information of the encoder's shallow layer to the corresponding decoder layer, thereby improving the edge clarity and detail accuracy of the depth map.

[0078] A multi-task loss function, including L1 loss, gradient loss, and structural similarity loss, is used during model training to balance the global consistency of depth values ​​and local structural details.

[0079] It should be noted that the method for obtaining the true posture of the virtual colonoscope includes: obtaining a deep learning-processed colon view after deep learning, in which the distal side of the colon view appears white and the proximal side appears black. During the colonoscope withdrawal process, it is necessary to ensure that the colonoscope camera is centered. Therefore, in the colon view processed by deep learning, the white area is in the middle of the picture, so that the reasonable withdrawal path of the colonoscope can be achieved.

[0080] Specifically, a colon view processed by deep learning is obtained, and the view is divided into nine areas: upper left, left side, lower left, upper right, right side, lower right, upper side, lower side, and middle. When the white area of ​​the colon view processed by deep learning is concentrated in the upper area, it means that the colon cavity is above the movement of the colonoscope. The colonoscope needs to move along the positive direction of the z-axis to raise the camera so that the white area is in the middle of the picture. At this time, the y and z-axis directions of the colonoscope remain unchanged, and the original direction is maintained for retraction until the white area changes again to other areas of the colon view processed by deep learning, and the y and z-axis operations are performed to keep the white area in the center of the picture.

[0081] S600 corrects the posture of the colonoscope terminal end based on the correction parameters to obtain the optimal posture of the colonoscope terminal end.

[0082] In actual practice, the final posture of the virtual colonoscope is achieved through coordinate system translation and rotation matrix transformation to achieve the collinearity of the x, y, and z axes of the two coordinate systems and scale enlargement, and finally converted into x, y, and z axis posture data in the real-world coordinate system and input into the STM32F103 microcontroller. After receiving the three-axis posture data, the microcontroller controls the three closed-loop stepper motors to rotate the corresponding angles to adjust the terminal posture of the colonoscope, thereby realizing fine control of the terminal posture of the colonoscope.

[0083] In this way, the pictures of suspected lesions in the colon taken by the colonoscope are compared again after adjustment, which have the characteristics of appropriate distance and suspected lesions being in the center of the taken image. This can help doctors effectively identify intestinal lesions, improve detection efficiency and lesion detection rate, and solve the technical problems in related technologies of lack of position detection of the end of the colonoscope when withdrawing the scope and effective adjustment of the posture.

[0084] In the colonoscope terminal posture control method of this embodiment, the preset image analysis model is specifically a neural network architecture based on Pytorch-defined intelligent agent, wherein the preset image analysis model includes a convolution layer, which reduces the amount of calculation by combining 3×3 convolution with 1×1 convolution, forms multi-scale feature reuse, alleviates gradient disappearance and enhances feature expression capabilities, and pre-trained weight initialization improves the efficiency of transfer learning.

[0085] It should be noted that the preset image analysis model of this embodiment is a neural network architecture based on the Pytorch-defined intelligent agent. The 320x240 JPEG image taken by the medical endoscope camera at the end of the colonoscope is used as the neural network input. Through neural network model training, the straight-line distance between the medical endoscope camera at the end of the colonoscope and the suspected lesion site and the center deviation value of the image of the suspected lesion site are finally output, thereby realizing medical endoscope camera image analysis based on neural network.

[0086] In the colonoscope terminal posture control method of this embodiment, the colonoscope terminal posture control method also includes: based on a preset colon model, obtaining a visualized virtual colonoscope motion trajectory diagram by performing real-time position sampling, visual rendering, and motion path superposition of the colonoscope terminal to display the position of the colonoscope terminal in the colon.

[0087] In actual practice, the virtual colonoscope moves in the virtual simulation platform, and basic simulation problems such as processing and detecting collisions are solved through Unity's built-in MeshCollider.

[0088] Specifically, using Unity's built-in XR Interaction Toolkit, through real-time position sampling, visual rendering, and motion path overlay, a visual virtual colonoscope motion trajectory map is obtained. In this way, the position of the colonoscope end in the colon can be displayed. By observing the trajectory map, doctors can clearly understand the position of the colonoscope in the intestine and identify some specific lesions.

[0089] The present embodiment provides a colonoscope terminal posture control system, which includes: a first data acquisition unit, which is used to acquire the initial image of the colonoscope and the real-time posture of the colonoscope terminal in real time; a first data processing unit, which is used to input the initial image of the colonoscope into a preset colon model to obtain the alignment parameters of the colonoscope terminal; a first correction unit, which is used to correct the posture of the colonoscope terminal based on the alignment parameters to obtain a first corrected posture; a second data acquisition unit, which is used to acquire a colonoscope correction image taken by the colonoscope terminal in the first corrected posture; a second data processing unit, which is used to analyze the colonoscope correction image using a preset image analysis model to obtain the correction parameters of the colonoscope terminal; and a second correction unit, which is used to correct the posture of the colonoscope terminal based on the correction parameters to obtain the optimal posture of the colonoscope terminal.

[0090] In the colonoscope terminal posture control system of this embodiment, when constructing the colon model, the primary human colon three-dimensional model is cut into thirteen segments using Zbrush engraving software, including the ileocecal and ascending colon segments, hepatic flexure segment a, hepatic flexure segment b, etc., in view of the characteristics of the colon's complex internal curvature, uneven outer surface and numerous grooves, so as to accurately restore the structure of each segment; the three-dimensional sub-model of the human colon outer mold corresponding to each single-segment bionic colon inner mold is composed of four unequal sub-blocks, and adjacent sub-blocks are set with positioning pins and positioning holes for easy assembly, and are provided with two pouring holes and one exhaust hole to ensure sufficient injection of silicone solution.

[0091] Preferably, the colonoscope terminal posture control system implemented in this embodiment is based on the three-dimensional data of the colon obtained by CT scanning. After improvement and modification, the stp file of the colon model is obtained, and the stp file is converted into OBJ format using Blender and imported into Unity. The material, texture and other information are adjusted to obtain a preset colon model, and then a "virtual colonoscope" intelligent agent is constructed based on Unity's built-in toolkit ML-Agents.

[0092] This embodiment provides a colonoscope terminal position control device, which includes: a first drive component 1, which is used to drive the colonoscope terminal to move horizontally along the x-axis direction; a second drive component 2, which is used to drive the colonoscope terminal to rotate along the y-axis direction; and a third drive component 3, which is used to drive the colonoscope terminal to rotate along the z-axis direction.

[0093] With the above-mentioned setting, the position control device of the end of the colonoscope is controlled by the STM32F103 single-chip microcomputer, which includes a first drive component 1, a second drive component 2 and a third drive component 3. Specifically, it protects three closed-loop stepper motors, two rotating wheels and several gear sets. The two rotating wheels in the second drive component 2 and the third drive component 3 fit tightly with the two knobs of the colonoscope operating part. The two closed-loop stepper motors change and transmit power through the gear set to rotate the rotating wheels, and then rotate the knobs to achieve the y- and z-axis movement of the end of the colonoscope. The closed-loop stepper motor in the first drive component 1 acts on the body of the colonoscope through the friction wheel. The motor drives the friction wheel to rotate, thereby moving the body of the colonoscope and achieving the x-axis movement of the end of the colonoscope.

[0094] In actual practice, the position control device of the colonoscope terminal cooperates with the virtual simulation platform. The virtual simulation platform builds a preset colon model based on the three-dimensional colon data obtained from CT scanning of the human colon, imports the three-dimensional colon data into Unity to form a virtual colon model, and then adds the intelligent body "virtual colonoscope" module to Unity and adds collision body properties to it to complete the construction of the virtual simulation platform.

[0095] It should be noted that before the colonoscope terminal posture control system works, the colonoscope terminal posture control device needs to be installed so that the two rotating wheels of the second drive component 2 and the third drive component 3 of the colonoscope terminal posture control device are tightly fitted with the two knobs of the colonoscope operating part respectively, and the friction wheel of the first drive component 1 is tightly fitted with the colonoscope body.

[0096] Preferably, the colonoscope terminal posture control device is equipped with an STM32F103 single-chip microcomputer to achieve precise closed-loop control of the motor and data solution to obtain the accurate colonoscope terminal posture.

[0097] Preferably, the colonoscope terminal position control device is equipped with three closed-loop stepper motors, two rotating wheels and several internal gear sets, and is encapsulated with several angle codes and aluminum sheets on the periphery.

[0098] Example 1:

[0099] The workflow of the colonoscope terminal posture control system is as follows:

[0100] First, the position control device of the colonoscope terminal is started, and the three closed-loop stepper motors corresponding to the first drive component 1, the second drive component 2 and the third drive component 3 are started to control the x, y and z axis movements of the colonoscope terminal and obtain the data of the three-axis movement of the colonoscope terminal.

[0101] It should be noted that the two closed-loop stepper motors corresponding to the second drive assembly 2 and the third drive assembly 3 drive the knobs to rotate, thereby driving the colonoscope tip to move in the form of rotation rather than translation on the y and z axes. Therefore, after collecting the motor rotation angles, it is necessary to obtain the y and z axis data of the colonoscope tip based on the matrix rotation transformation. Combined with the x-axis translation data, the posture coordinate conversion is performed to obtain the colonoscope tip posture data in the real-world coordinate system.

[0102] In practice, the calculated world coordinate pose data and the 320x240 JPEG image captured by the colonoscope are sent as input to the virtual simulation platform via the STM32F103 serial port. The virtual simulation platform then converts the data into x, y, and z axis data in the virtual simulation platform's world coordinate system. The "virtual colonoscope" agent is driven by this data to move to the designated location. During this process, the virtual simulation platform continuously receives and reads the colonoscope images, matching feature points between them and the virtual colonoscope's field of view in Unity. This process uses a data-driven image registration process to control the motion alignment of the virtual colonoscope and the colonoscope. After registration, the virtual colonoscope moves, and Unity's built-in agent's real-time sampling and rendering capabilities generate a visual map of the virtual colonoscope's motion trajectory, which helps determine the colonoscope's position within the colon.

[0103] Furthermore, the colonoscope terminal posture control system in this embodiment also defines the neural network architecture through Pytorch, and takes a 320x240 JPEG image taken by the medical endoscope camera at the end of the colonoscope as input. Through four stages of image acquisition, image feature processing, network control, and execution, the neural network finally outputs the straight-line distance between the camera and the suspected lesion site and the center deviation value of the suspected lesion site in the image.

[0104] In practice, the virtual simulation platform converts the distance and center deviation values ​​into the △x, △y, and △z values ​​required to adjust the virtual colonoscope in the virtual simulation platform's world coordinate system. It then updates the virtual colonoscope's coordinates, adjusts the virtual colonoscope's terminal posture, and converts the final posture into x-, y-, and z-axis posture data in the real-world coordinate system before outputting it to the colonoscope's terminal posture control device. The STM32F103 microcontroller receives the data and controls the movement of the three closed-loop stepper motors to precisely control the terminal posture of the colonoscope, ensuring that the colonoscope's terminal camera is 10-20mm away from the lesion and that the offset of the lesion in the image does not exceed 15%. Ultimately, it provides high-quality medical endoscope images that doctors can read and identify lesions.

[0105] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0106] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0107] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0108] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0109] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A colonoscope terminal posture control method, characterized in that: include: Acquiring the initial colonoscope image and the real-time position of the distal end of the colonoscope in real time; Inputting the initial colonoscope image into a preset colon model to obtain registration parameters of the colonoscope end; registering the actual posture of the colonoscope tip with the virtual posture in the preset colon model based on the registration parameters; Analyzing the initial colonoscope image using a preset image analysis model to obtain correction parameters for the end of the colonoscope; The position of the distal end of the colonoscope is corrected based on the correction parameters to obtain the optimal position of the distal end of the colonoscope.

2. The colonoscope terminal position control method according to claim 1, characterized in that: The method for obtaining the real-time position of the distal end of the colonoscope includes: Acquire the translation distance of the driving device along the x-axis, the rotation angle θy around the y-axis, and the rotation angle θz around the z-axis, where the x-axis, y-axis, and z-axis all belong to the colonoscope end coordinate system; Obtaining the x-axis coordinate of the colonoscope end in the colonoscope end coordinate system based on the translation distance; Calculating rotation matrices Ry and Rz based on the rotation angle θy of the driving device around the y-axis and the rotation angle θz around the z-axis; Obtaining the y-axis and z-axis coordinates of the colonoscope tip in the colonoscope tip coordinate system based on the rotation matrices Ry and Rz; The x-axis coordinate, y-axis coordinate and z-axis coordinate of the end of the colonoscope are converted into a coordinate posture to obtain a real-time posture of the end of the colonoscope.

3. The colonoscope terminal position control method according to claim 2, characterized in that: The method for performing coordinate posture conversion comprises: Aligning the origin of the colonoscope tip coordinate system with the origin of the world coordinate system; Aligning the x, y, and z axes of the colonoscope tip coordinate system with the x, y, and z axes of the world coordinate system through a rotation matrix transformation; A preset scale is selected to enlarge or reduce the colonoscope end coordinate system to obtain the real-time position of the colonoscope end in the world coordinate system.

4. The colonoscope terminal position control method according to claim 1, characterized in that: The method for obtaining the registration parameters of the colonoscope end includes: Converting the initial colonoscope image coordinate system and the coordinate system of the colonoscope end into the coordinate system of the preset colon model; Obtaining feature point pairs of the converted initial colonoscopy image; Comparing the feature point pair with the preset colon model to determine whether the distal end of the colonoscope deviates; If so, the registration parameters are obtained by calculating the homography matrix.

5. The colonoscope terminal position control method according to claim 4, characterized in that: The method for obtaining the registration parameters by calculating the homography matrix includes: A homography matrix H is calculated based on the feature point pairs of the initial colonoscopy image; Obtaining the actual position of the colonoscope end in the coordinate system of the preset colon model through the homography matrix H; The registration parameters are obtained based on the actual posture and the real-time posture of the colonoscope tip.

6. The colonoscope terminal position control method according to claim 1, characterized in that: The method for obtaining the calibration parameters of the colonoscope end includes: Obtaining a straight-line distance between the colonoscope terminal end endoscope camera and the suspected lesion site and a center deviation value of the colonoscope correction image based on the colonoscope correction image; Performing coordinate transformation on the straight-line distance and the center deviation value to obtain the true position of the colonoscope end in the coordinate system of the preset colon model; The correction parameter is obtained based on the true posture, the preset straight-line distance and the preset center deviation.

7. The colonoscope terminal position control method according to claim 1, characterized in that: The preset image analysis model is specifically a neural network architecture based on a Pytorch-defined intelligent agent, wherein the preset image analysis model includes a feature reconstruction layer, which is used to capture the boundary features of suspected lesion sites, and the feature reconstruction layer includes a 3×3 convolution combined with a 1×1 convolution.

8. The colonoscope terminal position control method according to claim 1, characterized in that: The colonoscope terminal posture control method also includes: based on a preset colon model, obtaining a visualized virtual colonoscope motion trajectory diagram by real-time position sampling, visual rendering and motion path superposition of the colonoscope terminal to display the position of the colonoscope terminal in the colon.

9. A colonoscope terminal posture control system, characterized in that: The colonoscope terminal posture control system includes: A data acquisition unit, the data acquisition unit is used to acquire the initial image of the colonoscope and the real-time posture of the end of the colonoscope in real time; a first data processing unit, configured to input the initial colonoscope image into a preset colon model to obtain registration parameters of the colonoscope end; a registration unit, configured to register the actual posture of the colonoscope tip with the virtual posture in the preset colon model based on the registration parameters; a second data processing unit, configured to analyze the initial colonoscope image using a preset image analysis model to obtain correction parameters for the distal end of the colonoscope; A correction unit is used to correct the posture of the colonoscope terminal based on the correction parameters to obtain the optimal posture of the colonoscope terminal.

10. A colonoscope terminal position control device, characterized in that: The colonoscope terminal position control device comprises: A first driving assembly (1), the first driving assembly (1) is used to drive the distal end of the colonoscope to move horizontally along the x-axis direction; A second driving assembly (2), the second driving assembly (2) is used to drive the distal end of the colonoscope to rotate along the y-axis direction; A third driving assembly (3) is used to drive the colonoscope end to rotate along the z-axis direction.

Images