Autonomously steerable flexible endoscope control method based on spatial perception

Abstract

The present invention relates to a method for controlling an autonomously guided flexible endoscope based on spatial perception, comprising: S1: acquiring a monocular endoscope image for three-dimensional spatial perception to obtain an environmental depth map; S2: processing the environmental depth map to extract and optimize a guidance path; and S3: calculating geometric information of the guidance path to implement adaptive guidance control. The present invention has the ability to perform three-dimensional perception of the space of the endoscope's working environment; the present invention has the ability to adapt to the spatial structure of the endoscope's working environment; under the control implementation of the present invention, a smaller endoscope transport force and a smoother endoscope transport effect can be achieved; the overall process of the present invention has the advantages of transparency and clear physical information.

Description

Technical Field

[0001] The present invention belongs to the field of control methods for medical electronic equipment, and in particular to a control method for an autonomously steerable flexible endoscope based on spatial perception. Background Art

[0002] Robotic motion and instrument control are fundamental issues in robotic endoscopy, endoscope control simulation, and endoscopy training. Based on experience, the main challenge of autonomous endoscopy is limited maneuverability due to limited spatial perception. Figure 1 In the figure (B), the obstruction of distal regions by proximal tissues makes operation difficult. Therefore, autonomous endoscopic navigation strategies should be able to sense the shape of the lumen and decide when, where, and at what speed the endoscope should be turned and delivered.

[0003] The intracavitary environment in which the endoscope is working is closed, in which the endoscope is the only light source (see Figure 1 Based on these insights, one type of navigation solution is to look for visual features, such as dark areas, which are often considered to be the center of the lumen indicating the direction of travel. However, because the methods for detecting dark areas or lumen centers can only provide a planar target on the endoscopic image, regardless of the tissue state, such methods can only instruct the robotic endoscope to turn and cannot make more adjustments to the delivery. In this case, the endoscope may be over-turned to the point where the endoscope is moving. Figure 1 The endoscope is positioned at position ②, which may contact tissue. Contact between the endoscope and tissue may result in less smooth delivery, resulting in greater delivery force.

[0004] To solve this problem, some researchers have designed navigation methods by using the geometric contour information of human anatomical structures. A good example is the use of the folds of the colon (see Figure 1 -A). These methods obtain spatial perception of partial areas with regular contours. However, when the contours are not visible, such as when the endoscope is too close to the tissue to detect the contours ( Figure 1 Anatomy-based methods fail when the tissue is subjected to drastic tissue motion (e.g., ④), or when the tissue is subjected to drastic tissue motion. Therefore, this approach is not robust and requires extensive additional processing, which may reduce the stability and practicality of autonomous endoscopy. In addition, the close relationship with the anatomy makes such methods less generalizable to other organoids with less recognizable contour features (see Figure 1 Middle D).

[0005] The aforementioned methods that utilize visual features or anatomical geometric contour information can be used to determine the steering direction of the endoscope, but due to the lack of three-dimensional spatial perception, they cannot achieve sufficient maneuverability. This may result in less than smooth delivery, greater delivery force, or even loss of navigation. Summary of the Invention

[0006] The present invention proposes an autonomously steerable flexible endoscope control method based on spatial perception, comprising acquiring a monocular endoscope image for three-dimensional spatial perception to obtain an environmental space depth map; processing the environmental space depth map to extract and optimize a guidance path; and calculating the geometric information of the guidance path to achieve adaptive guidance control.

[0007] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a method for controlling an autonomously steerable flexible endoscope based on spatial perception, comprising the following steps:

[0008] S1. Obtain a monocular endoscopic image for three-dimensional spatial perception and obtain a depth map of the environmental space;

[0009] S2. Process the environment space depth map, extract the guidance path and optimize it;

[0010] S3. Calculate the geometric information of the guidance path to achieve adaptive guidance control.

[0011] The method of acquiring a monocular endoscopic image for three-dimensional spatial perception and obtaining an environmental spatial depth map adopts an end-to-end supervised deep learning method, including the following steps:

[0012] The end-to-end supervised deep learning method takes a monocular endoscopic image as input and outputs a pixel-by-pixel depth map of the same resolution;

[0013] During training, the synthesized monocular endoscopic image is used as input, and the output depth map is compared with the synthesized ground-truth depth map to calculate the composite loss function. The training data used during training is obtained through the following steps:

[0014] Obtain a three-dimensional model of the environment space;

[0015] Add textures in the environment space and set virtual paths;

[0016] Let the virtual camera move along the virtual path;

[0017] Export color images and corresponding depth data through scripts.

[0018] The extraction guide path comprises the following steps:

[0019] Select the discrete coefficient and depth range, and voxelize the depth map to form a binary three-dimensional depth field grid;

[0020] The binary 3D depth field grid is iteratively reduced using a digital topology method to obtain a discrete guidance path.

[0021] The extraction guide path comprises the following steps:

[0022] First, the depth map D is corrected based on the depth conversion function: for the point ((i, j) on the depth map D T ,d), select the discretization resolution parameter S, then the depth conversion function is:

[0023]

[0024] get is the interior point of the topological graph; i and j represent the point position, d represents the depth, and d max d min Respectively represent the maximum depth and minimum depth of the depth map, Represents an integer;

[0025] Then, according to the definition of digital topology, for the interior point set Perform the test and delete some points that do not belong to the (26,6) topology to obtain the required digital topology. The interior point set is the set of topological domain grids corresponding to the spatial region smaller than the depth value on the depth map;

[0026] In the interior point set Get the anchor point set The anchor point set is a set of points that cannot be removed during topology reduction and is used to constrain the start and end positions of the guide path. The anchor point set needs to contain two points: the camera center point, i.e. Where M and N are the horizontal and vertical resolutions of the image respectively; for the target point, the point with the largest depth is selected, i.e. Where X and Y represent the corresponding d max The image coordinates of the pixel where

[0027] Constructing guided paths through an iterative reduction process

[0028] The guided path is constructed by an iterative reduction process The following steps are involved:

[0029] 1) Create an empty deleted set Initialization Path is the interior point set

[0030] 2) In the topology domain Perform step 3) on each of the six neighboring directions above;

[0031] 3) Remember the current neighborhood direction is dir and create an empty deletable set Continue with step 4);

[0032] 4) For each topological domain If point p does not belong to the anchor point set, and is a boundary point or a single point in the dir direction, then point p is added to the removable set. Continue to step 5);

[0033] 5) For each deletable set If point p is a single point, remove point p from the path Delete and add point p to the deleted set Traverse the 6 neighborhoods through steps 2)-5), continue to step 6), otherwise return to step 2);

[0034] 6) If the set has been deleted If it is empty, it ends and the path obtained is the guided path. Otherwise return to step 1).

[0035] The optimization comprises the following steps:

[0036] Convert the discrete guided path into a depth map and apply a correction function to the depth.

[0037] Perform inverse projection mapping on the corrected discrete points to obtain discrete center points in space;

[0038] The discrete center points are smoothed, fitted, and resampled to obtain the optimized guidance path.

[0039] The optimization comprises the following steps:

[0040] For each point on the discrete guided path Use the inverse depth conversion function:

[0041]

[0042] Get (i, j, d1) T ∈L is a point in the depth map domain D, constituting the guiding path L on the depth map D , where d1 is the transformed depth;

[0043] Use a depth correction function to reduce nonlinear effects:

[0044]

[0045] Where α is the correction factor, usually 1.5, and d2 is the corrected depth;

[0046] Then, according to the standard pinhole camera model, the guided path L on the depth map is D Convert to camera space The guiding path

[0047]

[0048] in, is a point on the camera space steering path; I represents the intrinsic parameter matrix.

[0049] Finally, use B-spline to fit the guide path Then resample at uniform intervals to obtain the guided path l.

[0050] The calculation of the geometric information of the guiding path to realize the adaptive guiding control includes the following steps:

[0051] Extracting differential geometric information for optimizing the guide path, including calculating curvature and tangent vectors; the curvature represents the degree of curvature of the guide path and is associated with the bending speed of the endoscope; the tangent vector represents the extension direction of the guide path and is associated with the bending direction of the endoscope and the delivery speed;

[0052] A gain matrix is ​​constructed from the differential geometry of the guiding path l. This gain matrix uses the inner product of the average tangents of the front and rear ends of the guiding path as an indicator parameter and adaptively selects conventional parameters or conditional parameters: when the path curvature is less than a threshold, conventional parameters are used; otherwise, conditional parameters are used, and the curvature and forward speeds of the endoscope are adjusted using a speed adaptation law.

[0053] The target error is mapped to the driving quantity through the gain matrix and kinematic Jacobian, and autonomous control is achieved through constraint parameters.

[0054] The calculation of the geometric information of the guiding path to realize the adaptive guiding control includes the following steps:

[0055] 1) The smooth guide path l is transformed into the parameterized equation l = l(s) in the Frenet coordinate system, where s is the arc length variable;

[0056] The differential geometry information includes the tangent vector t(s):

[0057]

[0058] The differential geometry information includes the curvature κ(s):

[0059]

[0060] The relationship between the target velocity v and the curvature κ(s) is as follows:

[0061]

[0062] Secondly, the coefficient function c=c(s),c>0,c∈[0,1] is introduced, and the target velocity direction v e The relationship with the tangent vector t(s) is as follows:

[0063]

[0064] The target speed is:

[0065]

[0066] Where, length(l) represents the total length of the guided path. The speed gain matrix K is introduced as the control adjustment coefficient, and the target speed after gain is:

[0067] v'=K·v

[0068] 2) The adaptive adjustment parameter E is the cosine difference of the average tangent direction of different segments of the guidance path:

[0069]

[0070] Where L = length (l) is the total length of the guided path, k c represents the segment ratio;

[0071] 3) Predetermine the switching threshold E c , if E is not less than E c , K selects the parameter K u ; On the contrary, select parameter K c ;

[0072] in,

[0073]

[0074] where k bx , k by and k z They represent the reference gain coefficients of the X-direction steering degree of freedom, the Y-direction steering degree of freedom, and the Z-direction transport degree of freedom, respectively, where k z →∞;

[0075] K c Associated with E, it is expressed as:

[0076]

[0077] where k incx , k incy and k dec Has the following form:

[0078] k incx =k bx (1+k G (1-E))

[0079] k incy =k by (1+k G(1-E))

[0080] k dec =k a (1-k S (1-E))

[0081] Among them, k z′ is selected differently from k z The transmission gain parameter, k G and k S Used to determine the response speed of the control;

[0082] 4) According to different selections of K, K is used as the proportional gain coefficient P of the PID control to control the speed of the endoscope in the X-direction steering degree of freedom, the Y-direction steering degree of freedom, and the Z-direction transport degree of freedom.

[0083] The autonomously steerable flexible endoscope control system based on spatial perception includes:

[0084] The three-dimensional space perception module is used to obtain the monocular endoscope image for three-dimensional space perception and obtain the environmental space depth map;

[0085] Path optimization module, used to process the environmental space depth map, extract the guidance path and optimize it;

[0086] The guidance control module is used to calculate the geometric information of the guidance path and realize adaptive guidance control.

[0087] The present invention has the following beneficial effects and advantages:

[0088] 1. The present invention has the ability to perform three-dimensional perception of the space of the endoscope working environment;

[0089] 2. The present invention has the ability to adapt to the spatial structure of the endoscope working environment;

[0090] 3. Under the control implementation of the present invention, a smaller endoscope delivery force and a smoother endoscope delivery effect can be achieved;

[0091] 4. The overall process of the present invention has the advantages of transparency and clear physical information. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 Schematic diagram of the working principle of the present invention;

[0093] Figure 2 Flow chart of the method of the present invention;

[0094] Figure 3 Flowchart of the data synthesis method of the present invention;

[0095] Figure 4 Schematic diagram of the digital topology extraction process;

[0096] Figure 5 Schematic diagram of deep nonlinear problems;

[0097] Figure 6 Block diagram of the control process of the present invention. DETAILED DESCRIPTION

[0098] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0099] like Figure 2 As shown in the figure, a cavity autonomous guidance control strategy based on spatial perception has the following main steps:

[0100] S1. 3D spatial perception from monocular endoscopic images;

[0101] S2. Process the spatial perception results, extract the guidance path and optimize it;

[0102] S3. Calculate the geometric information of the guidance path and implement adaptive guidance control.

[0103] Obtaining depth maps from monocular endoscopic images using an end-to-end supervised deep learning approach.

[0104] The deep learning method has an end-to-end structure, a supervised training method, synthetic training data, and the output is a pixel-dense depth map.

[0105] The end-to-end deep learning method used requires a monocular endoscopic image as input and outputs a pixel-by-pixel depth map of the same resolution.

[0106] The supervised deep learning method used requires a synthetic monocular endoscopic image as input during training, and the output depth map is compared with the synthetic ground-truth depth map to calculate a composite loss function.

[0107] The output depth map is pixel-wise dense, that is, the depth map has the same resolution as the input image and each pixel has a depth value.

[0108] The training data synthesis method includes the following steps:

[0109] Use the prosthetic model to scan and generate a three-dimensional model;

[0110] Add textures to the composite environment and set up virtual paths;

[0111] Let the virtual camera move along the virtual path;

[0112] Obtained by exporting color images and corresponding depth data through scripts.

[0113] The guidance path extraction and optimization method includes the following steps:

[0114] Extracting discrete guided paths from depth maps using digital topology methods.

[0115] The discrete guidance path is processed to obtain an optimized guidance path.

[0116] Extracting a guide path involves the following steps:

[0117] Select the discrete coefficient and depth range, and voxelize the depth map to form a binary three-dimensional depth field grid;

[0118] The binary grid is iteratively reduced using digital topology methods to obtain discrete guided paths.

[0119] Guided path optimization includes the following steps:

[0120] Convert the discrete guided path into a depth map and apply a correction function to the depth.

[0121] Processing guide path;

[0122] The discrete center points are smoothed, fitted, and resampled to obtain the optimized guidance path.

[0123] The adaptive steering control method includes the following steps:

[0124] Extract differential geometry information for optimized guidance paths, including computing curvature and tangent vectors.

[0125] The gain matrix is ​​constructed from the differential geometry information through the mapping relationship.

[0126] The target error is mapped to the driving quantity through the gain matrix and kinematic Jacobian, and the constraint parameters are designed to improve the autonomous control effect.

[0127] Extraction of differential geometric information of the guide path: The curvature represents the degree of curvature of the guide path and is related to the bending speed of the endoscope; the tangent vector represents the extension direction of the guide path and is related to the bending direction and delivery speed of the endoscope.

[0128] Gain matrix construction: The inner product of the average tangent of the front and rear ends of the guide path is calculated as an indicator parameter, and conventional parameters or conditional parameters are adaptively selected: when the path curvature is small, conventional parameters are used to control the stable and rapid delivery of the endoscope; when the path curvature is large, conditional parameters are used to adjust the bending speed and forward speed of the endoscope through the speed adaptation law to avoid deviation from the guide path.

[0129] Constraint parameter design includes the following parameters:

[0130] Minimum curvature condition: when the curvature is too small, the curvature adjustment is disabled;

[0131] Inertia parameters: balance the endoscope's historical operations and current manipulations to ensure the continuity of endoscope behavior.

[0132] like Figure 2 As shown in Figure 2, spatial perception is achieved by reconstructing the approximate surface of the tissue from the monocular endoscopic image. First, we use a neural network to reconstruct the approximate surface of the tissue from the endoscopic image ( Figure 2 The depth map at the pixel level estimated in A) Figure 2 To this end, we construct a synthetic dataset with pixel-level depth maps to train the network. We then use the depth map to derive the lumen guidance path of the current scene, where the lumen space is truncated by the endoscope's field of view. Furthermore, the target is marked as the deepest region in the depth map ( Figure 2 C). The lumen guidance path is obtained by skeletonizing the voxelized depth field through digital topology ( Figure 2 The extracted guide path needs to be smoothed and interpolated ( Figure 2 E) to optimize the robot endoscope steering ( Figure 2 Middle F).

[0133] Figure 2 This paper describes the overall technical route of spatial perception. In addition to the explanation, this paper samples the formula symbols and notes in Table 1.

[0134] Table 1 Description of formula symbols

[0135]

[0136]

[0137] 1. Three-dimensional space perception method

[0138] The three-dimensional space perception method obtains a depth map describing the three-dimensional space from endoscopic images. Specifically, the perception method uses a deep learning neural network, which takes the endoscopic image as input and outputs a depth map. The neural network is trained using a supervised method on synthesized endoscopic images and depth data.

[0139] 1.1. Dataset Construction Method

[0140] The synthetic dataset described in this section is used to train and evaluate the aforementioned deep learning neural networks.

[0141] The dataset construction consists of two main stages: 3D model acquisition and virtual environment construction. The 3D model acquisition stage aims to create a 3D model that simulates the structure of human organs; the virtual environment construction stage aims to give the model a texture material that resembles human tissue, thereby generating the required training dataset that composites the features of human cavities.

[0142] 1.1.1. 3D model acquisition

[0143] The acquisition of 3D models mainly includes the following steps:

[0144] 1. Use CT equipment to collect data from the upper digestive tract dummy model and obtain CT images of the dummy model;

[0145] 2. Using Slicer3D, a professional medical image processing software, the acquired CT images of the mannequin are segmented using a binary threshold, and the contours of the segmented regions are optimized to obtain the model segmentation regions. The segmented regions are then combined layer by layer to obtain the 3D data of the mannequin.

[0146] 3. Export the dummy model as an STL format model file. In the 3D viewing and processing software Meshlab, perform optimization processing on the 3D model by trimming redundant model structures, smoothing the inner and outer surfaces of the model, and so on. At this point, the 3D model is obtained.

[0147] 1.1.2. Virtual environment construction

[0148] Virtual environment construction refers to creating specific camera, lighting, motion path, and other information within a 3D modeling environment, thereby synthesizing images that resemble those found within real human cavities, along with pixel-level depth information. The 3D construction environment used in this example is Blender software.

[0149] like Figure 3 As shown, the main process includes the following steps:

[0150] 1. Import the aforementioned 3D model into the 3D environment, adjust the model's posture, and place it in the appropriate position; select the appropriate surface texture map and assign the map to the 3D model in the "Texture" page;

[0151] 2. Add a new focused light source on the "Modeling" page, set the light source to the origin of the environment, adjust the light source intensity parameters so that the light source is bright enough to illuminate the model cavity, and adjust the light source cone angle to no less than 60°;

[0152] 3. Add a new camera on the "Modeling" page, align the camera center with the light source, align the camera's main viewing direction with the light source direction, point the camera's upward direction toward the appropriate position, set the camera's field of view to 60°, and set the camera's focal length to a reasonable size so that the image content is similar to a conventional endoscopic image. Set the output image to a square and the output resolution to an appropriate number. Change the hierarchical relationship between the camera and the light source so that the light source becomes a child of the camera, forming a fixed connection.

[0153] 4. Add a new path on the "Modeling" page. Add a series of points in the 3D model cavity at a certain distance to the path to form the path. Set the path to spline smoothing. Place the aforementioned camera object at the starting point of the path. Add a new constraint condition in the "Geometry" property tab of the camera, add a path constraint, and set the main direction of the camera to coincide with the tangent direction of the path.

[0154] 5. Import the rendering script. In the script, set the rendering results, depth data of the image scene, and camera parameters to be read during the animation rendering process. Set the data to be saved in the specified format to the set output path. The image data is saved in PNG format, the depth data is saved in NPY format, and the camera information is saved in TXT format.

[0155] 6. Under the path's "Object Data" properties, set the number of path frame segments to an appropriate value. Under the camera's "Animation" properties, set the camera animation step size to 1 and the animation frame range to be within the path frame segment range. This will control the amount of data in the output dataset. In the public "Export" properties, set the dataset output path, output image format, resolution, and other parameters, and enable the aforementioned rendering script.

[0156] 7. Run the animation rendering function of the 3D software. The script will automatically export image, depth, camera parameters and other data, and the dataset is constructed.

[0157] 1.2. Network training and application

[0158] This embodiment uses an existing end-to-end UNet network (hereinafter referred to as the depth estimation network) for depth estimation. The network backbone uses the ResNet18 network. The depth estimation neural network structure is shown in Table 2.

[0159] Table 2

[0160]

[0161]

[0162] Note:

[0163] a) k: convolution kernel size, s: convolution step size, c: number of channels, r: resolution factor;

[0164] b) C = convolutional block, B = batch normalization, R18 = ResNet-18;

[0165] c) u* = 2x nearest neighbor upsampling block, m* = residual fusion input block.

[0166] The depth estimation network input is a three-channel endoscopic image I, stored in RGB format. The input image's pixel values ​​are normalized to a range of 0 to 1, with a required resolution of 320×320. Images that do not meet the required resolution are cropped and resampled to meet the required resolution. The depth estimation network outputs a depth map D with the same resolution as the input image.

[0167] The above depth estimation method can be summarized as a three-dimensional space perception (depth estimation) function depth(·), that is:

[0168] D = depth (I)

[0169] The depth estimation network described above was built and trained using the PyTorch environment. Training was performed using the Adam trainer with default parameters, 50 training cycles, and a starting learning rate of 0.001. Training was performed using a single RTX2080 GPU.

[0170] The training is supervised, and the output depth of the depth estimation network is compared with the true depth of the dataset to calculate the loss function. The loss function consists of two parts: the first part is the 2-norm of the depth value, that is, The second part is the 2-norm of the depth space gradient, i.e. in is the gradient operator in the X direction, It is the gradient operator in the Y direction, and the final loss function is L = L depth +αL grad , where α is the loss weight parameter, which can be set freely. The loss values ​​calculated for all pixels are averaged over the image domain to obtain the final loss value.

[0171] 2. Guide path extraction

[0172] This section describes the method of extracting the guide path from the depth map. First, the depth map D is discretized and voxelized to obtain a binary three-dimensional grid digital topology map. Then use the digital topology reduction method to reduce the topology graph Get discrete guided paths from Finally, for discrete guided paths Fit, smooth and resample to obtain a smooth guiding path

[0173] 2.1. Voxelization

[0174] The first step in guiding path extraction is to voxelize the depth map D to obtain a digital topology map The core of digital topological graph is to construct the interior point set on the topological domain Interior point set Refers to the set of topological domain grids corresponding to spatial regions smaller than the depth value on the depth map.

[0175] Voxelization first corrects the depth map D based on the depth conversion function. For a point ((i, j) on the depth map D T ,d), select a certain discretization resolution parameter S, and define the depth conversion function as:

[0176]

[0177] So, get is an interior point of the topological graph.

[0178] Then, according to the definition of digital topology, for the interior point set Perform a test and delete some points that do not belong to the (26,6) topology graph, and then you will get the required digital topology graph.

[0179] 2.2. Guided Path Extraction

[0180] Guided path extraction is done by analyzing the digital topology graph The discrete guided path is obtained by reducing

[0181] First, in the interior point set Get an anchor point set The anchor point set is a point that cannot be removed during topology reduction and is used to constrain the start and end positions of the guide path. Specifically, the anchor point set needs to contain two points: one is the camera center point, that is, Where M and N are the horizontal and vertical resolutions of the image respectively; the second is the target point, which is usually the point with the largest depth, that is Where X and Y represent the corresponding d max The image coordinates of the pixel.

[0182] Second, the guided path is constructed through an iterative reduction process like Figure 4 Specifically, the following steps are performed:

[0183] 1. Create an empty deleted set Initialization Path is the interior point set

[0184] 2. In the field Perform step 3 on each of the six neighboring directions above;

[0185] 3. Remember the current neighborhood direction is dir and create an empty deletable set Continue to step 4;

[0186] 4. For each topological domain For a point p that does not belong to the anchor point set, if point p is a boundary point and a single point in the dir direction, then point p is added to the deletable set Continue to step 5;

[0187] 5. For each deletable set For point p in the path, if point p is a single point, remove point p from the path set Delete and add point p to the deleted set If all 6 neighborhoods have been checked, continue to step 6, otherwise return to step 2;

[0188] 6. If the set has been deleted If it is empty, then end; otherwise, return to step 1.

[0189] Through the above digital topology reduction process, the discrete guided path is obtained The discrete guided path at this time It is a point set consisting of a series of connected points between two anchor points.

[0190] 2.3. Path Optimization

[0191] In this section, we will introduce the discrete guided paths Optimized for smooth guide paths The following steps are involved:

[0192] First, according to the depth conversion function, the depth conversion inverse function can be obtained. For each point on the discrete guide path Use the inverse depth conversion function:

[0193]

[0194] Thus, we get (i,j,d1) T ∈L is a point in the depth map domain D, which constitutes the guiding path L on the depth map D , where d1 is the transformed depth.

[0195] The camera's field of view is a quadrangular pyramid space, with the camera located at the top of the pyramid. Therefore, for the actual distance between two pixels corresponding to the same depth, the greater the depth, the greater the difference. This makes our guidance path sparser in distant areas, such as Figure 5 Schematic diagram of a deep nonlinear problem, δ1: near-end grid spacing; δ2: far-end grid spacing, and larger fluctuations.

[0196] In order to reduce the impact of this nonlinearity in the depth direction, the following depth correction function is introduced:

[0197]

[0198] Where α is the correction factor, usually 1.5, and d2 is the corrected depth.

[0199] Then, according to the standard pinhole camera model, the guided path L on the depth map can be D Convert to camera space The guiding path

[0200]

[0201] in, is a point on the camera space guide path.

[0202] Finally, use B-spline to fit the guide path Then resampling at uniform intervals can obtain a smooth guided path l. We use smooth(·) to represent this process, that is:

[0203]

[0204] The above steps complete the acquisition of the smooth guide path l.

[0205] 3. Adaptive guidance control

[0206] This section presents an autonomous steering control method for a flexible endoscope based on the extracted smooth guidance path. Specifically, it includes steps and components such as motion relationship mapping, extracting differential geometry information from the guidance path, constructing the control loop gain matrix, and adjusting constraint parameters.

[0207] 3.1. Relationship between hardware structure and kinematics

[0208] The flexible endoscope operating mechanism is a prior art; a flexible endoscope control robot (application number 202011370492.5) can also be used, wherein the X-rotation free q bx , Y turns to free q by , transport freedom q a They respectively correspond to the first motor, second motor, and driving wheel of the mechanism described in a flexible endoscope manipulation robot (application number 202011370492.5).

[0209] Each driving variable of the flexible endoscope operating part mechanism is synthesized into a driving vector q=[q bx ,q by ,q a ] T Each driving quantity of q is restricted to the range [-1,1]. All possible q are located in the driving space Up, that is

[0210] This paper realizes the drive control of flexible endoscope based on speed control. The forward kinematics of flexible endoscope drive can be expressed as the mapping relationship from drive space to task space (equivalent to camera space in camera coordinate system) Through the first-order derivative, the velocity mapping relationship is obtained as the Jacobian matrix:

[0211]

[0212] The J in this paper uses empirical results to realize the drive control as follows:

[0213]

[0214] Among them, J + is the inverse or Moore-Penrose pseudoinverse of J.

[0215] 3.2. Differential geometry information

[0216] This section calculates the differential geometry information from the smooth guided path l to construct the gain matrix for the next step, thereby determining the target velocity Recorded as The subscript e Indicates v e is the unit target velocity direction vector, and v is the scalar target velocity magnitude.

[0217] First, we transform the smooth guide path l into a parameterized equation l=l(s) in the Frenet coordinate system, where s is the arc length variable.

[0218] The first part of the differential geometry information is the tangent vector t(s):

[0219]

[0220] The second part of differential geometry information is the curvature κ(s):

[0221]

[0222] The control method in this paper designs the relationship between the target speed and the average curvature:

[0223]

[0224] Secondly, the target velocity direction is related to the tangent direction. Introducing the coefficient function c = c(s), c > 0, c ∈ [0, 1], the target velocity direction is:

[0225]

[0226] In summary, after omitting the parameter s, the target speed can be expressed as:

[0227]

[0228] Where K is the gain matrix for controlling the velocity, and length(l) represents the total length of the guided path.

[0229] 3.3. Gain matrix

[0230] This paper designs a gain matrix to adjust the speed response characteristics, and the method is as follows:

[0231] v'=K·v

[0232] Among them, v' is the target speed after gain. This paper designs the control scheme in two parts, namely the conventional parameter K used in most cases. u , and the conditional parameter K used in special cases c , and switch the parameter selection scheme by adaptively adjusting the parameter E.

[0233] The adaptive adjustment parameter is the cosine difference of the average tangent direction of different segments of the guidance path, that is,

[0234]

[0235] Where L = length (l) is the total length of the guided path, k c Indicates the segment ratio.

[0236] Predetermined switching threshold E c , if E is not less than E c , select parameter K u Anyway, choose K c .

[0237] K u It can be viewed as a constant matrix as follows:

[0238]

[0239] where k bx , k by and k z represents the reference gain coefficient for the corresponding degree of freedom, where k z →∞.

[0240] K c Associated with E, it can be expressed as the following matrix:

[0241]

[0242] where k incx , k incy and k dec Has the following form:

[0243] k incx =k bx (1+k G (1-E))

[0244] k incy =k by (1+k G (1-E))

[0245] k dec =k a (1-k S (1-E))

[0246] where k z′ is selected differently from k z The transmission gain parameter, k G and k S Used to determine the response speed of the control.

[0247] 3.4. Constraint Parameters

[0248] This section designs several additional constraints to improve the control effect:

[0249] 1. Minimum curvature Used to disable bending when the path is slightly curved, that is, to set the bending component v' of v' x and v' y is 0;

[0250] 2. The inertia parameter m is used to balance the historical operation and the target operation, that is, v' = m·v' k-1 +(1-m)·v' k , to output a smooth control amount.

[0251] In summary, the control output can be expressed as:

[0252]

[0253] Among them, K is adaptively selected according to E as K u or K c The overall control scheme diagram of the autonomous navigation framework can be seen Figure 6 .

Claims

1. A method for controlling an autonomously steerable flexible endoscope based on spatial perception, characterized in that: The following steps are involved: S1. Obtain a monocular endoscopic image for three-dimensional spatial perception and obtain a depth map of the environmental space; S2. Process the environment space depth map, extract the guidance path and optimize it; S3, calculate the geometric information of the guidance path to achieve adaptive guidance control; The optimization comprises the following steps: Convert the discrete guided path into a depth map and apply a correction function to the depth. Perform inverse projection mapping on the corrected discrete points to obtain discrete center points in space; The discrete center points are smoothed, fitted, and resampled to obtain the optimized guidance path.

2. The method for controlling an autonomously steerable flexible endoscope based on spatial perception according to claim 1, characterized in that: The method of acquiring a monocular endoscopic image for three-dimensional spatial perception and obtaining an environmental spatial depth map adopts an end-to-end supervised deep learning method, including the following steps: The end-to-end supervised deep learning method takes a monocular endoscopic image as input and outputs a pixel-by-pixel depth map of the same resolution; During training, the synthesized monocular endoscopic image is used as input, and the output depth map is compared with the synthesized ground-truth depth map to calculate the composite loss function. The training data used during training is obtained through the following steps: Obtain a three-dimensional model of the environment space; Add textures in the environment space and set virtual paths; Let the virtual camera move along the virtual path; Export color images and corresponding depth data through scripts.

3. The method for controlling an autonomously steerable flexible endoscope based on spatial perception according to claim 1, characterized in that: The extraction guide path comprises the following steps: A discrete coefficient and a depth range are selected, and the depth map is voxelized to form a binary three-dimensional depth field grid. The binary three-dimensional depth field grid is iteratively reduced using a digital topology method to obtain a discrete guided path.

4. The method for controlling an autonomously steerable flexible endoscope based on spatial perception according to claim 1 or 3, characterized in that: The extraction guide path comprises the following steps: First, the depth map D is corrected based on the depth conversion function: for the point ((i, j) on the depth map D T ,d), select the discretization resolution parameter S, then the depth conversion function is: Get (i, j, p) T ∈X is an interior point of the topological graph; i and j represent the point position, d represents the depth, and d max d min Respectively represent the maximum depth and minimum depth of the depth map, Represents an integer; Then, according to the definition of digital topology, the inner point set X is checked and some points that do not belong to the (26,6) topology map are deleted to obtain the required digital topology map M = (P, 26, 6, X); the inner point set X is the topological domain grid set corresponding to the spatial area less than the depth value on the depth map; The anchor point set A is obtained from the inner point set X. The anchor point set is a set of points that cannot be removed during topological graph reduction and is used to constrain the start and end positions of the guided path. The anchor point set needs to contain two points: the camera center point, i.e. Where M and N are the horizontal and vertical resolutions of the image respectively; for the target point, the point with the largest depth is selected, i.e. Where X and Y represent the corresponding d max The image coordinates of the pixel where The guided path S is constructed through an iterative reduction process.

5. The method for controlling an autonomously steerable flexible endoscope based on spatial perception according to claim 4, characterized in that: The construction of the guide path S by the iterative reduction process comprises the following steps: 1) Create an empty deleted set Initialize the path S to the interior point set X; 2) Perform step 3) in each of the six neighborhood directions on the topological domain P; 3) Record the current neighborhood direction as dir, create an empty deletable set K, and continue with step 4); 4) For each point p on the topological domain P that does not belong to the anchor point set, if point p is a boundary point and a single point in the dir direction, then add point p to the deletable set K and continue with step 5); 5) For each point p in the deletable set K, if point p is a single point, delete point p from the path S and add point p to the deleted set Traverse the 6 neighborhoods through steps 2)-5), continue to step 6), otherwise return to step 2); 6) If the set has been deleted If it is empty, the process ends and the path obtained is the guided path S; otherwise, return to step 1).

6. The method for controlling an autonomously steerable flexible endoscope based on spatial perception according to claim 1, characterized in that: The optimization comprises the following steps: For each point (i, j, p) on the discrete guided path T ∈S, use the inverse depth conversion function: Get (i, j, d1) T ∈L is a point in the depth map domain D, constituting the guiding path L on the depth map D , where d1 is the transformed depth; Use a depth correction function to reduce nonlinear effects: Where α is the correction factor, usually 1.5, and d2 is the corrected depth; Then, according to the standard pinhole camera model, the guided path L on the depth map is D Transform the guide path L to camera space C: Among them, x c ∈L is a point on the camera space steering path; I represents the intrinsic parameter matrix; Finally, the guiding path L is fitted using B-spline and then resampled at uniform intervals to obtain the guiding path l.

7. The method for controlling an autonomously steerable flexible endoscope based on spatial perception according to claim 1, characterized in that: Calculating the geometric information of the guide path to achieve adaptive guide control includes the following steps: extracting differential geometric information of the optimized guide path, including calculating curvature and tangent vectors; the curvature represents the degree of curvature of the guide path and is associated with the bending speed of the endoscope; the tangent vector represents the extension direction of the guide path and is associated with the bending direction of the endoscope and the conveying speed; A gain matrix is ​​constructed from the differential geometry of the guiding path l. This gain matrix uses the inner product of the average tangents of the front and rear ends of the guiding path as an indicator parameter and adaptively selects conventional parameters or conditional parameters: when the path curvature is less than a threshold, conventional parameters are used; otherwise, conditional parameters are used, and the curvature and forward speeds of the endoscope are adjusted using a speed adaptation law. The target error is mapped to the driving quantity through the gain matrix and kinematic Jacobian, and autonomous control is achieved through constraint parameters.

8. The method for controlling an autonomously steerable flexible endoscope based on spatial perception according to claim 1 or 7, characterized in that: The calculation of the geometric information of the guiding path to realize the adaptive guiding control includes the following steps: (1) The smooth guide path l is transformed into the parameterized equation l = l(s) in the Frenet coordinate system, where s is the arc length variable; The differential geometry information includes the tangent vector t(s): Differential geometry information includes curvature κ(s): The relationship between the target velocity v and the curvature κ(s) is as follows: Secondly, the coefficient function c=c(s),c>0,c∈[0,1] is introduced, and the target velocity direction v e The relationship with the tangent vector t(s) is as follows: The target speed is: Where, length(l) represents the total length of the guided path. The speed gain matrix K is introduced as the control adjustment coefficient, and the target speed after gain is: v'=K·v (2) The adaptive adjustment parameter E is the cosine difference of the average tangent direction of different segments of the guidance path: Where L = length (l) is the total length of the guided path, k c represents the segment ratio; (3) Predetermine the switching threshold E c , if E is not less than E c , K selects the parameter K u ; On the contrary, select parameter K c ; in, where k bx , k by and k z They represent the reference gain coefficients of the X-direction steering degree of freedom, the Y-direction steering degree of freedom, and the Z-direction transport degree of freedom, respectively, where k z →∞; K c Associated with E, it is expressed as: where k incx , k incy and k dec Has the following form: k incx =k bx ·(1+k G ·(1-E)) k incy =k by ·(1+k G ·(1-E)) k dec =k a ·(1-k S ·(1-E)) Among them, k z′ is selected differently from k z The transmission gain parameter, k G and k S Used to determine the response speed of the control; (4) According to the selection of different K, K is used as the proportional gain coefficient P of PID control to control the speed of the endoscope in the X-direction steering freedom, Y-direction steering freedom and Z-direction transport freedom.

9. An autonomously steerable flexible endoscope control system based on spatial perception, characterized in that: include: The three-dimensional space perception module is used to obtain the monocular endoscope image for three-dimensional space perception and obtain the environmental space depth map; The path optimization module is used to process the environmental space depth map, extract the guidance path and optimize it; The optimization is configured to perform: Convert the discrete guided path into a depth map and apply a correction function to the depth. Perform inverse projection mapping on the corrected discrete points to obtain discrete center points in space; Smoothing, fitting and resampling discrete center points to obtain an optimized guidance path; The guidance control module is used to calculate the geometric information of the guidance path and realize adaptive guidance control.

Images