Double-cavity catheter positioning system based on image registration

Abstract

The invention discloses a double-cavity catheter positioning system based on image registration, and belongs to the field of medical instruments. A double-cavity catheter positioning system based on image registration comprises a gas guide tube provided with a gas channel; the multiple air bags are arranged on the outer side of the air guide pipe in the axial direction of the air guide pipe; the air supply module is used for supplying air to the air bags so as to control the internal air pressure of each air bag; the miniature endoscope lenses are arranged at the end of the air guide pipe in a circumferential array mode. In the technical scheme provided by the invention, the micro endoscope lens can collect the image information of the inner wall of the bronchial tube so as to judge where pathology exists on the inner wall of the bronchial tube, so that part of the air bag is controlled to expand, the expanded air bag avoids the pathology position, and the air bag does not extrude the pathology part of the bronchial tube.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and more particularly, to a dual-lumen catheter positioning system based on image registration. Background Art

[0002] The double-lumen catheter for the lungs, namely the double-lumen bronchial catheter (DLT), is a special tracheal catheter mainly used to separate the two bronchi during one-lung ventilation for surgical operations or diagnostic examinations. The double-lumen catheter generally includes a trachea tube and a support tube. The support tube is arranged outside the trachea tube. The trachea tube is inserted into the bronchus, and the end is located at the end of the bronchus, with the support tube outside the trachea tube. When in use, the double-lumen catheter is inserted into the bronchus, and then gas is injected into the support tube. The support tube is made of a flexible material and expands after inflation, thereby blocking the entire bronchus. In this way, only the trachea tube can be used for gas exchange in this bronchus, and subsequent surgical operations can be carried out.

[0003] The arrangement of the double-lumen catheter includes two key parts. The first part: the end of the trachea tube is aligned with the outlet of the bronchus (the opening where gas enters the lungs) to prevent the outlet of the trachea tube from being unable to send gas into the lungs or the gas in the lungs from being unable to be discharged through the trachea tube. The second part: enough gas needs to be pumped into the support tube to ensure that the support tube fits tightly with the bronchus to ensure sealing.

[0004] Therefore, when the double-lumen catheter is inserted into the lungs, a camera is provided at the end of the double-lumen catheter to obtain the position of the end of the double-lumen catheter, and then the end of the trachea tube is aligned with the outlet of the bronchus. However, when arranging the support tube, if the air pressure of the support tube is directly increased, it is very easy to apply excessive pressure to some pathological parts on the inner wall of the bronchus, which will increase the damage to the bronchus. Summary of the Invention

[0005] The content part of the present application is used to briefly introduce the concepts, which will be described in detail in the following detailed implementation part. The content part of the present application is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] In order to solve the problem that it is easy to aggravate the pathological parts of the bronchus when inserting the double-lumen catheter, the present application provides a dual-lumen catheter positioning system based on image registration, including: A trachea tube, configured with a gas passage; An airbag array, with multiple flexible airbags distributed at intervals along the axial direction of the trachea tube; An air supply module, used to supply air to the airbags to control the internal air pressure of each airbag; A micro-endoscope lens, with multiple lenses arranged in a circumferential array at the end of the trachea tube; A control device, used for controlling the airway to move to the bronchial entrance; An image stitching device, used to obtain the video information collected by each micro-endoscope lens to stitch out a panoramic image of the bronchial inner wall; An image recognition device extracts the pathological locations of the bronchial inner wall from the panoramic image and calculates the tracheal location of each pathological location; The control device controls the expansion of the corresponding airbag based on the tracheal position of each pathological position, so that the inflated airbag is staggered to each pathological position.

[0007] In the technical solution provided in the present application, the micro-endoscope lens can collect image information of the inner wall of the bronchus, and then determine the pathological position on the inner wall of the bronchus, thereby controlling the expansion of part of the airbag, so that the inflated airbag avoids the pathological position, so the airbag will not cause squeezing with the pathological part of the bronchus. At the same time, because multiple airbags are inflated together, they can have a multi-layer sealing effect to increase airtightness.

[0008] Furthermore, the gas supply module comprises: There are multiple air supply pipes, each of which corresponds to an air bag; A gas source assembly, used to generate gas or inhale gas; The matrix reversing valve is connected to the air supply pipe and the air source respectively, and the air source is connected to the corresponding airbag through the reversing valve.

[0009] Furthermore, at least four micro-endoscope lenses are provided.

[0010] In this solution, four micro-endoscope lenses are provided, so that the four cameras can not only observe the entire inner wall of the bronchus, but also have many overlapping fields of view. Therefore, for each camera, it is only necessary to observe the surface image of the bronchus facing the lens, thus avoiding the influence of the peripheral image caused by the bending of the bronchus.

[0011] Because the bronchus is a tubular structure, it is easy to lose some image information. For this reason, this application provides the following technical solutions: The image stitching device comprises: An image information collection module is used to obtain video information of each micro-endoscope lens; A horizontal splicing module obtains video information from each micro-endoscope lens, extracts original images at the same time from the video information, and splices the original images into a ring image; The vertical stitching module obtains all the annular images and stitches the annular images together to obtain a panoramic image.

[0012] In the technical solution provided in the present application, the original images at the same time are extracted from each micro-endoscope lens, the original images are spliced ​​into a ring image, and then all the ring images are spliced ​​together to obtain a panoramic image of the entire bronchial inner wall; this splicing method can obtain a complete and accurate image of the inner wall of the entire bronchus.

[0013] When stitching the annular images, because the bronchi are not regular straight circular tube structures, the distance between each micro-endoscope lens and the inner wall of the bronchus is not equal, and thus the real-world length corresponding to each pixel point in the image obtained by each micro-endoscope lens is not consistent, and thus after the images are stitched together, the real positions of the various areas in the image cannot be accurately reflected. To this end, the present application provides the following technical solutions: The horizontal splicing module includes: A plurality of distance sensors are provided, each of which is provided on a lens of each micro-endoscope, and is used to measure the distance between the lens of the micro-endoscope and the inner wall of the bronchus; An image extraction unit extracts original images from the video information of each micro-endoscope lens; An image regularization unit, which obtains the original image of each micro-endoscope lens and the distance between each micro-endoscope lens and the bronchus, and regularizes the original image into a standard pixel grid with a fixed pixel ratio, so that the real-world distance corresponding to the pixel grid of each original image is the same; The image stitching unit stitches the original images of adjacent micro-endoscope lenses to obtain a ring image.

[0014] In the technical solution provided in the present application, the original image is processed using a standard pixel grid, so the corresponding distances of the pixel grids in each original image in the real world are equal. After the images are spliced ​​into a panoramic image, the pathological position of the bronchus can be accurately located based on the corresponding distances of the pixel grids in the real world.

[0015] Because there are many blurred images in the video information, if this information is used as the original image, it will lead to the inability to accurately stitch the images and the inability to accurately identify the images; for this reason, the present application provides the following technical solutions: The picture extraction unit extracts the original picture using the following steps: S1: From each video information Q a Intercept video stream q a , where a represents the index of the micro endoscope lens; S2: video stream q a Each image in is gray-scaled and Gaussian filtered; S3: Perform Fourier transform on the image; ; Among them, f(x, y) is the given picture, x and y are the coordinates of the pixels for Fourier transform, F(u, v) is the picture after Fourier transform, u and v are frequency variables, j is the imaginary unit, and e is the natural constant; Calculate the initial clarity B of each picture based on the amplitude |F(u, v)|; ; Among them, is the real part of the Fourier transform result, is the imaginary part of the Fourier transform result, and w is the conversion coefficient; S4: The preset clarity threshold B 0 , filter out the pictures with clarity lower than the clarity threshold B 0 to obtain the remaining image set; S5: Perform edge detection on the pictures in the remaining image set, and select the picture with the maximum clarity in the preset splicing area as the original picture.

[0016] In the technical solution provided by this application, by performing Fourier transform on the pictures, the clarity of the pictures can be roughly judged first, the relatively clear pictures can be screened out, and then edge detection can be performed to further detect the clarity of the splicing area to ensure the splicing quality.

[0017] Furthermore, S5 includes the following steps: S51: Obtain the preset splicing area of each picture; S52: Calculate the amplitude and direction of each pixel point in the preset splicing area; G x (x, y) = I fi (x, y) * Sob x ; G y (x, y) = I fi (x, y) * Sob y ; ; ; Among them, x and y are the abscissa and ordinate of the pixel, I fi (x, y) is the pixel value of the image after Gaussian filtering at the coordinate (x, y), G x (x, y) and G y (x, y) respectively represent the gradient values of the image in the x direction and y direction at the coordinate (x, y), M(x, y) represents the gradient intensity of (x, y) in the gradient amplitude image, and θ(x, y) represents the gradient direction of (x, y) in the gradient direction image; Sob represents the Sobel operator, which is a filter kernel for calculating the image gradient, Sob xThe operator representing the x direction, Sob y The operator representing the y direction; S53: Traverse the gradient magnitude image. For each pixel, check the two adjacent pixels in its gradient direction; If the gradient magnitude of the current pixel is not the local maximum in its direction, set it to 0; Set the high threshold C 1 and the low threshold C 0 , C 1 = 2C 0 ; Mark the pixels with gradient magnitude greater than C 1 as strong edges, the pixels less than C 0 as non - edges, and the pixels between the two as weak edges; Starting from the strong edges, connect along the weak edges to form a complete edge contour; If a weak - edge pixel is connected to a strong - edge pixel, or indirectly connected to a strong - edge pixel through other weak - edge pixels, mark it as a strong edge as well to obtain all edge components; S54: Calculate the sharpness coefficient D of each edge; ; where M is the number of edge pixels, m is the index of the edge pixel, H ed1 (m) and H ed2 (m) represent the pixel values on both sides of the m - th edge pixel respectively.

[0018] In the technical solution provided by this application, through the edge detection algorithm, the edge region for marking the characteristics of the picture object is found in the preset stitching area of the picture. If the object boundary in the edge region is complete, it indicates that the picture in this region is clear, and the accuracy is higher in subsequent picture registration, and it can better match the corresponding region.

[0019] When performing image stitching, in fact, it is to find the overlapping area between two adjacent pictures, and then align the pictures in the overlapping area according to the matching feature points. For this reason, this application determines the preset stitching area based on the following scheme; The boundary of the preset area in S51 is the straight line L 1 and the straight line L 2 , L 1 and L 2 are parallel to each other, and L 1 is the outer boundary of the preset area; The distance between the straight line L 1 and L 2 is L 3 , L 3 = d 1×L 0 ; where d 1 is the first preset coefficient, which is positively correlated with the distance between the micro-endoscope lens and the inner wall of the bronchus. 0 is the first preset standard value; L 1 The distance from the center of the image is L 4 , L 4 =d 2 ×L 5 , where d 2 is the second preset coefficient, which is negatively correlated with the distance between the micro-endoscope lens and the inner wall of the bronchus. 5 It is the second preset standard value.

[0020] In the solution provided in the present application, when the distance between the micro-endoscope lens and the inner wall of the bronchus is closer, the preset boundary range will be smaller, and the preset area will be further away from the central area. Conversely, the preset boundary range will be larger and closer to the central area, so that more useful information can be accurately aligned from the image.

[0021] Furthermore, in S1, from each video information Q a Intercept video stream q a When the distance moved by the current micro-endoscope lens is less than the preset value, the images collected during the time when the micro-endoscope lens moves less than the preset value are used as the video stream.

[0022] In this solution, images with a moving distance less than a preset value are collected as video streams, so the image information collected during this period is, to a certain extent, information at the same height in the bronchus, and can be registered together within the allowable error. In this way, the selection range when splicing the annular image can be increased, and the images in the annular image can be as clear as possible.

[0023] The image regularization unit regularizes the original image into a standard pixel grid with a fixed pixel ratio, including the following steps: S01: Preset a standard pixel grid. The real-world length of each pixel in the standard pixel grid is l 0 ; S02: Calculate the real-world length of the pixel grid in the original image l a , l a represents the length of the pixel grid in the original image of the a-th micro-endoscope lens; ; in, d 0 Indicates the distance between the micro endoscope lens and the inner wall of the bronchus.f 0 Indicates the focal length of the microendoscope lens Indicates the length of the pixel grid in the original image within the image; S03: Based on l a and l 0 The length relationship, regularize the original image into a standard pixel grid.

[0024] In the solution provided by this application, by calculating the pre-acquired object distance, the length of each pixel grid in the real world can be accurately calculated, and then it can be regularized into a standard pixel grid of the same format. Thus, when the lengths of the microendoscope lenses from the bronchus are different, the images can also be processed together in a standard manner. Description of the Drawings

[0025] Figure 1 Is a schematic structural diagram of an air duct; Figure 2 Is a schematic structural diagram of an image stitching device.

[0026] Reference Signs:

[0027] 1. Air duct; 2. Airbag. Detailed Embodiments

[0028] Hereinafter, embodiments of the present application will be described in more detail with reference to the drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not used to limit the protection scope of the present application.

[0029] In addition, it should also be noted that for the sake of convenience of description, only parts related to the relevant invention are shown in the drawings. Without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0030] Hereinafter, the present application will be described in detail with reference to the drawings and in combination with the embodiments.

[0031] Refer to Figure 1, the double-lumen catheter positioning system based on image registration includes: an air duct, an airbag array, a gas supply module, a multi-ocular vision acquisition unit, and a control device. The air duct adopts a hollow tubular structure, and the airbag array is composed of multiple flexible capsules spaced along the axial direction of the air duct. In clinical applications, the air duct has both the functions of a surgical channel and ventilation. Each airbag is made of a biocompatible polymer material, and after inflation, it generates radial expansion to achieve contact sealing with the inner wall of the bronchus, forming a closed ventilation cavity for the target bronchus. By selecting airbags at different axial positions for inflation, it is possible to actively avoid pathological areas in the bronchus and effectively reduce the mechanical damage that a traditional single-airbag catheter may cause to the lesion site.

[0032] The gas supply module includes an air supply pipe, a gas source assembly, and a matrix-type reversing valve. Each airbag corresponds to an independently arranged air supply pipe, and the air supply pipes are arranged in an embedded manner along the circumferential direction of the outer wall of the air duct. They are made of a polyurethane composite material and have excellent anti-bending performance while ensuring the smoothness of the lumen. The gas source assembly integrates a positive / negative pressure generating device to achieve precise inflation and deflation control of the airbag through air pressure adjustment. The matrix-type reversing valve adopts a multi-channel solenoid valve group, and through circuit control, it realizes the connection selection between the gas source and the target airbag. This component is arranged in the proximal operation area of the catheter.

[0033] The multi-ocular vision acquisition unit includes 6 micro-endoscope lenses, and the 6 micro-endoscope lenses are arranged in an equiangular distribution of 60° on the distal end face of the catheter. This layout design can synchronously obtain a 360° panoramic view of the inner wall of the bronchus without rotating the catheter. Each lens is equipped with an adaptive light compensation system to ensure the quality of in-cavity image acquisition.

[0034] The control device integrates a catheter propulsion mechanism and an air circuit control system, and uses a master-slave robotic arm to achieve precise pose adjustment of the catheter. The operation end is equipped with a force feedback device, which can real-time sense the contact stress between the catheter and the tissue, and realize safe intubation in combination with visual navigation.

[0035] The innovation of this system lies in: on the basis of the traditional double-lumen catheter structure, through the axial array design of multiple airbags combined with panoramic visual navigation, it realizes precise occlusion with lesion avoidance. Its core technologies include: Panoramic image reconstruction system: Through an image stitching device, three-dimensional registration is performed on the real-time video streams collected by each lens, and a panoramic unfolded map of the inner wall of the bronchus is generated using a feature point matching algorithm. By establishing the mapping relationship between the image coordinate system and the physical coordinate system of the catheter, the conversion from the pixel coordinates of the panoramic image to the actual anatomical position is realized.

[0036] Airbag optimization selection algorithm: According to the identified spatial distribution of the lesions, the optimal occlusion plan is calculated. This algorithm takes lesion avoidance as a constraint condition and the anatomical adaptability of the occluded bronchus segment as an optimization goal, and automatically recommends the best combination of inflated airbags.

[0037] This system realizes sub - millimeter spatial mapping of bronchial anatomical structures (registration error < 0.5mm) through non - rigid registration of preoperative CT images and intraoperative real - time panoramic images. Combining with the fiber - optic shape sensor (sampling frequency 100Hz) built into the catheter, it can calculate the three - dimensional shape of the trachea in real time, and finally realize the dynamic matching of the spatial relationship between the lesion location and the balloon array.

[0038] Reference Figure 2 , the method for the image stitching device to obtain the panoramic image is introduced as follows: The image stitching device includes: an image information collection module, a horizontal stitching module, and a vertical stitching module.

[0039] The image information collection module is used to obtain the video information of each micro - endoscope lens; The horizontal stitching module obtains the video information of each micro - endoscope lens, extracts the original pictures at the same moment from the video information, and stitches the original pictures into a circular image; The vertical stitching module obtains all the circular images and stitches the circular images together to obtain the panoramic image.

[0040] The horizontal stitching module includes a distance sensor and a picture extraction unit. Among them, multiple distance sensors are provided, and 1 distance sensor is provided on each micro - endoscope lens. The distance sensor is used to measure the distance between the micro - endoscope lens and the inner wall of the bronchus. The picture extraction unit extracts the original pictures from the video information of each micro - endoscope lens. Here, the original pictures refer to the pictures at the same horizontal position taken by each micro - endoscope lens in the bronchus.

[0041] When the picture extraction unit extracts the original pictures, the following steps are adopted: S1: Intercept the video stream q a from the video information Q a , where a represents the index of the micro - endoscope lens.

[0042] Specifically, Q a represents the video information of the a - th micro - endoscope lens, and q a represents the video stream of the a - th micro - endoscope lens.

[0043] The video information is composed of image frames arranged in time sequence, and its frame rate is synchronized with the imaging sampling rate of the micro - endoscope. This system adopts a standard acquisition rate of 60fps (frames per second), that is, 60 RGB image data streams with a resolution of 1920×1080 are generated per second. The total duration of the video information strictly corresponds to the continuous working time of the device, and completely records the in - cavity dynamic images during the catheter advancement process. The video stream is a continuous subsequence extracted from the video information through the time - window sliding algorithm.

[0044] Intercept the video stream q a from each video information Qa When, based on the distance that the current micro-endoscope lens moves; the images collected within the time when the moving distance of the micro-endoscope lens is less than a preset value are used as a video stream.

[0045] Specifically, if the preset value is 1 mm and the time for the micro-endoscope lens to move 1 mm is 2 seconds, then the length of the time window is set to 2 seconds to generate a video stream. Correspondingly, if the moving speed of the micro-endoscope lens increases, the length of the time window becomes shorter, and vice versa.

[0046] S2: Perform grayscale processing and Gaussian filtering on each picture in the video stream q a in the video stream.

[0047] S3: Perform Fourier transform on the pictures; ; where f(x, y) is the given picture, x and y are the coordinates of the pixel points for Fourier transform, F(u, v) is the picture after Fourier transform, u and v are frequency variables, and j is the imaginary unit; S4: Calculate the initial sharpness B of each picture based on the amplitude |F(u, v)|; ; where, is the real part of the Fourier transform result, is the imaginary part of the Fourier transform result, and w is a conversion coefficient; the preset sharpness threshold B 0 , and filter out the pictures with sharpness lower than the sharpness threshold B 0 to obtain the remaining image set; S5: Perform edge detection on the pictures in the remaining image set, and screen out the picture with the maximum sharpness within the preset splicing area as the original picture.

[0048] The screening in S4 is for the overall sharpness of the pictures, and it also uses whether there are high-frequency components in the pictures to judge whether the overall sharpness of the pictures meets the expected requirements. However, this detection method cannot distinguish the blurred areas caused by lens jitter and the blurred areas caused by liquid reflection on the edge part.

[0049] The blur caused by lens jitter is that when the video is moving, the lens is not focused, resulting in afterimages in the picture. The blurred area caused by liquid reflection is that after some textures on the inner wall of the bronchus are blocked by the liquid (sputum, water) in the bronchus, the entire bronchial surface appears too smooth to match accurate feature points. Therefore, S5 is also needed for further screening.

[0050] S5 includes the following steps: S51: Obtain the preset splicing regions of each picture; The boundary of the preset region in S51 is the straight line L 1 and the straight line L 2 , L 1 and L 2 are parallel to each other, and L 1 is the outer boundary of the preset region; The straight line L 1 and L 2 The distance between them is L 3 , L 3 = d 1 × L 0 ; where d 1 is the first preset coefficient, which is positively correlated with the distance between the micro-endoscope lens and the inner wall of the bronchus, and L 0 is the first preset standard value; The distance between L 1 and the picture center is L 4 , L 4 = d 2 × L 5 , where d 2 is the second preset coefficient, which is negatively correlated with the distance between the micro-endoscope lens and the inner wall of the bronchus, and L 5 is the second preset standard value; S52: Calculate the amplitude and direction of each pixel point within the preset splicing region; G x (x, y) = I fi (x, y) * Sob x ; G y (x, y) = I fi (x, y) * Sob y ; ; ; Among them, x and y are the abscissa and ordinate of the pixel, and I fi (x, y) is the pixel value of the image after Gaussian filtering at the coordinate (x, y). G x (x, y) and G y (x, y) respectively represent the gradient values of the image in the x direction and y direction at the coordinate (x, y). M(x, y) represents the gradient intensity of (x, y) in the gradient amplitude image, and θ(x, y) represents the gradient direction of (x, y) in the gradient direction image; Sob represents the Sobel operator, which is a filter kernel for calculating the image gradient. Sob x represents the operator in the x direction, and Sob y represents the operator in the y direction; S53: Traverse the gradient magnitude image. For each pixel, check the two adjacent pixels in its gradient direction; If the gradient magnitude of the current pixel is not the local maximum in its direction, set it to 0; Set the high threshold C 1 and the low threshold C 0 , C 1 = 2C 0 ; Mark the pixels with gradient magnitude greater than C 1 as strong edges, the pixels less than C 0 as non-edges, and the pixels in between as weak edges; Starting from the strong edges, connect along the weak edges to form a complete edge contour; If a weak edge pixel is connected to a strong edge pixel, or indirectly connected to a strong edge pixel through other weak edge pixels, mark it as a strong edge as well to obtain all edge components; S54: Calculate the sharpness coefficient D of each edge; ; where M is the number of edge pixels, m is the index of the edge pixel, H ed1 (m) and H ed2 (m) represent the pixel values on both sides of the m-th edge pixel respectively. Take the picture with the highest sharpness coefficient D as the original picture.

[0051] In S5, it is actually to detect the edges of the instances in the image. If the edges are clear, it means that there will be sufficient specific feature points in the image, and these feature points will have higher matching accuracy in subsequent registration.

[0052] After the image extraction unit extracts the pictures from the video information obtained by each micro-endoscope lens, since the distances between each micro-endoscope lens and the bronchial inner wall are inconsistent, the reduction ratios of the instances in the pictures obtained by each micro-endoscope lens are inconsistent. In some pictures, 1 cm in length corresponds to 1 mm in the bronchial inner wall, and in some pictures, 1 cm in length corresponds to 2 mm in the bronchial inner wall. Therefore, it is not conducive to the registration of the pictures and the positioning of the bronchial pathological position in the panoramic picture.

[0053] For this reason, the image regularization unit will regularize the collected original pictures. Specifically: The image regularization unit obtains the original pictures of each micro-endoscope lens and the distances between each micro-endoscope lens and the bronchus, and regularizes the original pictures into a standard pixel grid with a fixed pixel ratio, so that the distances in the real world corresponding to the pixel grids of each original picture are the same.

[0054] The way for the image regularization unit to regularize the original picture into a standard pixel grid with a fixed pixel ratio includes the following steps: S01: Preset a standard pixel grid, and the length of each pixel grid in the standard pixel grid corresponding to the real world is l 0 ; S02: Calculate the length of the pixel grid in the real world in the original picture l a , l a which represents the length of the pixel grid in the original picture of the a-th micro-endoscope lens; ; Among them, d 0 represents the distance between the micro-endoscope lens and the inner wall of the bronchus, f 0 represents the focal length of the micro-endoscope lens, represents the length of the pixel grid in the original picture in the picture; S03: Based on the length relationship between l a and l 0 , regularize the original picture into the standard pixel grid.

[0055] For example, l a is twice that of l 0 , which means that 1 cm in the original picture is equivalent to 2 cm in the standard pixel grid. In this case, the original picture needs to be reduced by half, so that the original picture can be regularized into the standard pixel grid.

[0056] The above is the specific method for obtaining the pictures to be registered from each micro-endoscope lens. After obtaining these pictures, image stitching is required.

[0057] The image stitching unit stitches the original pictures of adjacent micro-endoscope lenses to obtain an annular picture.

[0058] In this solution, the stitching is roughly divided into two steps. One is to horizontally stitch the original pictures obtained by each micro-endoscope lens at the same moment in the bronchus to obtain an annular picture, and the other step is to vertically stitch the annular pictures obtained at different times. Although the two stitching methods use different modules, they both adopt the same image stitching model.

[0059] The image registration model includes: An image feature extraction network for extracting feature points in two pictures to be stitched; A feature matching network is used to match feature points and then stitch images based on the matched feature points.

[0060] Specifically, for two images to be stitched, it is essentially to find the same feature points in the images and then match the same feature points together. Thus, it is necessary to first use an image feature extraction network to extract the feature points. After the feature points are extracted, the feature points are then matched. Feature points can be directly extracted using the SIFT algorithm or extracted using a neural network. The specific method will not be elaborated here. However, when the feature points of two images are extracted, the similarity between feature points is actually very high. How to accurately match these feature points together? The present application adopts the following solution: Specifically, the image feature extraction network matches the feature points of two images in the following way: Step 1: For the images Pa and Pb to be stitched, extract the feature points therein respectively, and calculate the empirical distributions uA and uB of the feature points in the images Pa and Pb respectively; uA(x, y) = T A (x, y), uB(x, y) = T B (x, y), where, T A (x, y) and T B (x, y) respectively represent the feature values of the images Pa and Pb at the position (x, y), and uA(x, y) and uB(x, y) respectively represent the empirical distribution probability values of the images Pa and Pb at the position (x, y); When extracting the feature points of an image, it is impossible for every pixel point to be a feature point. The feature value here is essentially the probability that the output pixel point is a feature point. For example, when using a neural network model to extract the feature points of an image, before finally outputting the position of the feature points, the neural network will actually calculate the probability that all pixel points in the image belong to the feature points, and then take the pixel points exceeding the probability threshold as the feature points; therefore, the feature value in this solution is the probability value generated by the last layer of the cross-attention network.

[0061] Step 2: Construct a probability matrix C; The probability matrix C is a two-dimensional array, and each element C ij in the probability matrix C represents the probability that the i-th feature point in the image Pa matches the j-th feature point in the image Pb. The size of the probability matrix is v×z, where v is the number of feature points in the image Pa and z is the number of feature points in the image Pb. During the optimization process, the element values of the probability matrix C will change until they converge to the global optimal matching probability matrix C∗; Step 3: Calculate the global optimal matching probability; ; where i represents the index of the feature point in picture Pa, j represents the index of the feature point in picture Pb, M ij represents the matching probability between the i-th feature point and the j-th feature point, and argmax represents the global optimal matching optimization.

[0062] Configure the constraint conditions; ; ; represents the cumulative value of the j-th row in the probability matrix C, represents the cumulative value of the i-th column in the probability matrix C, uA(i) represents the empirical distribution probability value of the i-th feature point in picture Pa, and uB(j) represents the empirical distribution probability value of the j-th feature point in picture Pb.

[0063] Step 4: Use the Sinkhorn-Knopp algorithm to solve the above optimization problem to output the global optimal matching probability matrix C*; Step 5: Use the negative log-likelihood function as the loss function to perform model optimization training and output the optimized allocation matrix.

[0064] In Step 5, the allocation matrix provides the matching relationship of the feature points between picture Pa and picture Pb, and the matching of the feature points can be completed based on the optimized allocation matrix.

[0065] In the solution provided by this application, when matching two pictures, it is possible to avoid local optimality, thereby improving the matching effect.

[0066] The above description is only some preferred embodiments of this application and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this application is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features with similar functions disclosed in the embodiments of this application.

Claims

1. A dual-lumen catheter positioning system based on image registration, comprising: An airway tube is constructed with a gas passage; An airbag array, wherein a plurality of flexible airbags are spaced apart along the axial direction of the airway; An air supply module, used to supply air to the airbags to control the internal air pressure of each airbag; A multi-eye vision acquisition unit, including a plurality of micro-endoscope lenses, which are arranged in a circular array at the end of the airway tube; A control device, used for controlling the airway to move to the bronchial entrance; Features: The dual-lumen catheter positioning system based on image registration also includes: An image stitching device, used to obtain the video information collected by each micro-endoscope lens to stitch out a panoramic image of the bronchial inner wall; An image recognition device extracts the pathological locations of the bronchial inner wall from the panoramic image and calculates the tracheal location of each pathological location; The control device controls the expansion of the corresponding airbag based on the tracheal position of each pathological position, so that the inflated airbag is staggered to each pathological position.

2. The dual-lumen catheter positioning system based on image registration according to claim 1, characterized in that: The air supply module includes: There are multiple air supply pipes, each of which corresponds to an air bag; A gas source assembly, used to generate gas or inhale gas; The matrix reversing valve is connected to the air supply pipe and the air source respectively, and the air source is connected to the corresponding airbag through the reversing valve.

3. The dual-lumen catheter positioning system based on image registration according to claim 1, characterized in that: There are at least four micro-endoscope lenses.

4. The dual-lumen catheter positioning system based on image registration according to claim 1, characterized in that: The image stitching device comprises: An image information collection module is used to obtain video information of each micro-endoscope lens; A horizontal splicing module obtains video information from each micro-endoscope lens, extracts original images at the same time from the video information, and splices the original images into a ring image; The vertical stitching module obtains all the annular images and stitches the annular images together to obtain a panoramic image.

5. The dual-lumen catheter positioning system based on image registration according to claim 4, characterized in that: The horizontal splicing module includes: A plurality of distance sensors are provided, each of which is provided on a lens of each micro-endoscope, and is used to measure the distance between the lens of the micro-endoscope and the inner wall of the bronchus; An image extraction unit extracts original images from the video information of each micro-endoscope lens; An image regularization unit, which obtains the original image of each micro-endoscope lens and the distance between each micro-endoscope lens and the bronchus, and regularizes the original image into a standard pixel grid with a fixed pixel ratio, so that the real-world distance corresponding to the pixel grid of each original image is the same; The image stitching unit stitches the original images of adjacent micro-endoscope lenses to obtain a ring image.

6. The dual-lumen catheter positioning system based on image registration according to claim 5, characterized in that: The picture extraction unit extracts the original picture using the following steps: S1: From each video information Q a Intercept video stream q a , where a represents the index of the micro endoscope lens; S2: video stream q a Each image in is gray-scaled and Gaussian filtered; S3: Perform Fourier transform on the image; ； Where f(x, y) is the given image, x and y are the coordinates of the pixel points that undergo Fourier transformation, F(u, v) is the image after Fourier transformation, u and v are frequency variables, j is the imaginary unit, and e is a natural constant; Calculate the initial definition B of each image based on the amplitude |F(u,v)|; ； in, is the real part of the Fourier transform result, is the imaginary part of the Fourier transform result, and w is the conversion factor; S4: a preset clarity threshold B0 is used to filter out images with clarity below the threshold B0 to obtain a set of remaining images; S5: Perform edge detection on the images in the remaining image set, and select the image with the highest definition in the preset stitching area as the original image.

7. The dual-lumen catheter positioning system based on image registration according to claim 6, characterized in that: S5 includes the following steps: S51: Obtaining a preset stitching area of ​​each image; S52: Calculate the amplitude and direction of each pixel point in the preset stitching area; G x (x,y)=I fi (x,y)*Sob x ; G y (x,y)=I fi (x,y)*Sob y ; ； ； Among them, x and y are the horizontal and vertical coordinates of the pixel, I fi (x, y) is the pixel value of the image at coordinate (x, y) after Gaussian filtering. x (x, y) and G y (x, y) represents the gradient value of the image along the x direction and y direction at the coordinate (x, y), M(x, y) represents the gradient intensity of (x, y) in the gradient magnitude image, θ(x, y) represents the gradient direction of (x, y) in the gradient direction image; Sob represents the Sobel operator, which is the filter kernel used to calculate the image gradient. x represents the operator in the x direction, Sob y represents the operator in the y direction; S53: traverse the gradient magnitude image, and for each pixel, check the two adjacent pixels in the gradient direction; If the gradient magnitude of the current pixel is not the local maximum in its direction, it is set to 0; Set the high threshold C1 and the low threshold C0, C1=2C0; Pixels with gradient magnitude greater than C1 are marked as strong edges, pixels with gradient magnitude less than C0 are marked as non-edges, and pixels between the two are marked as weak edges; Start from the strong edge and connect along the weak edge to form a complete edge outline; If a weak edge pixel is connected to a strong edge pixel, or is indirectly connected to a strong edge pixel through other weak edge pixels, it is also marked as a strong edge to obtain all edge components; S54: Calculate the clarity coefficient D of each edge; ； Where M is the number of edge pixels, m is the index of edge pixels, and H ed1 (m) and H ed2 (m) represents the pixel values ​​on both sides of the m-th edge pixel, and the picture whose clarity coefficient D exceeds the preset value is taken as the original picture.

8. The dual-lumen catheter positioning system based on image registration according to claim 7, characterized in that: In S51, the boundaries of the preset area are straight lines L1 and L2, L1 and L2 are parallel to each other, and L1 is the outer boundary of the preset area; The distance between the straight lines L1 and L2 is L3, L3=d1×L0; wherein d1 is a first preset coefficient, which is positively correlated with the distance between the micro-endoscope lens and the inner wall of the bronchus, and L0 is a first preset standard value; The distance between L1 and the center of the image is L4, L4=d2×L5, wherein d2 is the second preset coefficient, which is negatively correlated with the distance between the micro-endoscope lens and the inner wall of the bronchus, and L5 is the second preset standard value.

9. The dual-lumen catheter positioning system based on image registration according to claim 6, characterized in that: S1 from each video information Q a Intercept video stream q a When the distance moved by the current micro-endoscope lens is less than the preset value, the images collected during the time when the micro-endoscope lens moves less than the preset value are used as the video stream.

10. The dual-lumen catheter positioning system based on image registration according to claim 8, characterized in that: The image regularization unit regularizes the original image into a standard pixel grid with a fixed pixel ratio, including the following steps: S01: Preset a standard pixel grid. The real-world length of each pixel in the standard pixel grid is l 0 ; S02: Calculate the real-world length of the pixel grid in the original image l a , l a represents the length of the pixel grid in the original image of the a-th micro-endoscope lens; ； in, d 0 Indicates the distance between the micro endoscope lens and the inner wall of the bronchus. f 0 Indicates the focal length of the micro endoscope lens, Indicates the length of the pixel grid in the original image; S03: Based on l a and l 0 The length relationship of is used to normalize the original image into a standard pixel grid.

Images