Intelligent catheter auxiliary positioning method and system based on visual identification

Abstract

The invention belongs to the field of visual auxiliary positioning, and discloses an intelligent catheter auxiliary positioning method and system based on visual identification, and the method comprises the steps: 1, obtaining an image It fusing a visual state and a physiological state, a structure image # imgabs0 # heart rate Ht and a respiratory rate Rt, generating a state tensor Xt based on It, # imgabs1 # Ht and Rt, and t representing a sampling moment; 2, acquiring a single-channel probability graph Pt based on the Xt, and acquiring a segmentation graph St and a structure graph Gt of the cavity region based on Pt; 3, acquiring a state vector of each node in the Gt based on the St, and inputting the state vectors of all the nodes into a preset graph neural network for calculation to obtain an action sequence representing a catheter path; 4, acquiring a waypoint sequence based on the action sequence, and correcting the waypoint sequence to obtain a corrected waypoint sequence; and 5, drawing a conduit track based on the corrected path point sequence. Through dynamic adaptive path adjustment, the operation precision is improved, and the operation risk is reduced.

Description

Technical Field

[0001] The present invention relates to the field of vision-assisted positioning, and in particular to an intelligent catheter-assisted positioning method and system based on vision recognition. Background Art

[0002] In the medical industry, especially during surgeries involving complex organs and cavities, catheter insertion techniques are crucial for ensuring patient safety and treatment success. Traditional catheter insertion methods often rely on the physician's experience and intuitive judgment. When operating on cavities such as the trachea and esophagus, the physician uses an endoscope or imaging device to observe the morphology of the cavity and the position of the catheter. Although endoscopic technology has made significant advances, providing high-definition, real-time video images, significant challenges remain during catheter insertion. First, the physician needs to observe and determine the relative position of the catheter and the cavity through the endoscope, which increases the complexity and uncertainty of the procedure for inexperienced physicians or high-risk surgeries. Especially in complex structures such as the trachea and esophagus, due to their diverse morphology and individual patient differences, traditional endoscopic guidance methods often cannot provide accurate catheter positioning. Second, traditional methods ignore the dynamic changes in the patient's physiological state (such as respiration and heart rate), resulting in a failure to timely reflect real-time changes in cavity morphology and path, which in turn affects the precise insertion of the catheter. More importantly, existing technologies lack the ability to dynamically adjust to issues such as lumen deformation, patient physiological fluctuations, and catheter path deviation. This can lead to deviations and misoperations during catheter insertion in complex surgical environments, increasing patient risk. Furthermore, existing systems often rely on manual calibration and simple path calculation methods, making them difficult to adapt to individual patient anatomy, making it difficult to guarantee accuracy and safety in specific patients.

[0003] Although some intelligent auxiliary positioning systems have been applied to the catheter insertion process with the development of technologies such as computer vision and artificial intelligence, these systems still face the following shortcomings: First, most intelligent systems are based only on static image analysis and ignore the impact of physiological dynamic changes on the catheter insertion path. As a result, when the patient's body cavity deforms, the system cannot make timely and effective path adjustments. Second, most existing artificial intelligence models rely on single visual information. In complex medical scenarios, visual recognition alone often cannot provide sufficient support. In particular, when faced with individual differences and dynamic changes in patients, existing technologies often cannot provide efficient and accurate positioning and navigation. Furthermore, although auxiliary systems based on deep learning technologies such as convolutional neural networks (CNNs) have made some progress in some static tasks, due to the lack of real-time feedback and adaptive adjustment of the operation process, they cannot cope with the complex changes and high real-time requirements during catheter insertion. Summary of the Invention

[0004] The purpose of the present invention is to disclose an intelligent catheter assisted positioning method and system based on visual recognition, so as to solve the technical problems pointed out in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides an intelligent catheter assisted positioning method based on visual recognition, comprising:

[0007] Step 1: Obtain an image I that combines visual and physiological states t , structural image Heart rate t and respiratory rate R t , based on I t 、 H t and R t Generate state tensor X t , t represents the sampling time;

[0008] Step 2, based on X t Get single channel probability map P t , based on P t Get the segmentation map S of the cavity area t and structure diagram G t ;

[0009] Step 3, based on S t Get G t The state vector of each node in the pipeline is input into a preset graph neural network for calculation to obtain an action sequence representing the catheter path;

[0010] Step 4: obtaining a path point sequence based on the action sequence, and correcting the path point sequence to obtain a corrected path point sequence;

[0011] Step 5, drawing the catheter trajectory based on the corrected path point sequence;

[0012] Among them, the structural image The acquisition process includes:

[0013] Get the endoscopic image V collected during surgery t and CT image A acquired before surgery;

[0014] V t Perform registration processing with A to obtain the structural image

[0015] Furthermore, I t The acquisition process includes:

[0016] Endoscopic image V tIntroducing sine and cosine waveforms to simulate image modulation, forming an image that combines visual and physiological states t .

[0017] Furthermore, based on X t Get single channel probability map P t ,include:

[0018] X t Image I t and Splicing through the channel dimension to obtain the enhanced image tensor Z t ;

[0019] Z t Input into the preset structure recognition network to obtain a single channel probability map P t .

[0020] Furthermore, based on P t Get the segmentation map S of the cavity area t and structure diagram G t ,include:

[0021] P t Perform binarization processing to obtain the segmentation map S t ;

[0022] Extract S t The edge of the S is extracted using the morphological skeleton method and the centerline fitting technique. t Obtain the center trajectory of the cavity;

[0023] Discretize the central trajectory of the cavity into a set of nodes N t ;

[0024] Constructing structure diagram G t , G t =(N t ,E t ), E t It is a set of edges connecting adjacent nodes, and the edges contain two attributes: patency score and topological deviation.

[0025] Furthermore, the process of obtaining the node's state vector includes:

[0026] For each node n i ∈N t , construct its state vector x i :

[0027]

[0028] in:

[0029] Coord(n i) represents node n i The spatial coordinates of

[0030] φ ij and θ ij Node n i and node n j The patency score and topological deviation of the edges between them;

[0031] Indicates the catheter head and node n i The Euclidean distance between

[0032] Indicates the changing trend of respiratory rate;

[0033] Indicates the trend of heart rate changes;

[0034] Represents node n i The position of the segmentation map S t The image gradient of the corresponding pixel point;

[0035] σ(I t (n i )) represents the grayscale variance of the local image of the region, and the local image of the region is t In the figure, node n i The image is composed of the pixel points in the square window with a side length of ki and the pixel point corresponding to the position of centered, where ki is the preset side length.

[0036] Furthermore, the state vectors of all nodes are input into a preset graph neural network for calculation to obtain an action sequence representing the catheter path, including:

[0037] For node q, the action of the catheter corresponding to q is represented as a q , Indicates the forward direction of the catheter, expressed as a unit direction vector; δ q represents the step size, which is a scalar with the same unit as the node position; θ q Indicates the attitude angle;

[0038] The action sequence of the catheter path is expressed as NQ represents the total number of nodes.

[0039] Furthermore, a path point sequence is obtained based on the action sequence, including:

[0040] The action sequence Decoded into a path point sequence P 1:NQ , the qth path point P q The update formula for the position is:

[0041]

[0042] R(θ q ) is based on θ q The two-dimensional rotation matrix of Angle θ q P0 represents the initial position of the catheter head.

[0043] Furthermore, the path point sequence is corrected to obtain a corrected path point sequence, including:

[0044] Perform the following processing on each path point to obtain the corrected path point sequence

[0045] For the path point P q , calculate the path point P q Credibility

[0046] Calculate the dynamic credibility threshold τ;

[0047] For the path point P q ,like Then perform path correction.

[0048] Furthermore, the catheter trajectory is drawn based on the corrected path point sequence, including:

[0049] Calculate the color and transparency of each path point separately;

[0050] Based on color and transparency in image I t Draw the catheter trajectory on the

[0051] In a second aspect, the present invention provides an intelligent catheter auxiliary positioning system based on visual recognition, comprising a generation module, an acquisition module, a calculation module, a correction module, and a drawing module;

[0052] The generation module is used to obtain an image that combines visual and physiological states. t , structural image Heart rate t and respiratory rate R t , based on I t 、 H t and R t Generate state tensor X t , t represents the sampling time;

[0053] Get the module for X-based t Get single channel probability map P t , based on P t Get the segmentation map S of the cavity areat and structure diagram G t ;

[0054] The calculation module is used for S-based t Get G t The state vector of each node in the pipeline is input into a preset graph neural network for calculation to obtain an action sequence representing the catheter path;

[0055] The correction module is used to obtain a path point sequence based on the action sequence, and correct the path point sequence to obtain a corrected path point sequence;

[0056] The drawing module is used for drawing the catheter trajectory based on the corrected path point sequence;

[0057] Among them, the structural image The acquisition process includes:

[0058] Get the endoscopic image V collected during surgery t and CT image A acquired before surgery;

[0059] V t Perform registration processing with A to obtain the structural image

[0060] Beneficial effects:

[0061] In order to address the deficiencies in the prior art, the present invention proposes an intelligent catheter-assisted positioning method and system based on visual recognition. By innovatively combining a variety of cutting-edge technologies, it overcomes the limitations of traditional methods in dynamic changes, path accuracy and personalized adaptability. First, the present invention introduces a dynamic adaptive path planning mechanism that can respond in real time to the impact of the patient's physiological state (such as breathing, swallowing, etc.) on the structure and morphology of the cavity. In actual operation, changes in the patient's physiological state may cause the cavity to contract or expand, and traditional methods cannot handle such dynamic changes, resulting in deviations in the catheter path. One of the core innovations of the present invention is that by combining a variety of intelligent algorithms, it can capture the dynamic deformation of the cavity in real time and automatically adjust the catheter path according to the patient's physiological feedback, thereby avoiding deviation from the target area and ensuring the accuracy and safety of catheter insertion.

[0062] In addition, the present invention provides personalized path planning for catheter insertion by combining endoscopic images, patient physiological signals (such as heart rate, respiratory rate) and anatomical data. Traditional technologies mostly rely solely on endoscopic images for path selection, ignoring individual differences and physiological fluctuations in patients. The present invention, through the fusion of multimodal data, can not only more accurately model the patient's anatomical structure, but also adjust the path based on real-time feedback, providing highly personalized navigation support. In particular, the intelligent algorithm combined with the patient's physiological data can dynamically adjust the depth, angle and direction of catheter insertion according to different surgical scenarios and patient conditions, further improving the accuracy and safety of the operation.

[0063] In summary, the main innovation of this invention lies in the combination of dynamic adaptive path adjustment, personalized data fusion and intelligent algorithms, which overcomes the problems of low path accuracy, poor adaptability, lack of real-time feedback and ability to respond to physiological changes existing in traditional catheter positioning methods. It provides a new and efficient intelligent auxiliary solution for catheter insertion in complex surgeries, which can significantly reduce surgical risks, improve treatment effects, and provide doctors with more powerful decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 Schematic diagram of the intelligent catheter assisted positioning method based on visual recognition of the present invention. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0067] like Figure 1 In one embodiment shown, the present invention provides an intelligent catheter assisted positioning method based on visual recognition, comprising:

[0068] Step 1: Obtain an image I that combines visual and physiological states t , structural image Heart rate t and respiratory rate R t , based on I t 、 H t and R t Generate state tensor X t , t represents the sampling time;

[0069] Step 2, based on X t Get single channel probability map P t , based on P t Get the segmentation map S of the cavity area t and structure diagram G t ;

[0070] Step 3, based on S t Get G t The state vector of each node in the pipeline is input into a preset graph neural network for calculation to obtain an action sequence representing the catheter path;

[0071] Step 4: obtaining a path point sequence based on the action sequence, and correcting the path point sequence to obtain a corrected path point sequence;

[0072] Step 5, drawing the catheter trajectory based on the corrected path point sequence;

[0073] Among them, the structural image The acquisition process includes:

[0074] Get the endoscopic image V collected during surgery t and CT image A acquired before surgery;

[0075] V t Perform registration processing with A to obtain the structural image

[0076] Endoscopic images can be obtained by periodically sampling the endoscopic video stream, for example, by using an electronic endoscope system (such as Olympus GIF-H290) to collect data during surgery. Olympus GIF-H290 can provide a color video stream with a resolution of 1920×1080. In this invention, OpenCV is used to collect 30 frames per second, crop and scale it to 512×512 to match the subsequent model input resolution. Each frame of the image is in three-channel RGB format, denoted as

[0077] Accompanying this is the CT image A collected before the operation. Typically in DICOM format, with slice thickness ranging from 0.5 mm to 1.25 mm. D indicates the number of slices in the CT image.

[0078] The purpose of step 1 is to integrate preoperative structural information, intraoperative image information and physiological state signals to construct a unified dynamic state tensor X t and a physiologically modulated image I t , serving as the direct input for the subsequent structure recognition model. This operation is the foundation of the entire system and the first step in solving the problem of "dynamic changes in the cavity + individual differences". All variables are generated clearly in this step.

[0079] Furthermore, for V t Perform registration processing with A to obtain the structural image include:

[0080] Use SimpleITK to read and reconstruct the volume of A:

[0081] Volume reconstruction involves constructing a 3D volume image based on A. Specifically, the SimpleITK.ImageSeriesReader class is used in conjunction with the GetGDCMSeriesFileNames() interface to automatically read all slices of CT images in a DICOM folder. SimpleITK automatically sorts, reads, aligns, and outputs a 3D volume image based on the DICOM header of each slice.

[0082] After volume reconstruction, the body position calibration module is used to perform non-rigid registration so that the three-dimensional volume image is aligned with the current endoscopic image.

[0083] Since CT images are usually in the supine position and endoscopes are in the prone position, the registration process is not only geometric alignment, but also involves anatomical region matching. Here, the present invention adopts a registration strategy based on local key points:

[0084] Extracting endoscopic image V using SIFT t and the key points in CT image A;

[0085] Use RANSAC to match the transformation matrix T and obtain the registered structural image

[0086] If the number of keypoint matches is less than 50 or the matching residual is greater than 15 pixels, an alternative registration process (anatomical region matching based on ellipse fitting) will be triggered as a remedy.

[0087] After completing the registration The anatomical annotation information is retained and mapped to the field of view of the current endoscopic image, providing a structural reference for subsequent image segmentation.

[0088] In addition to visual information, the system also needs to consider physiological signals. The heart rate and blood pressure are collected by a three-lead electrocardiogram (ECG) patch worn on the patient's chest and a nasal cannula flow sensor. and respiratory rate The signal acquisition module is based on the ESP32 Bluetooth node and works with open source physiological sensor boards (such as AD8232 and SPX-06) to synchronously sample and upload to the central processor at sampling rates of 250Hz and 20Hz respectively. At the sampling point t corresponding to each endoscopic image, the system will extract the average value of the two time series signals within a window of a set length (for example, 1 second) as the H value of the current frame. t 、R t , further mapped into image modulation.

[0089] For example, if the set time length is 1 second, the time range corresponding to the window is [t-0.5, t+0.5].

[0090] Furthermore, I t The acquisition process includes:

[0091] Endoscopic image V t Introducing sine and cosine waveforms to simulate image modulation, forming an image that combines visual and physiological states t .

[0092] The present invention observes that in endoscopic images, the cavity opens and closes periodically with the respiratory cycle (similar to the width and narrowness of the QRS band in the electrocardiogram), so the sine and cosine waveforms are introduced to simulate image modulation to form an image that combines visual and physiological states. t :

[0093]

[0094] in:

[0095] I t (x,y,c) represents I t The pixel value on channel c of the pixel at position (x, y); c∈{R, G, B};

[0096] V t (x,y,c) is the value of V t The pixel value on channel c of the pixel at position (x, y) in the middle;

[0097] R t It is V t The corresponding respiratory rate (unit: times / minute) is collected through the nasal cannula flow sensor;

[0098] H t It is V tThe corresponding heart rate (unit: beats / minute) is collected through ECG;

[0099] t is V t Timestamp (unit: seconds);

[0100] α is the respiratory modulation factor (set to 0.05 in this system);

[0101] β is the heart rate modulation factor (set to 0.02).

[0102] For example, if the patient's respiratory rate is 18 breaths / minute (R t =18), heart rate is 72 beats / min (i.e. H t =72), and the current timestamp t = 12.3 seconds, then the physiological modulation factor is:

[0103] The breathing term is α·sin(2π·0.3·12.3)≈α·0.5

[0104] The heart rate term is β·cos(2π·1.2·12.3)≈β·(-0.4)

[0105] After substitution into the calculation, the current pixel value is multiplied by 1+0.05·0.5-0.02·0.4=1.019, that is, the image brightness is slightly enhanced to simulate the current state of mild expansion of the cavity.

[0106] Through this approach, the color channel values ​​in the image dynamically respond to the patient's physiological state. The model learning process is no longer based on a "fixed-value image," but rather on a "physiological-visual linkage image." This is crucial for determining cavity boundaries. For example, when intubating at the tracheobronchial junction, images during inspiration are clearer than those during expiration, improving the robustness of the system's structural recognition.

[0107] Furthermore, based on I t 、 H t and R t Generate state tensor X t ,include:

[0108] Image I t Aligned structure images And the original physiological signal H t 、R t Unified packaging into state tensors:

[0109]

[0110] This state tensor is used for subsequent cavity structure identification and graph modeling.

[0111] The core innovation of Step 1 is the introduction of a physiological rhythm-based image modulation method. By embedding respiratory and heart rate signals into the image brightness and texture representation, this method establishes a direct link between visual information and the dynamic changes in the actual cavity. Furthermore, non-rigid registration is used to map CT anatomical structures in real time, laying a personalized and structured foundation for image segmentation and path planning. This mechanism not only increases the image's sensitivity to structural changes but also improves the system's generalization capabilities across different patients and surgical scenarios, greatly enhancing the effectiveness and reproducibility of subsequent structure recognition and path reasoning.

[0112] The core task of step 2 is to convert the state tensor output by the previous step The image information and structural information in the image are deeply integrated and processed to segment the trachea, esophagus and other cavity areas with high precision, and further construct the structural map G t to express the geometry and connection relationship of these cavities. Unlike traditional image segmentation methods, this step not only processes static structures, but also needs to adapt to the deformation of the cavities caused by factors such as breathing and heart rate during the operation. Therefore, it is necessary to build a graph model with dynamic response capabilities. In addition, considering the extremely high requirements of the catheter insertion task on "path patency" and "risk of misinsertion", the present invention introduces a physiological dynamic-driven structural weighting mechanism in the graph modeling stage, so as to achieve anatomical mapping with high robustness under individual differences and dynamic environments.

[0113] Furthermore, based on X t Get single channel probability map P t ,include:

[0114] X t Image I t and Splicing through the channel dimension to obtain the enhanced image tensor Z t ;

[0115] Z t Input into the preset structure recognition network to obtain a single channel probability map P t .

[0116] Furthermore, X t Image I t and Splicing through the channel dimension to obtain the enhanced image tensor Z t ,include:

[0117] The dimensions are 512×512×4.

[0118] Furthermore, the encoder part of the preset structure recognition network uses a residual convolution module (4 layers, each layer contains 3×3 convolution + BN + ReLU, the number of channels is 64-128-256-512 respectively), and the decoder is a symmetric deconvolution structure. The output is a single-channel probability map P t , P t ∈[0,1] 512×512 , P t Contains the probability that each pixel belongs to the lumen area.

[0119] The present invention adopts a method based on structural image The soft guidance supervision mechanism of the prediction result P t Strengthen structural boundary consistency.

[0120] Furthermore, in order to enhance the structure recognition network's perception of dynamic cavity changes, the present invention designs a special physiological rhythm perception regularization term and adds it to the loss function, thereby guiding the structure recognition network to learn the "cavity deformation pattern under breathing and heartbeat" during training. Therefore, the preset objective function of the structure recognition network during training is:

[0121]

[0122] in:

[0123] It is the standard Dice loss, which is used to measure the degree of overlap between the single-channel probability map and the manually annotated cavity area image;

[0124] The calculation method is:

[0125] The two matrices consisting of the single-channel probability map and the pixel points of the manually labeled cavity area image are multiplied bit by bit, and the overall sum is multiplied by 2, divided by the sum of the bitwise squares of the two matrices, and the final result is 1 minus this sum.

[0126] is the structural image guided regularization term, used to calculate P t and Similarity at the boundary position, specifically the L2 boundary error;

[0127] The L2 boundary error is calculated by extracting corresponding points between the predicted catheter position and the boundary of the structural image. The square of the pixel coordinate difference between them is then calculated and averaged across all points. A smaller L2 error indicates a closer fit between the catheter path and the structural boundary. This value reflects whether the model's predicted path closely adheres to the contours of the actual anatomical structure.

[0128] is an innovative “physiological dynamic structural consistency term”, which has the form:

[0129]

[0130] in:

[0131] represents the predicted probability value of the z-th pixel; N represents the total number of pixels;

[0132] is the horizontal gradient of the z-th pixel, indicating edge sharpness;

[0133] α = 0.03, β = 0.02 are the respiratory and heartbeat modulation coefficients, respectively;

[0134] The meaning of this item is to encourage the change trend of the predicted mask edge to be consistent with the physiological cycle fluctuation pattern, thereby improving the temporal stability of the segmentation boundary.

[0135] In practice, this regularization term significantly reduces boundary blurring during peaks and troughs of breathing. For example, during inspiration, the cavity expands, so the model should enhance the cavity area, while at the end of exhalation, the cavity shrinks, so the model should reduce the predicted area. This mechanism enables the segmentation network to predict dynamic boundaries in a way that adapts to breathing.

[0136] Furthermore, based on P t Get the segmentation map S of the cavity area t and structure diagram G t ,include:

[0137] P t Perform binarization processing to obtain the segmentation map S t ;

[0138] During the binarization process, the threshold value 0.5 can be used for judgment, and P t The grayscale values ​​of pixels with a probability greater than 0.5 are set to 255, and the grayscale values ​​of the remaining pixels are set to 0;

[0139] Extract S t The edge of the S is extracted using the morphological skeleton method and the centerline fitting technique. t Obtain the center trajectory of the cavity;

[0140] S can be obtained by edge detection algorithm t The edge of the cavity is obtained by morphological skeleton extraction, and then the edge is processed using the morphological skeleton extraction method to obtain a simpler edge of the cavity. Finally, the edges on both sides of the cavity are calculated using the centerline fitting method to obtain the center trajectory of the cavity.

[0141] Morphological skeleton extraction has multiple applications in image processing and computer vision. First, it simplifies the shape representation of an object by extracting its backbone structure and removing redundant edge information, making the object's shape more compact. This simplification not only reduces the amount of data processed but also preserves the object's topology, namely its connectivity and shape characteristics.

[0142] Discretize the central trajectory of the cavity (for example, select a point at each fixed length on the central trajectory) into a set of nodes N t ; Each node corresponds to a physically meaningful position in the cavity structure (such as branch, bend, entrance, etc.).

[0143] Constructing structure diagram G t , G t =(N t ,E t ), E t It is a set of edges connecting adjacent nodes, and the edges contain two attributes: patency score and topological deviation.

[0144] Node n in the graph i and n j The edge between ij Comes with two key attributes:

[0145] Patency score φ ij : Measures the feasibility of the catheter passing through the segment, which is affected by the image gradient and physiological status. The calculation method is as follows:

[0146]

[0147] in:

[0148] κ is a parameter that controls the degree of gradient influence, and the value of κ can be 1;

[0149] and Node n i and n j the sharpness of the predicted image boundaries nearby; The calculation method is:

[0150] In P t Chinese i As the center, get a window with a side length of ki, and take the average value of the gradient amplitude of the pixels in the window as n i The sharpness of the image prediction boundary nearby.

[0151] The calculation method and The calculation method is the same.

[0152] η is the respiratory rate change adjustment factor, η = 0.05;

[0153] is the rate of change of the current respiratory rate, which is used to penalize path crossing during rapid fluctuations;

[0154] The innovation lies in the introduction of Consider the negative impact of “physiological instability” on path selection.

[0155] Topological deviation θ ij : Used to characterize edge e ij The geometric deviation from the standard anatomical path (which can be e ij The acute angle between the standard anatomical path and the Extract a standard path as a reference.

[0156] If the angle between the edge and the standard path is greater than 15°, it will be considered a deviation and can be set as a "penalty path" or "low-trust path" in the subsequent graph neural network.

[0157] Step 2 constructs innovations from three dimensions: first, a guided segmentation network that fuses preoperative structural images and modulation images is introduced to improve the ability to identify structural consistency in the cavity area; second, a boundary dynamic regularization term synchronized with the physiological cycle is added to the training target, so that the model has the ability to adjust the boundary under the respiratory rhythm, which significantly improves the temporal stability of the segmentation; third, a structural graph that integrates information such as gradients, respiratory frequency changes, and topological deviations is designed in the graph modeling stage, which improves the subsequent graph neural network's judgment accuracy on "crossing feasibility."

[0158] Step 3 is the core of patent decision-making, and its goal is to obtain the structure diagram G of the cavity based on the output of step 2. t =(N t ,E t ) and segmentation map S t , combined with the physiological state parameter H introduced in step 1 t (heart rate), R t (Respiratory rate) and image I t , build a graph neural network with dynamic adaptability and structural understanding capabilities to output high-precision, real-time, and safe catheter insertion control instructions.

[0159] Furthermore, the process of obtaining the node's state vector includes:

[0160] For each node n i ∈N t , construct its state vector x i :

[0161]

[0162] in:

[0163] Coord(n i ) represents node n i The spatial coordinates (i.e. n i The position of the corresponding pixel point);

[0164] φ ij and θ ij Node n i and node n j The patency score and topological deviation of the edges between them;

[0165] Indicates the catheter head and node n i The Euclidean distance between

[0166] Indicates the changing trend of respiratory rate;

[0167] Indicates the trend of heart rate changes;

[0168] Represents node n i The position of the segmentation map S t The image gradient of the corresponding pixel point reflects the clarity of the cavity boundary;

[0169] σ(I t (n i )) represents the grayscale variance of the local image of the region, and the local image of the region is t In the figure, node n i The image is composed of the pixel points in the square window with the position of the center and the side length of ki, where ki is a preset side length, for example, ki can be 11.

[0170] σ(I t (n i )) is used to reflect visual complexity or ambiguity.

[0171] Furthermore, the state vectors of all nodes are input into a preset graph neural network for calculation to obtain an action sequence representing the catheter path, including:

[0172] The state vectors of all nodes are input into the preset graph neural network for modeling, and the position attention and physiological modulation gating mechanisms are introduced to obtain the preset graph neural network output action;

[0173] For node q, the action of the catheter corresponding to q is represented as a q , δ q and θ qThey represent the advancing direction, step length and attitude angle of the catheter respectively;

[0174] The action sequence of the catheter path is expressed as NQ represents the total number of nodes.

[0175] Furthermore, the preset graph neural network of the present invention may be GAT.

[0176] To ensure the controllability and explainability of the strategy, this paper does not use a black-box reinforcement learning algorithm, but instead combines structure graphs and reward items for guided training.

[0177] The core innovation of this step is to construct a multi-dimensional reward function r t , not only considers navigation efficiency, but also focuses on introducing risk avoidance and physiological safety factors. The present invention proposes the following total reward function:

[0178]

[0179] λ1 and λ2 are two hyperparameters that control the weight of the physiological risk penalty and image blur penalty, respectively, in the total reward. Their values ​​are typically set based on training experience, for example, to 0.5 and 1, to balance propulsion efficiency and safety. Larger values ​​increase the impact of the corresponding penalty term during training.

[0180] Represents the positive target reward, which is related to the current catheter head Ph t and target node n target The distance is negatively correlated. ||Ph t -n target ||2 means calculating Ph t and n target the distance between them;

[0181] represents the structural reward, and φ ij and θ ij Positive correlation, can be set as

[0182] Represents the physiological risk item, which is used to comprehensively consider the risk of current respiratory rate and heart rate fluctuations:

[0183]

[0184] ρ1 and ρ2 are two hyperparameters that control the influence of breathing and heart rate on the physiological risk term. They are typically initialized to 1.0. To increase the penalty for respiratory fluctuations, ρ1 can be increased to 1.5 or 2.0. If heart rate fluctuations have little impact on system behavior, ρ2 can be appropriately reduced to 0.5. During training, these two parameters can be manually adjusted by observing the model's behavior under conditions of high physiological fluctuations. A common approach is to fine-tune them after each round of training based on reward trends. The typical range of values ​​is 0.1 to 2.0.

[0185] In actual operation, if the patient's breathing suddenly speeds up (for example, from 10 to 18 rpm), this item will increase significantly, and the preset graph neural network will tend to reduce the propulsion step size or posture angle to avoid rapid insertion in an unstable state.

[0186] Represents the image blur area penalty term, which is used to suppress the catheter from entering the structural blur area:

[0187]

[0188] whereσ(I t (Ph t )) is the grayscale variance of the image around the current catheter head, and is calculated by t , calculate the variance of the grayscale values ​​of the pixels in the square area with a side length of ki centered at the current position of the catheter head;

[0189] If it is too small, it means that the area is too uniform (such as being blocked by liquid), which may lead to difficulty in identification;

[0190] is the structural boundary gradient, which is calculated by t , calculating the average value of the gradient amplitude of the pixel points in the square area with a side length of ki centered at the current position of the catheter head;

[0191] The smaller the gradient, the blurrier the area.

[0192] ζ and η are two hyperparameters used to balance the influence of the two components of the image blur penalty. ζ controls the weight of the image grayscale variance term, and η controls the weight of the structural boundary gradient term. They are generally initially set to 1.0. If the model is found to be overly dependent on image texture information, ζ can be adjusted lower. If the path frequently strays into the blurred boundary region, η can be adjusted higher. Fine-tuning can be done based on the frequency of the catheter's behavior in the blurred region, observing the trajectory distribution and changes in the training loss. A typical adjustment range is 0.1 to 2.0.

[0193] The innovation of this reward function lies in introducing "physiological risk drive" and "structural visibility drive" into path strategy decision-making, so that the model not only pursues reaching the goal, but also pursues moving forward in a safe, clear, and low-risk area, thereby building an "explainable safety strategy system."

[0194] The entire training adopts a reinforcement learning strategy based on PPO. The structural graph input is encoded through the graph attention network (GAT), and the action preference distribution of each node is output. The strategy is obtained through softmax and then updated based on the feedback reward after the action is executed.

[0195] Step 3: The dynamic structure diagram G of the previous stage t and segmentation map S t It is effectively integrated into the reinforcement learning path strategy and introduces multiple risk factors unique to the patent scenario as reward modifiers. By constructing a multidimensional reward system that incorporates factors such as structural patency, physiological state stability, and image recognizability, the pre-set graph neural network has stronger dynamic adaptability and security. It is particularly suitable for the "dynamic cavity identification and catheter positioning" tasks involved in this patent. It fills the gap in traditional graph neural networks in the joint perception and judgment of "physiology-image-structure" in dynamic medical scenarios and is one of the key supports for the creativity of this patent.

[0196] Furthermore, a path point sequence is obtained based on the action sequence, including:

[0197] The action sequence Decoded into a path point sequence P 1:NQ , the qth path point P q The update formula for the position is:

[0198]

[0199] R(θ q ) is based on θ q The two-dimensional rotation matrix of Angle θ q P0 represents the initial position of the catheter head, which is usually obtained through automatic recognition of intraoperative images or manual calibration by doctors. It is a two-dimensional position point in the image coordinate system (such as pixel coordinates) and serves as the starting point for trajectory generation.

[0200] Furthermore, the path point sequence is corrected to obtain a corrected path point sequence, including:

[0201] Perform the following processing on each path point to obtain the corrected path point sequence

[0202] For the path point P q , calculate the path point Pq Credibility

[0203] Calculate the dynamic credibility threshold τ;

[0204] For the path point P q ,like Then perform path correction.

[0205] Further, The calculation formula is:

[0206]

[0207] Variable Description:

[0208] φ ij Represents the path point P q Side e ij The structural patency score is derived from the structural diagram G t , the larger the value, the smoother;

[0209] μ q Indicates the image clarity score, P q Normalized value of the grayscale variance of the surrounding 5×5 image block. A larger value indicates a clearer image.

[0210] γ q Represents the physiological fluctuation influence term, defined as Reflects the change rate of current respiratory rate and heart rate;

[0211] θ ij The topological deviation of the edge is defined as the inverse of the cosine of the angle between the catheter path direction and the anatomical centerline direction. It has been normalized to the range of 0 to 1. The larger the value, the more serious the deviation of the path from the anatomical structure.

[0212] α1=0.35, α2=0.25, α3=0.2, α4=0.2 α1 to α4 are the four set weight coefficients.

[0213] Furthermore, the dynamic credibility threshold τ is calculated, including:

[0214]

[0215] Variable Description:

[0216] For S t The average gradient amplitude of , which indicates the overall boundary clarity;

[0217] std(I t ) is image I t The grayscale standard deviation represents the image complexity;

[0218] γ mean For the path point sequence P 1:NQ The average value of physiological fluctuation influence items at each path point;

[0219] β1=0.5,β2=0.3,β3=0.2, all are adjustment coefficients.

[0220] Furthermore, path correction is divided into two levels:

[0221] Local replacement: Search for a more credible path point in the neighborhood of the current point (5×5 grid) to replace it;

[0222] Full map replanning: starting from the current point, with the target unchanged, replanning in G t Search for a path with higher patency score and lower topological deviation (as long as it is smaller than the previous one, the search ends).

[0223] Step 4 provides an uncertainty modeling mechanism tailored to real-world medical environments. By constructing a dynamic, multidimensional credibility calculation function, it effectively identifies structural risks, visual uncertainty, and physiological perturbations within the path. It then utilizes dynamic thresholds and a structure-guided replanning mechanism to correct the path. This mechanism bridges the gap between path strategy output and final navigation execution, and is the core of this patent's approach to ensuring catheter path stability and safety during actual surgical procedures.

[0224] Step 5 aims to convert the catheter pathpoint sequence and pathpoint credibility output from Step 4 into visual information and overlay it onto the intraoperative image as an augmented reality (AR) layer, enabling real-time interaction between the physician and the system, risk visualization, and path guidance. The key to this step is mapping the structural and credibility modeling results into actionable visual elements, allowing the physician to make strategic choices based on this visual information during the procedure, thus completing the collaborative closed loop of "intelligent path recommendation → physician confirmation and execution."

[0225] Furthermore, the catheter trajectory is drawn based on the corrected path point sequence, including:

[0226] Calculate the color and transparency of each path point separately;

[0227] Based on color and transparency in image I t Draw the catheter trajectory on the

[0228] Furthermore, the color and transparency of each path point are calculated separately, including:

[0229]

[0230] The variables have the following meanings:

[0231] Color(P q ): The color of the waypoint on the AR interface, using Jet color mapping From blue (low confidence) to red (high confidence);

[0232] Alpha(P q ): transparency, used to adjust the visualization intensity of different credibility segments;

[0233] γ: basic transparency coefficient, the empirical value is 0.6. The larger it is, the more eye-catching the low credibility area is.

[0234] The innovation of this mapping function lies in the use of a dynamic transparency mechanism, which automatically "highlights" the risk segments visually without the need for active screening by doctors, greatly improving the perceptual efficiency of path decision-making.

[0235] Subsequently, the present invention The continuous coordinate information in image I t The catheter trajectory is drawn on the . The trajectory is formed by sequentially connecting the path points to form a set of vector segments. Each of these Each line segment inherits the color and transparency information of its starting point, encoding the "credibility distribution" of the entire path.

[0236] To support doctors in operating the path during surgery, the present invention also provides an interactive path control module that supports the following operations:

[0237] Path highlight switch:

[0238] Doctors can adjust the current path credibility display threshold c through voice (such as "high risk hidden") or perioperative controller (such as slider) thresh , only show Path segment.

[0239] For example: At complex tracheal branches, doctors can set c thresh =0.7, the system automatically hides path segments below this value to avoid misinsertion.

[0240] Suspicious segment feedback:

[0241] During surgery, the doctor can click on a certain section of the path, and the system will record it as a "suspicious point" and automatically generate a feedback item. Used for postoperative model retraining.

[0242] For example, if a segment clearly has poor visibility but is marked as highly reliable by the system, when the doctor clicks on the segment, the system will record the abnormal judgment and use it for subsequent model tuning.

[0243] Tips for goal confirmation and advancement:

[0244] Track whether the end of the path has reached the anatomical area preset by the doctor (such as the bottom of the main airway). If so, the system prompts "Advance can be performed" and highlights the path; if not, the system points to the recommended direction with a transparent dotted arrow.

[0245] For example: When the path has not yet reached the target area, the system draws a "turn down 10 degrees" arrow on the image with a red dotted line to guide the doctor to manually fine-tune the direction of the catheter.

[0246] Region blocking and skipping suggestions (optional extension):

[0247] For certain continuous low-confidence areas, the system can not only reduce their transparency, but also overlay a blurred mask layer around the path to prompt the doctor to try not to perform violent pushing operations in the area.

[0248] For example: If a path is continuous The system automatically adds a mask and gives a voice prompt: "The area is unstable, it is recommended to adjust the angle."

[0249] Operation rhythm guidance:

[0250] Although this step no longer introduces R t ,H t But due to Physiological risk is implicitly incorporated and this value can also be used to indirectly manipulate rhythm recommendations.

[0251] For example: If continuous P q The credibility dropped sharply in a short period of time, and the system popped up a window prompting "It is recommended to wait for one breathing cycle before moving forward."

[0252] The present invention also provides an intelligent catheter auxiliary positioning system based on visual recognition, which includes a generation module, an acquisition module, a calculation module, a correction module and a drawing module;

[0253] The generation module is used to obtain an image that combines visual and physiological states. t , structural image Heart rate t and respiratory rate R t , based on I t 、 H t and R t Generate state tensor X t , t represents the sampling time;

[0254] Get the module for X-based t Get single channel probability map P t , based on P t Get the segmentation map S of the cavity area t and structure diagram G t;

[0255] The calculation module is used for S-based t Get G t The state vector of each node in the pipeline is input into a preset graph neural network for calculation to obtain an action sequence representing the catheter path;

[0256] The correction module is used to obtain a path point sequence based on the action sequence, and correct the path point sequence to obtain a corrected path point sequence;

[0257] The drawing module is used for drawing the catheter trajectory based on the corrected path point sequence;

[0258] Among them, the structural image The acquisition process includes:

[0259] Get the endoscopic image V collected during surgery t and CT image A acquired before surgery;

[0260] V t Perform registration processing with A to obtain the structural image

[0261] The present invention proposes an intelligent catheter assisted positioning method and system based on visual recognition. By innovatively combining a variety of cutting-edge technologies, it overcomes the limitations of traditional methods in dynamic changes, path accuracy and personalized adaptability. First, the present invention introduces a dynamic adaptive path planning mechanism that can respond in real time to the impact of the patient's physiological state (such as breathing, swallowing, etc.) on the structure and morphology of the cavity. In actual operation, changes in the patient's physiological state may cause the cavity to contract or expand, and traditional methods cannot handle such dynamic changes, resulting in deviations in the catheter path. One of the core innovations of the present invention is that by combining a variety of intelligent algorithms, it can capture the dynamic deformation of the cavity in real time and automatically adjust the catheter path according to the patient's physiological feedback, thereby avoiding deviation from the target area and ensuring the accuracy and safety of catheter insertion.

[0262] In addition, the present invention provides personalized path planning for catheter insertion by combining endoscopic images, patient physiological signals (such as heart rate, respiratory rate) and anatomical data. Traditional technologies mostly rely solely on endoscopic images for path selection, ignoring individual differences and physiological fluctuations in patients. The present invention, through the fusion of multimodal data, can not only more accurately model the patient's anatomical structure, but also adjust the path based on real-time feedback, providing highly personalized navigation support. In particular, the intelligent algorithm combined with the patient's physiological data can dynamically adjust the depth, angle and direction of catheter insertion according to different surgical scenarios and patient conditions, further improving the accuracy and safety of the operation.

[0263] In summary, the main innovation of this invention lies in the combination of dynamic adaptive path adjustment, personalized data fusion and intelligent algorithms, which overcomes the problems of low path accuracy, poor adaptability, lack of real-time feedback and ability to respond to physiological changes existing in traditional catheter positioning methods. It provides a new and efficient intelligent auxiliary solution for catheter insertion in complex surgeries, which can significantly reduce surgical risks, improve treatment effects, and provide doctors with more powerful decision support.

[0264] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An intelligent catheter assisted positioning method based on visual recognition, characterized in that: include: Step 1: Obtain an image I that combines visual and physiological states t , structural image Heart rate t and respiratory rate R t , based on I t 、 H t and R t Generate state tensor X t , t represents the sampling time; Step 2, based on X t Get single channel probability map P t , based on P t Get the segmentation map S of the cavity area t and structure diagram G t ; Step 3, based on S t Get G t The state vector of each node in the pipeline is input into a preset graph neural network for calculation to obtain an action sequence representing the catheter path; Step 4: obtaining a path point sequence based on the action sequence, and correcting the path point sequence to obtain a corrected path point sequence; Step 5, drawing the catheter trajectory based on the corrected path point sequence; Among them, the structural image The acquisition process includes: Get the endoscopic image V collected during surgery t and CT image A acquired before surgery; V t Perform registration processing with A to obtain the structural image 2. The intelligent catheter assisted positioning method based on visual recognition according to claim 1 is characterized in that: I t The acquisition process includes: Endoscopic image V t Introducing sine and cosine waveforms to simulate image modulation, forming an image that combines visual and physiological states t .

3. The intelligent catheter assisted positioning method based on visual recognition according to claim 2 is characterized in that: Based on X t Get single channel probability map P t , include: X t Image I t and Splicing through the channel dimension to obtain the enhanced image tensor Z t ; Z t Input into the preset structure recognition network to obtain a single channel probability map P t .

4. The intelligent catheter assisted positioning method based on visual recognition according to claim 3 is characterized in that: Based on P t Get the segmentation map S of the cavity area t and structure diagram G t ,include: P t Perform binarization processing to obtain the segmentation map S t ; Extract S t The edge of the S is extracted using the morphological skeleton method and the centerline fitting technique. t Obtain the center trajectory of the cavity; Discretize the central trajectory of the cavity into a set of nodes N t ; Constructing structure diagram G t , G t =(N t ,E t ), E t It is a set of edges connecting adjacent nodes, and the edges contain two attributes: patency score and topological deviation.

5. The intelligent catheter assisted positioning method based on visual recognition according to claim 4 is characterized in that: The process of obtaining the node's state vector includes: For each node n i ∈N t , construct its state vector x i : in: Coord(n i ) represents node n i The spatial coordinates of φ ij and θ ij Node n i and node n j The patency score and topological deviation of the edges between them; Indicates the catheter head and node n i The Euclidean distance between Indicates the changing trend of respiratory rate; Indicates the trend of heart rate changes; Represents node n i The position of the segmentation map S t The image gradient of the corresponding pixel point; σ(I t (n i )) represents the grayscale variance of the local image of the region, and the local image of the region is t In the figure, node n i The image is composed of the pixels in a square window with a side length of ki and a center at the position of , where ki is the preset side length.

6. The intelligent catheter assisted positioning method based on visual recognition according to claim 5 is characterized in that: The state vectors of all nodes are input into the preset graph neural network for calculation to obtain the action sequence representing the catheter path, including: For node q, the action of the catheter corresponding to q is represented as a q , Indicates the forward direction of the catheter, expressed as a unit direction vector; δ q represents the step size, which is a scalar with the same unit as the node position; θ q Indicates the attitude angle; The action sequence of the catheter path is expressed as NQ represents the total number of nodes.

7. The intelligent catheter assisted positioning method based on visual recognition according to claim 6 is characterized in that: Obtain a pathpoint sequence based on an action sequence, including: The action sequence Decoded into a path point sequence P 1:NQ , the qth path point P q The update formula for the position is: R(θ q ) is based on θ q The two-dimensional rotation matrix of Angle θ q P0 represents the initial position of the catheter head.

8. The intelligent catheter assisted positioning method based on visual recognition according to claim 7 is characterized in that: The waypoint sequence is corrected to obtain a corrected waypoint sequence, including: Perform the following processing on each path point to obtain the corrected path point sequence For the path point P q , calculate the path point P q Credibility Calculate the dynamic credibility threshold τ; For the path point P q ,like Then perform path correction.

9. The intelligent catheter assisted positioning method based on visual recognition according to claim 8, characterized in that: Drawing the catheter trajectory based on the corrected waypoint sequence includes: Calculate the color and transparency of each path point separately; Based on color and transparency in image I t Draw the catheter trajectory on the 10. Intelligent catheter auxiliary positioning system based on visual recognition, characterized in that: It includes generation module, acquisition module, calculation module, correction module and drawing module; The generation module is used to obtain an image that combines visual and physiological states. t , structural image Heart rate t and respiratory rate R t , based on I t 、 H t and R t Generate state tensor X t , t represents the sampling time; Get the module for X-based t Get single channel probability map P t , based on P t Get the segmentation map S of the cavity area t and structure diagram G t ; The calculation module is used for S-based t Get G t The state vector of each node in the pipeline is input into a preset graph neural network for calculation to obtain an action sequence representing the catheter path; The correction module is used to obtain a path point sequence based on the action sequence, and correct the path point sequence to obtain a corrected path point sequence; The drawing module is used for drawing the catheter trajectory based on the corrected path point sequence; Among them, the structural image The acquisition process includes: Get the endoscopic image V collected during surgery t and CT image A acquired before surgery; V t Perform registration processing with A to obtain the structural image

Images