Endoscope-catheter cooperative control and anchoring optimization system

Abstract

The invention relates to the crossing field of medical imaging and minimally invasive medicine, and discloses an endoscope-catheter cooperative control and anchoring optimization system which comprises an acquisition module used for continuously acquiring an image sequence through an endoscope and continuously acquiring a pose sequence of a catheter in the process of probing a target tissue through the endoscope and the catheter; the feature extraction module is used for performing feature extraction on the image sequence to obtain an image feature set corresponding to the image sequence, and performing feature extraction on the pose sequence to obtain a pose feature set of the catheter; the feature fusion module is used for fusing the image feature set and the pose feature set to obtain a feature fusion result; and the cooperative control module is used for determining a control signal based on the feature fusion result, and cooperatively controlling the anchoring posture of the catheter and the endoscope posture of the endoscope based on the control signal until the catheter is located in the visual field range of the endoscope. According to the scheme, the probability that the catheter is lost in endoscope view and accidentally hurts target tissue can be reduced.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of medical imaging and minimally invasive medicine, specifically involving, but not limited to, an endoscopy-catheter collaborative control and anchoring optimization system. Background Technology

[0002] In the interdisciplinary field of medical imaging and minimally invasive medicine, endoscopy has become a key tool for the early diagnosis and treatment of diseases in various organs, including the digestive tract. During actual endoscopic procedures, physicians must simultaneously control the endoscopic view and catheter position. Consequently, when the catheter is inserted into a bend in the digestive tract, it can easily become difficult for the endoscopic field of view to track the target area where the catheter is located. This can lead to problems such as loss of the catheter within the endoscopic field of view or accidental catheter injury, resulting in low examination efficiency and increased operational risks. Summary of the Invention

[0003] Based on the above technical problems, this application provides an endoscope-catheter collaborative control and anchoring optimization system, which can improve the matching degree between the endoscopic field of view and the area where the catheter is located, reduce the probability of the endoscopic field of view losing the catheter and the catheter accidentally damaging the target tissue, thereby improving the examination efficiency and reducing the operational risks generated when probing the target tissue through the catheter-endoscope.

[0004] The technical solution provided in this application is as follows: This application provides an endoscope-catheter coordinated control and anchoring optimization system, including: The acquisition module is used to continuously acquire image sequences through the endoscope and the pose sequence of the catheter during the exploration of target tissue through the endoscope and catheter. The feature extraction module is used to extract features from the image sequence to obtain an image feature set corresponding to the image sequence, and to extract features from the pose sequence to obtain a pose feature set of the catheter. The feature fusion module is used to fuse the image feature set and the pose feature set to obtain the feature fusion result; The collaborative control module is used to determine a control signal based on the feature fusion result, and to collaboratively control the anchoring posture of the catheter and the endoscopic posture of the endoscope based on the control signal until the catheter is within the field of view of the endoscope.

[0005] The endoscope-catheter coordinated control and anchoring optimization system provided in this application has at least the following beneficial effects: In the endoscope-catheter coordinated control and anchoring optimization system provided in this application embodiment, the acquisition module continuously acquires image sequences via the endoscope and the catheter pose sequence during the exploration of target tissue using the endoscope and catheter. This enables continuous acquisition of both image and pose sequences during the exploration. Furthermore, the feature extraction module extracts features from the image sequences to obtain image feature sets and from the pose sequences to obtain catheter pose feature sets, achieving tracking-based feature extraction of both sequences. Based on this, the feature fusion module fuses the image feature sets and pose feature sets to obtain a feature fusion result, achieving the fusion of the endoscope's image feature sets and the catheter's pose feature sets, thereby improving the accuracy of feature extraction. The feature fusion results enhance the comprehensiveness and completeness of the features. On the other hand, the collaborative control module determines the control signal based on the feature fusion results, so that the control signal is associated not only with the image feature set of the endoscope, but also with the pose feature set of the catheter, thereby improving the comprehensiveness and accuracy of the control signal. Furthermore, based on the control signal, the catheter's anchoring pose and the endoscope's pose are collaboratively controlled until the catheter is within the endoscope's field of view. This enables automated and intelligent collaborative control of the catheter and endoscope, and also improves the coordination and matching degree between the catheter's anchoring pose and the endoscope's pose. This improves the matching degree between the endoscope's field of view and the area where the catheter is located, reduces the probability of the endoscope losing the catheter or accidentally damaging the target tissue, and thus improves the examination efficiency and reduces the operational risks generated when exploring target tissues through catheter-endoscope. Attached Figure Description

[0006] Figure 1 This is a schematic diagram of the endoscope-catheter coordinated control and anchoring optimization system provided in an embodiment of this application; Figure 2 This is a schematic diagram of the training structure provided in an embodiment of this application; Figure 3 This is another structural schematic diagram of the endoscope-catheter coordinated control and anchoring optimization system provided in the embodiments of this application. Detailed Implementation

[0007] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0008] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0009] In the interdisciplinary field of medical imaging and minimally invasive medicine, endoscopy has become a key tool for the early diagnosis and treatment of diseases in various organs, including the digestive tract. During actual endoscopic procedures, physicians must simultaneously control the endoscopic view and catheter position. Consequently, when the catheter is inserted into a bend in the digestive tract, it can easily become difficult for the endoscopic view to track the target area, leading to problems such as loss of endoscopic view or catheter injury. This results in low examination efficiency and increased operational risks.

[0010] Based on the above technical problems, this application provides an endoscope-catheter coordinated control and anchoring optimization system.

[0011] Figure 1 This is a schematic diagram of the endoscope-catheter coordinated control and anchoring optimization system provided in the embodiments of this application, as shown below. Figure 1 As shown, the endoscope-catheter coordinated control and anchoring optimization system 100 may include: The acquisition module 101 is used to continuously acquire image sequences through the endoscope and continuously acquire the pose sequence of the catheter during the exploration of target tissue through the endoscope and catheter. The feature extraction module 102 is used to extract features from the image sequence to obtain the image feature set corresponding to the image sequence, and to extract features from the pose sequence to obtain the pose feature set of the catheter. Feature fusion module 103 is used to fuse image feature set and pose feature set to obtain feature fusion result; The collaborative control module 104 is used to determine the control signal based on the feature fusion result, and to collaboratively control the anchoring posture of the catheter and the endoscope posture based on the control signal until the catheter is within the field of view of the endoscope.

[0012] In some embodiments, the target tissue may include at least one tissue that can be observed endoscopically and sampled via a catheter, such as the intestines, stomach, etc.

[0013] For example, the acquisition module may include an endoscope, which may include an endoscope body, an illumination and imaging assembly, and an operating channel; wherein the operating channel may include a hollow channel for a catheter to pass through.

[0014] For example, the images in the image sequence can characterize the displacement state of the endoscope in the target tissue in the time dimension, and may also include the morphological changes of at least a portion of the target tissue over time.

[0015] For example, the acquisition module may also include at least one sensor associated with the catheter; for example, the pose sequence can characterize the change in position and / or orientation of the catheter within the operating channel in a time dimension, and can also characterize the change in position and / or orientation of the catheter within the target tissue after penetrating the operating channel in a time dimension.

[0016] In some embodiments, the image feature set may include high-order image semantics carried by the image sequence; for example, the image semantics may include the current organizational morphology of the target tissue; wherein, the organizational morphology may include the surface state, color texture, and structural integrity of the target tissue, etc.; accordingly, the image feature set may be obtained in the following ways: The first model, which has the function of image feature extraction, performs feature extraction on the images in the image sequence in turn to obtain image features respectively. Then, the image features are integrated to obtain an image feature set. For example, the first model may include the backbone network of a convolutional neural network (CNN) or a residual network (ResNet).

[0017] In some embodiments, the pose feature set may include higher-order pose semantics of the pose sequence; for example, the pose semantics may include at least one of the length, rotation angle, and bending angle of the conduit displacement within a time period with the current time as the cutoff time; correspondingly, the pose feature set may be obtained in the following manner: The second model is used to extract position and / or pose characteristics represented in the pose sequence to obtain pose features. Then, the pose features are integrated in chronological order to obtain a pose feature set. The second model may include a one-dimensional convolutional neural network (1D-CNN).

[0018] In some embodiments, the feature extraction module can extract features from the image sequence and the pose sequence when specified conditions are met; wherein, the specified conditions may include the number of images in the newly acquired image sequence being a first number, the number of poses in the pose sequence being a second number, and the first number of image sequences and the second number of pose sequences being synchronized in the time dimension.

[0019] In some implementation sets, the feature fusion result may include the fusion result of pose semantics and image semantics in the time dimension; correspondingly, the feature fusion result can completely, clearly and intuitively characterize the endoscope's field of view change process and the catheter's pose change process in the time dimension; correspondingly, the feature fusion result can be implemented in any of the following ways: Based on the image timestamps carried by the image sequence and the pose timestamps carried by the pose sequence, the image feature set and the pose feature set are concatenated in the time dimension to obtain the feature fusion result.

[0020] First weights and second weights are determined, and then the image feature set and pose feature set are fused based on the first weights and second weights to obtain the feature fusion result; wherein, the first weights and second weights can be predetermined or flexibly adjusted.

[0021] In some embodiments, the anchoring pose may include the time corresponding to the pose sequence, the spatial position of the conduit, and the angular orientation; accordingly, the anchoring pose can be obtained in the following ways: The first fully connected network maps the duct-related features in the feature fusion result to the coordinate system of the target tissue to obtain the anchor pose.

[0022] In some embodiments, the endoscope pose may include the orientation, position, and angular orientation of the endoscope corresponding to the time represented by the image sequence; for example, the endoscope pose can be obtained in the following ways: The endoscope-related features in the feature fusion result are mapped to the coordinate system of the target tissue through a second fully connected network to obtain the endoscope pose.

[0023] In some embodiments, the control signal can be determined in the following ways: Based on the image semantics carried in the feature fusion results, the endoscopic field of view and the position of the endoscope relative to a specified reference point are determined, and the endoscope pose is determined. Based on the pose semantics carried in the feature fusion results, the catheter pose is determined. If the endoscopic field of view includes the lesion area, a control signal for adjusting the catheter pose is determined based on the difference between the endoscopic pose and the catheter pose, and the control signal is output. If the endoscopic field of view does not include the lesion area, a control signal for systematically adjusting the catheter pose and the endoscopic pose is determined, and the control signal is output to prompt continued coordinated adjustment of the endoscopic pose and the catheter pose. During the above adjustment process, the image sequence and pose sequence are continuously acquired by the acquisition module, the feature extraction operation is continuously performed by the feature extraction module, and the feature fusion operation is continuously performed by the feature fusion module until the endoscopic field of view includes the lesion area and the catheter.

[0024] For example, the control signal may include at least one action that coordinates the orientation of the catheter and / or endoscope. For instance, the control signal may include coordinating the adjustment of the catheter's bending degree, displacement amplitude, and rotation angle while keeping the endoscope's orientation unchanged.

[0025] For example, the control signal may also include an adjustment strategy for adjusting the position of the catheter and / or endoscope. The adjustment strategy may include single or multiple adjustments, the order and step size of the single adjustments in multiple adjustments, and conditions for stopping a single adjustment. For example, the control signal may include first adjusting the position and / or posture of the endoscope to increase the proportion of the lesion area in the endoscopic field of view until the proportion is greater than or equal to the proportion threshold. Then, based on the degree of difference between the endoscope posture and the catheter position, the anchoring posture of the catheter is gradually adjusted so that the catheter can appear within the lesion area in the endoscopic field of view.

[0026] As can be seen from the above, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application embodiment, the acquisition module is used to continuously acquire image sequences through the endoscope and continuously acquire the pose sequence of the catheter during the exploration of target tissue through the endoscope and catheter. Thus, continuous acquisition of image sequences and pose sequences can be achieved during the exploration of target tissue through the endoscope and catheter. Furthermore, the feature extraction module is used to extract features from the image sequences to obtain the image feature set corresponding to the image sequences, and to extract features from the pose sequences to obtain the pose feature set of the catheter, realizing tracking-based feature extraction of the image sequences and pose sequences. Based on this, the feature fusion module fuses the image feature set and the pose feature set to obtain the feature fusion result, realizing the fusion of the image feature set of the endoscope and the pose feature set of the catheter, thereby enabling… On the one hand, the feature fusion results are improved in terms of comprehensiveness and completeness. On the other hand, the collaborative control module determines the control signal based on the feature fusion results, so that the control signal is associated not only with the image feature set of the endoscope, but also with the pose feature set of the catheter, thereby improving the comprehensiveness and accuracy of the control signal. Furthermore, based on the control signal, the catheter's anchoring pose and the endoscope's pose are collaboratively controlled until the catheter is within the endoscope's field of view. This enables automated and intelligent collaborative control of the catheter and endoscope, and also improves the coordination and matching degree between the catheter's anchoring pose and the endoscope's pose. This improves the matching degree between the endoscope's field of view and the area where the catheter is located, reduces the probability of the endoscope losing the catheter or accidentally damaging the target tissue, and thus improves the examination efficiency and reduces the operational risks generated when exploring target tissues through catheter-endoscope.

[0027] Based on the foregoing embodiments, the endoscope-catheter coordinated control and anchoring optimization system provided in this application includes: The feature extraction module is used to determine the image weight sequence based on the image sequence and the pose weight sequence based on the pose sequence. The feature fusion module is used to fuse image feature sets and pose feature sets based on image weight sequences and pose weight sequences to obtain feature fusion results.

[0028] In some embodiments, the image weights in the image weight sequence can characterize the importance of image features in the image feature set, or, to characterize the influence of image features in the image feature set on the control signal; accordingly, the image weight sequence can be determined in the following ways: The image weight sequence is determined based on the fluctuation state of the image features in the image feature set; for example, the image weights in the image weight sequence can increase as the fluctuation state increases, so that the pose of the endoscope can be adjusted efficiently at a faster speed or with a larger step by a control signal.

[0029] In some embodiments, the image weight sequence can also be associated with the exploration progress of the exploration process; for example, the exploration progress can include an insertion phase, an exploration phase, and an exit phase; for example, the weight values ​​in the image weight sequences corresponding to the insertion phase and the exploration phase can be greater than the weight values ​​in the image weight sequence corresponding to the exit phase; for example, the exploration progress can be set by a doctor or professional technician, or it can be determined by analyzing the image sequence using the endoscope-catheter collaborative control and anchoring system provided in this application embodiment; wherein, the insertion phase can include the phase in which the endoscope and catheter are gradually inserted from outside the body into the target tissue, and the exit phase can include the phase in which the exploration ends and the endoscope and catheter are controlled to exit the target tissue.

[0030] In some embodiments, the pose weight sequence can characterize the degree of influence of at least one feature among the catheter's position, orientation, and angle on the control signal; correspondingly, the pose weight sequence can be determined in the following manner: Based on the change state of at least one feature in the pose feature set used to characterize the position, orientation, and angle of the catheter, a pose weight sequence is determined; wherein, the weight in the pose weight sequence can increase as the aforementioned change state intensifies, so that the pose of the catheter can be adjusted at a faster speed or with larger steps through the control signal.

[0031] In some embodiments, the pose weight sequence can also be associated with the exploration progress of the exploration process; for example, the values ​​of the weights in the pose weight sequences corresponding to the insertion phase and the exploration phase can be greater than the values ​​of the weights in the pose weight sequence corresponding to the exit phase.

[0032] In some embodiments, the feature fusion result can be obtained in the following way: The image feature set is fused based on the image weight sequence to obtain the image fusion result. The pose feature set is fused based on the pose weight sequence to obtain the pose fusion result. Then, the image fusion result and the pose fusion result are concatenated in the time dimension to obtain the feature fusion result.

[0033] For example, the feature fusion result can be calculated using equation (1): (1) in, For the feature fusion result, M is an integer greater than 1, representing the number of image features in the image feature set; m is an integer greater than or equal to 1 and less than or equal to M; N is an integer greater than 1, representing the number of pose features in the pose feature set; and n is an integer greater than or equal to 1 and less than or equal to N. Let m be the weight of the m-th image in the image weight sequence. Let n be the pose weight in the pose weight sequence. to Constructing an image weight sequence, to Construct a pose weight sequence, For the m-th image feature in the image feature set, It is the nth pose feature in the pose feature set.

[0034] As can be seen from the above, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application embodiment, the feature extraction module is used to determine the image weight sequence based on the image sequence and the pose weight sequence based on the pose sequence. In this way, by associating the image weight sequence and the pose weight sequence with the image sequence and the pose sequence respectively, the image weight sequence and the pose weight sequence can change with the changes of the image sequence and the pose sequence, thereby improving the dynamics and accuracy of the image weight sequence and the pose weight sequence. On this basis, the feature fusion module is used to fuse the image feature set and the pose feature set based on the image weight sequence and the pose weight sequence, which can realize targeted fusion processing of the image feature set and the pose feature set, thereby improving the consistency between the feature fusion result and the features carried by the image sequence and the pose sequence.

[0035] Based on the foregoing embodiments, in the endoscope-catheter coordinated control and anchoring optimization system provided in this application embodiment, the feature extraction module is used to extract the edge density sequence corresponding to the tissue edge of the target tissue in the image sequence, extract the texture features of the image sequence to obtain the texture sequence, and determine the image weight sequence based on the edge density sequence and the texture sequence.

[0036] In some embodiments, the edge density sequence may include the edge distribution state of the tissue edges of the target tissue carried in the image sequence; exemplarily, the edge distribution state may include at least one of gradient changes, abrupt changes in direction, or curvature features of the tissue edges in the image sequence; correspondingly, the edge density sequence may be obtained in the following manner: The image sequence is processed by a feature extraction module to obtain an image feature set. Then, the edge density sequence is extracted from the image feature set using the Sobel operator. For example, the edge density corresponding to the m-th image feature map in the image feature set can be calculated using equation (2): (2) in, Let H be the edge density of the m-th image feature map, H be the height of the m-th image feature map, and W be the width of the m-th image feature map. Let be the derivative of the feature map of the m-th image in the horizontal direction. The derivative of the m-th image feature map in the vertical direction is given by , where m is an integer greater than or equal to 1.

[0037] In some embodiments, texture features may include the texture distribution state of the surface texture of the target tissue carried by the image sequence; exemplarily, the texture sequence may include a set of texture features corresponding to the image sequence; correspondingly, the texture sequence can be obtained in the following ways: The image feature set is obtained by extracting features from the image sequence through the feature extraction module. Then, texture features are extracted from the image feature set through the gray-level co-occurrence matrix (GLCM), and the set of texture features is determined as the texture sequence. Specifically, the above process can be implemented by equation (3): (3) in, The texture features of the m-th image feature map, This represents the probability that gray values ​​i and j appear simultaneously in the feature map of the m-th image. The purpose of setting texture correction parameters is to reduce... The calculation error caused by the value of 0.

[0038] In some embodiments, the image weights in the image weight sequence can be calculated in the following way: The edge density in the edge density sequence and the texture features in the texture sequence are normalized to obtain the image weights; as shown in Equation (4): (4) Where M is the number of edge densities in the edge density sequence, and its value is greater than 1, and m is an integer greater than or equal to 1 and less than or equal to M.

[0039] As described above, in the endoscopy-catheter coordinated control and anchoring optimization system provided in this application embodiment, the feature extraction module is used to extract the edge density sequence corresponding to the tissue edge of the target tissue in the image sequence, extract the texture features of the image sequence to obtain the texture sequence, and determine the image weight sequence based on the edge density sequence and the texture sequence. In this way, the image weight sequence can be associated with both the tissue edge and texture features of the target tissue in the image sequence, thereby enabling the image weight sequence to accurately reflect the importance of features in the image feature set from the dimensions of tissue edge and texture features.

[0040] Based on the foregoing embodiments, in the endoscope-catheter coordinated control and anchoring optimization system provided in this application, the feature extraction module is used to extract features from the m-th image in the image sequence through a Feature Pyramid Network (FPN) to the m-th sub-feature set, and integrate the m-th sub-feature set to obtain the image features of the m-th image.

[0041] The m-th sub-feature set includes the local features, feature contours, texture features, context features, and global scene features of the target organization; m is an integer greater than or equal to 1.

[0042] In some embodiments, an image FPN may include a combination of a first model and an FPN; for example, the first model may include x feature extraction layers, and the x feature extraction layers may be configured in a cascaded manner.

[0043] Specifically, when the first model is ResNet, the x-th feature extraction layer in ResNet can extract features from the extraction result output by the previous feature extraction layer and the residual connection to obtain the x-th extraction result, and the first extraction result to the x-th extraction result are determined as the m-th sub-feature set; where x is an integer greater than or equal to 1 and less than or equal to X, and X is an integer greater than 1, and is used to represent the number of feature extraction layers contained in ResNet; where the first feature extraction layer in ResNet is used to extract features from the m-th image to obtain the first extraction result.

[0044] Specifically, when X is 5, the image FPN can process the second to fifth sub-features in the m-th sub-feature set. For example, the number of channels of the fifth sub-feature C5 can be compressed to a fixed dimension through convolution operation, and then bilinear interpolation upsampling can be performed on the fixed-dimensional C5 to make its resolution consistent with the resolution of the fourth sub-feature. Similarly, convolution operation is performed on the fourth sub-feature to compress its number of channels to a fixed dimension to obtain the compression result corresponding to the fourth sub-feature. Finally, the fixed-dimensional C5 and the compression result corresponding to the fourth sub-feature are added element by element to obtain the fused feature P5.

[0045] Specifically, repeating the above operations sequentially fuses the features from the fifth sub-feature to the first sub-feature, and determines the fused image features of the m-th image. Then, the fused image features of the m-th image, along with the set of P2 to P5, are determined as the image features of the m-th image.

[0046] In some embodiments, local features may include the local morphology, structure, color, texture, and lesion features of the target tissue surface as represented by the image sequence; for example, local features may include the second sub-feature in the m-th sub-feature set.

[0047] In some embodiments, contour features may include the edge shape, contour integrity, and spatial morphology of the target tissue as represented by the image sequence; for example, contour features may include the third sub-feature in the m-th sub-feature set.

[0048] In some embodiments, texture features may include the microstructure arrangement of the target tissue, grayscale or color distribution patterns, etc.; for example, texture features may include the third and fourth sub-features in the m-th sub-feature set.

[0049] In some embodiments, contextual features may include the relative position and mutual influence between the regions corresponding to local features and / or contour features and the surrounding tissue; wherein, the surrounding tissue may include adjacent regions of the regions corresponding to local features and / or contour features.

[0050] In some embodiments, global scene features may include the overall cavity environment and macroscopic functional status of the target organization.

[0051] For example, contextual features and global scene features can be represented by the fourth and fifth sub-features in the m-th sub-feature set, respectively.

[0052] As can be seen from the above, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application embodiment, the feature extraction module is used to extract features from the m-th image in the image sequence through the image FPN to obtain the m-th sub-feature set. The m-th sub-feature set includes local features, contour features, texture features, context features, and global scene features of the target tissue. In this way, comprehensive feature extraction of the m-th image can be achieved. Furthermore, integrating the m-th sub-feature set to obtain the image features of the m-th image can improve the comprehensiveness of the image features of the m-th image, thereby improving the accuracy of the image features of the m-th image.

[0053] Based on the foregoing embodiments, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application embodiment, the feature extraction module is used to statistically analyze the poses in the pose sequence to obtain the pose change rate, determine the pose entropy sequence of the pose sequence, and determine the pose weight sequence based on the pose change rate sequence and the pose entropy sequence.

[0054] In some embodiments, the pose change rate may include the number of changes in at least one of the catheter's position, orientation, and angle per unit time or unit length; exemplarily, the pose change rate may characterize the state of at least one part of the catheter, such as the catheter tip micro-vibration; accordingly, the pose change rate can be obtained in the following ways: A statistical unit is determined, and the differences between poses in the pose sequence are statistically analyzed within that unit to obtain the pose change rate; wherein, the statistical unit may include a unit of time or a unit of length; for example, the pose change rate... It can be calculated using equation (5): (5) Where N is an integer greater than 1, and is used to represent the number of poses in the pose sequence. Used to characterize the i-th pose in the pose sequence.

[0055] In some embodiments, the pose entropy in the pose entropy sequence can characterize the degree of disorder in the pose change of the catheter tip within a unit time or unit length. For example, the pose entropy sequence can be obtained by performing time statistics on the pose entropy over a specified time period. Furthermore, the specified time period can include multiple unit times or multiple unit lengths; that is, the pose entropy sequence can correspond to a sequence including multiple pose change rates. For example, the pose entropy can be obtained in the following ways: The pose change rate in the pose change rate sequence is normalized to obtain the pose probability. Then, the pose probability is statistically analyzed to obtain the pose entropy. Specifically, it can be shown in equation (6): (6) in, For pose entropy, Let be the pose probability corresponding to the i-th pose change rate in the pose change rate sequence.

[0056] In some embodiments, the pose weights in the pose weight sequence can be determined in the following ways: The pose change rate sequence and pose entropy sequence within a specified time period are normalized to obtain the pose weight sequence, as shown in equation (7): (7) in, Let y be the pose weight in the pose weight sequence. Let be the j-th pose change rate in the sequence of pose change rates. Let y be the j-th pose entropy in the pose entropy sequence, where y is an integer greater than 1, J is an integer greater than 1 and used to characterize the number of pose entropies in the pose entropy sequence, and j is an integer greater than or equal to 1 and less than or equal to J.

[0057] As can be seen from the above, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application embodiment, the feature extraction module is used to statistically analyze the pose in the pose sequence to obtain the pose change rate of the pose sequence. Thus, the pose change rate can provide a comprehensive representation of the pose change state and frequency of the catheter. Furthermore, by determining the pose entropy sequence of the pose sequence, the disordered state of the catheter pose can be determined. Based on this, the pose weight sequence is determined based on the pose change rate sequence and the pose entropy sequence, so that the pose weight sequence can reflect the dynamic change state of the pose sequence, thereby improving the accuracy of the pose weight sequence.

[0058] Based on the foregoing embodiments, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application, the feature extraction module is used to extract features from N consecutive poses in the pose sequence through pose FPN to obtain a pose feature subset, and integrate the pose feature subset to obtain pose features.

[0059] The pose feature subset includes the duct bending trend, motion inertia, path smoothness, displacement pattern, and displacement trajectory; the pose feature set includes a set of multiple pose features; N is an integer greater than 1.

[0060] In some embodiments, the pose FPN may include a combination of a second model and an FPN; wherein, when the second model is a 1D-CNN, the sub-features in the pose subset may include the result of feature extraction performed by the q-th extraction unit in the 1D-CNN on the data output by the (q-1)-th feature extraction unit; wherein q is an integer greater than 1, and the first sub-feature in the pose subset may include the result of feature extraction performed sequentially by the first extraction unit in the 1D-CNN on N poses, wherein q is an integer greater than 1.

[0061] In some embodiments, the 1D-CNN in the pose FPN can extract a set of pose sub-features, and the FPN can perform upsampling, lateral connection and element-wise addition on the pose sub-features in the set of pose sub-features to obtain pose features.

[0062] In some embodiments, the catheter bending trend may include the changing trend of the catheter tip bending over at least one time period; for example, the catheter bending trend may be obtained by the results of the first feature extraction unit and the second feature extraction unit in the 1D-CNN, such as the catheter bending trend may be reflected by the short-term details and medium-short-term features output by the pose FPN.

[0063] In some embodiments, motion inertia may include the ability of the catheter tip to maintain displacement motion over at least one time period; for example, motion inertia may be reflected by mid-layer features output by the pose FPN.

[0064] In some embodiments, the path smoothing state can characterize the curvature change and position fluctuation amplitude of the catheter displacement path; for example, the path smoothing state can be reflected by the underlying features output by the pose FPN.

[0065] In some embodiments, the displacement pattern can characterize the motion pattern of the duct; for example, the displacement pattern can be reflected by the short-term displacement details and long-term displacement trend output by the pose FPN.

[0066] In some embodiments, the displacement trajectory can characterize the spatiotemporal path of the catheter tip movement; for example, the displacement trajectory can be reflected by the long-term trajectory trend contained in the high-level features of the pose FPN output.

[0067] As can be seen from the above, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application embodiment, the feature extraction module is used to extract features from N poses in the pose sequence through a pose feature pyramid network to obtain a pose sub-feature set. The pose sub-feature set includes catheter bending trend, motion inertia, path smoothness state, displacement mode, and displacement trajectory. Thus, through the above operations, the comprehensiveness of the features in the pose sub-feature set can be improved. On this basis, the pose sub-feature set is integrated to obtain the pose features in the pose feature set, which can improve the completeness and accuracy of the pose features.

[0068] Based on the foregoing embodiments, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application, the collaborative control module is used to obtain the endoscope focal length, optimize the endoscope pose based on the endoscope focal length and image parameters of the image sequence, and obtain an endoscope pose adjustment instruction for the endoscope.

[0069] The control signals include endoscope position adjustment instructions.

[0070] In some embodiments, image parameters may include the image height and image depth of the images in the image sequence.

[0071] In some embodiments, the endoscope pose adjustment instruction may include at least one indication of direction, step, and duration for adjusting at least one of position, orientation, and angle in the current endoscope pose; correspondingly, taking the endoscope rotation angle in the endoscope pose as an example, the endoscope pose adjustment instruction for the endoscope rotation angle can be obtained in the following manner: A first scaling factor is determined based on the deviation between the current endoscope pose and the center of the image in the image sequence. Then, the current endoscope pose is adjusted based on the first scaling factor, the image width, and the endoscope focal length to obtain the target endoscope angle in the horizontal direction. The current endoscope position is adjusted based on the first scaling factor, the image height, and the endoscope focal length to obtain the target endoscope angle in the vertical direction. The target endoscope angles in the horizontal and vertical directions are determined as endoscope pose adjustment instructions for the endoscope rotation angle. Specifically, the target endoscope angle in the horizontal direction... and the vertical target endoscope angle It can be calculated using equations (8) and (9): (8) (9) in, Here, w is the first scaling factor, w is the image width, and h is the image height. For the focal length of the endoscope, ( , ) represents the horizontal and vertical coordinates in the current endoscopic pose.

[0072] In some embodiments, the current endoscope pose may include the endoscope pose at the current moment.

[0073] As described above, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application embodiment, the collaborative control module is used to obtain the endoscope focal length, optimize the endoscope pose based on the endoscope focal length and image parameters of the image sequence, and obtain an endoscope pose adjustment instruction for the endoscope. The control signal includes the endoscope pose adjustment instruction. Thus, through the above operations, the endoscope pose adjustment instruction for the endoscope is not only related to the endoscope pose, but also to the endoscope focal length and image parameters of the image sequence. This allows the endoscope pose adjustment instruction to compensate for errors caused by the endoscope focal length and image parameters, thereby improving the accuracy of the endoscope pose adjustment instruction.

[0074] Based on the foregoing embodiments, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application, the collaborative control module is used to correct the anchoring posture based on the catheter's depth and historical anchoring posture, and obtain a catheter posture adjustment instruction for the anchoring posture.

[0075] The control signals include at least catheter position adjustment instructions.

[0076] In some embodiments, the catheter insertion length may include the length of the catheter inserted into the target tissue as indicated by the pose sequence.

[0077] In some embodiments, the historical anchoring pose may include the anchoring pose of the conduit at the time preceding the current time.

[0078] In some embodiments, the catheter position adjustment indicator may include indication information indicating at least one of the following: the insertion length of the catheter, the bending state of the catheter, and the rotation angle of the catheter; correspondingly, the catheter position adjustment indicator for catheter position adjustment can be obtained in the following ways: Based on the depth length and the historical anchoring posture, the horizontal position in the anchoring posture compared to the historical anchoring posture, and the vertical position in the anchoring posture compared to the historical anchoring posture, a correction step is determined for the depth length, the horizontal position of the anchoring posture, and the vertical position. This correction step is then used as the guide tube posture adjustment indicator for guide tube position adjustment. Specifically, it can be calculated using equation (10): (10) in, For the horizontal correction step of the anchor positioning posture, To correct the vertical positioning posture of the anchor, For correction steps in the catheter length dimension, the steps used to characterize catheter insertion or withdrawal. This is the second scaling factor. The third scaling factor, ( , () represents the horizontal and vertical positions in the historical anchor positioning pose. The depth length at the current moment. The length of the historical depth is defined; it should be noted that the second and third scaling factors can be predetermined or adjusted, and this application embodiment does not limit this.

[0079] As can be seen from the above, in the endoscope-catheter collaborative control and anchoring optimization system provided in this application embodiment, the collaborative control module is used to correct the anchoring posture based on the catheter's insertion length and historical anchoring posture, obtaining a catheter posture adjustment instruction for the anchoring posture, and the control signal includes at least the catheter posture adjustment instruction. Thus, by associating the catheter posture adjustment instruction with the catheter's insertion length and historical anchoring posture through the above process, the continuity and stability of the catheter posture adjustment instruction in correcting the anchoring posture can be improved, thereby reducing the probability of target tissue damage caused by significant adjustments to the anchoring posture.

[0080] It should be noted that, in the embodiments of this application, the parameters of the feature extraction module can be adjusted in advance based on the sample data to improve the targeting of feature extraction. Figure 2 This is a schematic diagram of the training structure provided in the embodiments of this application, such as... Figure 2 As shown in the diagram, the training structure 200 may include an endoscope sample input unit, a catheter sample input unit, a preprocessing module, a feature extraction module, a feature fusion module, and an optimization module.

[0081] Specifically, the sample data may include endoscopic samples and catheter samples; endoscopic samples may include image sample sequences acquired by endoscopy during historical explorations, and catheter samples may include catheter pose samples obtained during historical explorations; it should be noted that the sample data may be used with the consent of the hospital, doctors and patients.

[0082] For example, the preprocessing module can perform image preprocessing operations on image samples in the image sample sequence; wherein, the image preprocessing operations may include: first, performing normalization processing on the image samples to uniformly scale the pixel values ​​in the image samples to a specified range to eliminate brightness differences caused by different lighting conditions, and obtaining an image normalization result; then, performing denoising processing on the image normalization result through a median filter to remove salt-and-pepper noise and obtain an image denoising result; then, performing illumination correction on the image denoising result to compensate for local overexposure or underexposure problems caused by the non-uniformity of the endoscopic light source, and obtaining an image correction result; finally, performing adaptive histogram equalization on the image correction result to enhance image contrast and highlight the texture details of the lesion area, so that the image features have high recognizability and stability in subsequent processing, thereby obtaining the image preprocessing result.

[0083] For example, the preprocessing module can also perform the following processing on the catheter samples in the catheter sample sequence: filter out high-frequency noise in the catheter samples to obtain catheter filtering results, and then remove abnormal data in the catheter filtering results to obtain catheter preprocessing results.

[0084] For example, the image preprocessing result is used to extract features through the first model + FPN to obtain a sample image feature set. The catheter preprocessing result is used to extract features through the second model + FPN to obtain a sample catheter feature set. Then, the sample image feature set and the sample catheter feature set are fused through the feature fusion module to obtain the sample feature fusion result.

[0085] For example, the optimization module can determine the sample control information based on the sample feature fusion result, and then generate a parameter adjustment instruction based on the difference information between the sample control information and the actual control information for endoscopes and catheters in the sample data. Then, the parameters of the feature extraction module are adjusted according to the parameter adjustment instruction until the difference information represents the degree of difference between the actual control information and the sample control information less than or equal to the difference threshold. At this time, the feature extraction module and the feature fusion module with adjusted parameters can be applied to the technical solution provided in this application.

[0086] Specifically, the sample feature fusion results can be processed using a fully connected layer to determine the initial anchoring position at time t. Then, the ideal bending radius of the catheter at time t is calculated using the catheter bending radius fitting formula shown in equation (11). : (11) in, ε is the elastic modulus of the catheter, used to characterize the flexibility of the catheter material. The depth of the catheter at time t represents the catheter sample.

[0087] Next, taking the optimization process of the initial anchoring position at time t as an example, the optimization process of the optimization module is explained by combining equation (12) with the endoscope focal length. and the sample height of the image samples Calculate the ideal angle of the endoscope at time t. : (12) The bending radius of the catheter at time t is calculated using equation (13). : (13) in, The normalization coefficient is... Let t be the initial anchoring position at time t-1.

[0088] The perspective at time t is calculated using equation (14). : (14) in, This represents the image height.

[0089] For example, the optimization module can calculate the training loss at time t using the geometric constraint loss function shown in equation (15). : (15) in, The bending radius constraint weight, The above two parameters, representing the view range constraint weights, are used to balance the importance of the bending radius and the view range constraint.

[0090] For example, the optimization module can perform optimization using equation (16) combined with the gradient descent method to obtain the optimized anchoring position at time t. : (16) in, This is a smoothing coefficient used to improve the continuity of anchorage position optimization, thereby reducing the probability of abrupt changes in anchorage position.

[0091] It should be noted that the initial anchor pose at time t may include the initial anchor position at time t.

[0092] The supervised training system provided by the above training structure can improve the relevance and accuracy of feature extraction in the feature extraction module.

[0093] Figure 3 Another schematic diagram of the endoscope-catheter coordinated control and anchoring optimization system provided in this application embodiment. (See diagram below.) Figure 3 As shown, the endoscopy data input unit can be used to input image sequences, and the catheter data input unit can be used to input pose sequences. In other words, the acquisition module can include an endoscopy data input unit and a catheter data input unit.

[0094] For example, the preprocessing module can perform preprocessing operations on the image sequence and the pose sequence, and input the preprocessed image sequence and pose sequence into the feature extraction model, so that the feature extraction model can extract features from the preprocessed image sequence through the first model + FPN to obtain the image feature set, and extract features from the preprocessed pose sequence through the second model + FPN to obtain the pose feature set.

[0095] For example, the feature fusion module can perform fusion processing on the image feature set and the pose feature set using the method provided in the foregoing embodiments to obtain the feature fusion result.

[0096] For example, the control module can be the collaborative control module in the foregoing embodiments, which can process the feature fusion results to determine the anchor pose and endoscope pose, and can also generate control signals for collaborative control of the anchor pose and endoscope pose.

[0097] For example, the output module can output control signals containing endoscope position adjustment instructions and catheter position adjustment instructions to prompt doctors or professional technicians to control the endoscope and / or catheter according to the endoscope position adjustment instructions and / or catheter position adjustment instructions.

[0098] The above structure enables the organic integration of image sequence and pose sequence acquisition, feature extraction and fusion, control signal determination, and output, thereby improving the efficiency and safety of endoscopic-catheter exploration of target tissues.

[0099] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0100] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0101] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0102] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0103] It should be noted that the aforementioned computer-readable storage media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0104] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0105] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware nodes. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0107] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An endoscope-catheter collaborative control and anchoring optimization system, characterized in that, The method comprises the following steps: The acquisition module is used for continuously acquiring an image sequence through an endoscope and continuously acquiring a pose sequence of a catheter during the exploration of a target tissue through the endoscope and the catheter; The feature extraction module is used for extracting features from the image sequence to obtain an image feature set corresponding to the image sequence and extracting features from the pose sequence to obtain a pose feature set of the catheter; The feature fusion module is used for fusing the image feature set and the pose feature set to obtain a feature fusion result; The collaborative control module is used for determining a control signal based on the feature fusion result and collaboratively controlling an anchor pose of the catheter and an endoscope pose of the endoscope based on the control signal until the catheter is located within a field of view of the endoscope.

2. The system of claim 1, wherein, The feature extraction module is used for determining an image weight sequence based on the image sequence and determining a pose weight sequence based on the pose sequence; The feature fusion module is used for fusing the image feature set and the pose feature set based on the image weight sequence and the pose weight sequence to obtain the feature fusion result.

3. The system of claim 2, wherein, The feature extraction module is used for extracting an edge density sequence corresponding to a tissue edge of the target tissue in the image sequence, extracting texture features of the image sequence to obtain a texture sequence, and determining the image weight sequence based on the edge density sequence and the texture sequence.

4. The system of claim 2, wherein, The feature extraction module is used for extracting features of an mth image in the image sequence through an image feature pyramid network to obtain an mth sub-feature set, integrating the mth sub-feature set to obtain an image feature of the mth image; wherein the mth sub-feature set comprises local features, contour features, texture features, context features and global scene features of the target tissue; m is an integer greater than or equal to 1.

5. The system of claim 2, wherein, The feature extraction module is used for performing statistics on poses in the pose sequence to obtain a pose change rate, determining a pose entropy sequence of the pose sequence, and determining the pose weight sequence based on the sequence of the pose change rate and the pose entropy sequence.

6. The system of claim 2, wherein, The feature extraction module is used for extracting features of consecutive N poses in the pose sequence through a pose feature pyramid network to obtain a pose sub-feature set, integrating the pose sub-feature set to obtain a pose feature; wherein the pose sub-feature set comprises bending tendency, motion inertia, path smoothing state, displacement mode and displacement trajectory of the catheter; the pose feature set comprises a plurality of sets of the pose features; N is an integer greater than 1.

7. The system of claim 1, wherein, The collaborative control module is used for obtaining an endoscope focal length of the endoscope, optimizing the endoscope pose based on the endoscope focal length and image parameters of the image sequence to obtain an endoscope pose adjustment instruction for the endoscope; wherein the control signal comprises the endoscope pose adjustment instruction.

8. The system of claim 1, wherein, The cooperative control module is configured to correct the anchor pose based on the insertion length of the catheter and a historical anchor pose, to obtain a catheter pose adjustment instruction for the anchor pose; and the control signal at least includes the catheter pose adjustment instruction.

Images