Three-dimensional reconstruction method and system for nasal jejunum catheterization robot

Abstract

The invention relates to the technical field of medical robots, in particular to a three-dimensional reconstruction method and system for a nasal jejunum catheterization robot. The method comprises the following steps: acquiring endoscope image data in real time through an endoscope camera and a data transmission channel, and performing preliminary denoising and enhancement; a parallel processing architecture is adopted, and the endoscope image data is divided, subjected to image denoising and enhancement again, improved in image contrast and subjected to distortion correction processing in sequence; carrying out dynamic modeling on the newly collected image data in the nasal jejunum dynamic three-dimensional model; performing dynamic path planning and attitude adjustment, and detecting errors in real time; the system comprises a data acquisition module, a data parallel processing module, a dynamic modeling module and an adjusting module. In this way, the real-time performance, the soft tissue adaptability and the catheter positioning information precision can be met, and therefore the reliability of nasal jejunum catheterization operation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical robots, and in particular, to a three-dimensional reconstruction method and system for a nasojejunal intubation robot. Background Art

[0002] The insertion of medical catheters is one of the core techniques in clinical nursing. Hundreds of millions of catheters are inserted globally every year to support treatments such as drug administration, oxygen supply, and nutrition delivery. Nasojejunal catheters (such as nasogastric tubes and nasointestinal tubes) are an important part of them. Approximately 40 million nasogastric tubes are consumed globally every year, and about 17% of long-term nutrition support patients use nasogastric tubes or nasointestinal tubes. With the aging of the population and the increase in related diseases, the demand for nasojejunal intubation continues to rise. However, the currently commonly used blind insertion method relies on body surface measurement and empirical judgment, lacks visual guidance, and is prone to inaccurate catheter position or difficult tip control, increasing the risks of tissue contusion, tearing, etc., and even causing inflammatory reactions and severe internal bleeding. It is reported that the success rate of blindly inserting nasointestinal tubes is only 61.9%, and the rate of nasogastric tubes misentering the airway is 3.2%, especially with a higher risk in patients with swallowing disorders, and it may even misenter the brain and endanger life.

[0003] In recent years, the application of robot-assisted technology in the medical field has significantly improved the success rate of nasojejunal intubation. Some medical institutions have tried to use endoscopic navigation robots to assist doctors in operations through real-time image processing and path planning, thereby reducing the risk of misinsertion and tissue damage. In addition, artificial intelligence-based image analysis technology has gradually been applied to nasojejunal three-dimensional reconstruction, providing support for the recognition of complex anatomical structures and path planning. However, there are still many deficiencies in the current application of robot-assisted technology in nasojejunal intubation care. On the one hand, most existing three-dimensional reconstruction methods are based on offline processing and are difficult to generate accurate anatomical structure models in real time. Especially during the operation, they lack efficient real-time data processing and dynamic update capabilities. This directly affects the accuracy of catheter placement path planning and is difficult to meet the high requirements for real-time performance and robustness during the operation. On the other hand, the dynamic characteristics of nasojejunal soft tissues, such as peristalsis and deformation, pose challenges to traditional modeling methods based on rigid assumptions. These methods are insufficient in adapting to the complex and dynamic internal environment of the nasojejunum and cannot accurately capture the real-time changes in the soft tissue morphology, resulting in deviations between the model and the actual anatomical structure. In addition, when the robot works inside the nasojejunum, the inaccurate pose estimation caused by its own deformation and the movement of the nasojejunum will lead to low accuracy of catheter positioning information, affecting its path planning and execution in complex anatomical regions. In addition, the limitations of endoscopic image quality, such as insufficient light, motion blur, and limited field of view, also limit the performance of existing visual navigation technologies in complex environments and increase the risk of misinsertion during catheter placement operations.

[0004] Therefore, it is highly necessary to propose a three-dimensional reconstruction method and system for a naso-jejunal intubation robot that can meet the requirements of real-time performance, soft tissue adaptability, and catheter positioning information accuracy, thereby improving the reliability of naso-jejunal intubation operations. Summary of the Invention

[0005] The purpose of the present invention is to provide a three-dimensional reconstruction method and system for a naso-jejunal intubation robot, aiming to meet the requirements of real-time performance, soft tissue adaptability, and catheter positioning information accuracy, thereby improving the reliability of naso-jejunal intubation operations.

[0006] To achieve the above objective, a three-dimensional reconstruction method for a naso-jejunal intubation robot adopted by the present invention includes the following steps:

[0007] Real-time obtain endoscopic image data through an endoscopic camera and a data transmission channel, and perform preliminary denoising and enhancement.

[0008] Adopt a parallel processing architecture to sequentially divide the endoscopic image data, perform image denoising and enhancement again, enhance the image contrast, and perform distortion correction processing.

[0009] Perform dynamic modeling on the newly acquired image data in the naso-jejunal dynamic three-dimensional model.

[0010] Perform dynamic path planning and attitude adjustment, and detect errors in real time.

[0011] Among them, in the step of performing dynamic modeling on the newly acquired image data in the naso-jejunal dynamic three-dimensional model:

[0012] Construct a multi-source knowledge base for the naso-jejunum, and divide the structure of the multi-source knowledge base into a static template, a dynamic template, and a personalized template.

[0013] Based on the multi-source knowledge base, construct static and motion three-dimensional models of the naso-jejunum, and couple the static and motion three-dimensional models into a naso-jejunal dynamic three-dimensional model.

[0014] Dynamically adjust the coupling weight of the static and motion models, and perform local deformation of the naso-jejunal dynamic three-dimensional model.

[0015] Among them, after the step of dynamically adjusting the coupling weight of the static and motion models and performing local deformation of the naso-jejunal dynamic three-dimensional model:

[0016] Use the naso-jejunal dynamic three-dimensional model as prior information to perform real-time three-dimensional reconstruction of the naso-jejunum.

[0017] Among them, in the step of constructing a multi-source knowledge base for the naso-jejunum and dividing the structure of the multi-source knowledge base into a static template, a dynamic template, and a personalized template:

[0018] The data of the multi-source knowledge base includes static images, ultrasound data, endoscopic videos, anatomical and physiological data, and path planning reference templates.

[0019] Among them, in the step of constructing the nasojejunal multi-source knowledge base and dividing the structure of the multi-source knowledge base into static templates, dynamic templates, and personalized templates:

[0020] The static template is used to store the basic anatomical structure of the nasojejunum;

[0021] The dynamic template is used to store dynamic motion characteristics;

[0022] The personalized template is used to store the characteristic information of the patient.

[0023] Among them, in the step of performing dynamic path planning and pose adjustment and real-time error detection:

[0024] Perform environmental perception and feature extraction based on deep learning;

[0025] Perform pose estimation on the robot based on multi-sensor data, and fuse environmental perception and feature extraction to estimate the position and pose of the robot inside the nasojejunum.

[0026] Among them, after the step of performing pose estimation on the robot based on multi-sensor data, and fusing environmental perception and feature extraction to estimate the position and pose of the robot inside the nasojejunum:

[0027] Compare the current pose with the predicted pose, generate targeted correction information based on real-time sensor data, and feedback the error.

[0028] The present invention also provides a three-dimensional reconstruction system for a nasojejunal intubation robot, including a data acquisition module, a data parallel processing module, a dynamic modeling module, and an adjustment module; wherein:

[0029] The data acquisition module is used to, through an endoscopic camera and a data transmission channel, acquire endoscopic image data in real time, and perform preliminary denoising and enhancement;

[0030] The data parallel processing module is used to, using a parallel processing architecture, sequentially divide the endoscopic image data, perform image denoising and enhancement again, improve the image contrast, and perform distortion correction processing;

[0031] The dynamic modeling module is used to perform dynamic modeling on the newly acquired image data in the nasojejunal dynamic three-dimensional model;

[0032] The adjustment module is used to perform dynamic path planning and pose adjustment, and detect errors in real time.

[0033] A three-dimensional reconstruction method and system for a nasojejunal intubation robot according to the present invention performs the following steps by using the data acquisition module, the data parallel processing module, the dynamic modeling module, and the adjustment module: real-time acquisition of endoscopic image data through an endoscopic camera and a data transmission channel, and preliminary denoising and enhancement; using a parallel processing architecture to sequentially divide the endoscopic image data, perform image denoising and enhancement again, enhance the image contrast, and perform distortion correction processing; perform dynamic modeling on the newly acquired image data in a nasojejunal dynamic three-dimensional model; perform dynamic path planning and attitude adjustment, and detect errors in real time; and can meet the requirements of real-time performance, soft tissue adaptability, and catheter positioning information accuracy, thereby improving the reliability of nasojejunal intubation operations. Description of the Drawings

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0035] Figure 1 is a flowchart of the steps of the three-dimensional reconstruction method for a nasojejunal intubation robot according to the present invention.

[0036] Figure 2 is a flowchart of the steps of S300 of the present invention.

[0037] Figure 3 is a flowchart of the steps of S400 of the present invention.

[0038] Figure 4 is a schematic flow diagram of the three-dimensional reconstruction method for a nasojejunal intubation robot according to the present invention.

[0039] Figure 5 is a schematic flow diagram of S300 of the present invention.

[0040] Figure 6 is a schematic flow diagram of S400 of the present invention.

[0041] Figure 7 is a schematic structural diagram of the three-dimensional reconstruction system for a nasojejunal intubation robot according to the present invention.

[0042] Figure 8 is a schematic structural diagram of the electronic device according to the present invention.

[0043] 501 - Data acquisition module, 502 - Data parallel processing module, 503 - Dynamic modeling module, 504 - Adjustment module. Detailed Embodiments

[0044] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application.

[0045] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "the", and "said" used in this application and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0046] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0047] Please refer to Figures 1 to 6 , where Figure 1 is a flowchart of the steps of a three-dimensional reconstruction method for a nasoenteral intubation robot, Figure 2 is a flowchart of the steps of S300, Figure 3 is a flowchart of the steps of S400, Figure 4 is a schematic diagram of the process of a three-dimensional reconstruction method for a nasoenteral intubation robot, Figure 5 is a schematic diagram of the process of S300, Figure 6 is a schematic diagram of the process of S400.

[0048] The present invention provides a three-dimensional reconstruction method for a nasoenteral intubation robot, including the following steps:

[0049] S100: Real-time obtain endoscopic image data through an endoscopic camera and a data transmission channel, and perform preliminary denoising and enhancement;

[0050] In this embodiment, endoscopic image data is acquired in real time through a high-frame-rate and high-resolution endoscopic camera (such as Sony IMX series CMOS image sensors) and an optimized data transmission channel (such as Thunderbolt interface or PCI-E channel) to ensure the minimization of data transmission latency. For scenarios with extremely high real-time requirements, an FPGA hardware acceleration module can be combined to directly complete some preliminary denoising and enhancement operations at the acquisition end to reduce the computational pressure on the backend.

[0051] S200: Adopt a parallel processing architecture to sequentially divide the endoscopic image data, perform image denoising and enhancement again, improve image contrast, and perform distortion correction processing;

[0052] In this embodiment, after image acquisition, a parallel processing architecture is adopted to divide each frame of image into multiple data blocks, which are processed simultaneously by multiple GPU threads, thus significantly improving the speed of data cleaning and providing high-quality input for subsequent reconstruction. In terms of image denoising and enhancement, GPU acceleration technologies (such as CUDA or OpenCL frameworks) are used to achieve fast denoising and enhancement of endoscopic images. For the low-light and high-noise characteristics of endoscopic acquisitions, an image denoising algorithm based on convolutional neural networks (such as Denoising Autoencoder or FFDNet) can be adopted, and techniques such as histogram equalization and adaptive illumination correction are combined to improve image contrast. In terms of distortion correction, due to the wide-angle design of the endoscopic lens, obvious image barrel or pincushion distortion may occur. Use the distortion correction module based on OpenCV to obtain the lens parameter matrix (intrinsic and extrinsic parameters) through camera calibration to achieve efficient geometric correction. For fisheye lenses, a polynomial correction model can be introduced.

[0053] S300: Dynamically model the newly acquired image data in the nasojejunal dynamic three-dimensional model;

[0054] Furthermore, in the step of dynamically modeling the newly acquired image data in the nasojejunal dynamic three-dimensional model:

[0055] S301: Construct a multi-source knowledge base for the nasojejunum, and divide the structure of the multi-source knowledge base into static templates, dynamic templates, and personalized templates; the data in the multi-source knowledge base includes static images, ultrasound data, endoscopic videos, anatomical and physiological data, and path planning reference templates. The static template is used to store the basic anatomical structure of the nasojejunum, the dynamic template is used to store dynamic motion characteristics, and the personalized template is used to store the characteristic information of the patient;

[0056] S302: Based on the multi-source knowledge base, construct static and motion three-dimensional models of the nasojejunum, and couple the static and motion three-dimensional models into a nasojejunal dynamic three-dimensional model;

[0057] S303: Dynamically adjust the coupling weights of the static and motion models to perform local deformation of the dynamic three-dimensional model of the nasojejunum;

[0058] S304: Use the dynamic three-dimensional model of the nasojejunum as prior information to perform real-time three-dimensional reconstruction of the nasojejunum.

[0059] In this embodiment, based on the 3D Gaussian Splatting (3D GS) algorithm framework, the Gaussian distribution is used to describe the density distribution of the spatial point cloud, which is suitable for the dynamic update of sparse point clouds. Combining prior information to optimize the algorithm performance and improve the dynamic adaptability of the algorithm for the nasojejunum. At the same time, in order to reduce the computational burden, an incremental update mechanism is adopted, and only the newly acquired image data is dynamically modeled, rather than performing repeated calculations on the entire three-dimensional model, which greatly improves the processing speed.

[0060] Among them, construct a multi-source knowledge base for the nasojejunum:

[0061] In terms of data sources, a variety of medical imaging data need to be integrated to cover the structural characteristics and dynamic change characteristics of the nasojejunum from macro to micro, including high-resolution static images provided by CT and MRI to present the overall structure of the nasojejunum; ultrasound data is used to supplement the dynamic change characteristics of soft tissues; endoscopic video records the visual details during real-time operation, such as surface texture and motion patterns. In addition, anatomical and physiological data, such as tissue thickness, peristaltic frequency, local pressure distribution, etc., need to be combined to establish a detailed anatomical structure library. At the same time, it is also necessary to collect the catheterization path, error and risk areas in the surgical record data to form a reference template for path planning. Patient population data is also important. This data comes from patients of different ages, genders and health statuses, which can effectively ensure that the knowledge base has wide applicability.

[0062] In order to ensure the effective integration of multi-source data, strict preprocessing is required. The spatial, intensity and time dimensions of multi-modal data are unified through normalization processing. Use rigid registration algorithms based on feature points (such as SIFT) and non-rigid registration algorithms based on optical flow fields to align CT, MRI and endoscopic data to a unified coordinate system; segment and label the key anatomical structures of the nasojejunum through deep learning segmentation models (such as UNet or nnU-Net), such as the esophageal inlet, gastric fundus, pylorus, etc.

[0063] To construct a comprehensive multi-source knowledge base of the nasojejunum, its structure needs to be divided into three major parts: static templates, dynamic templates, and personalized templates, which store the basic anatomical structure, dynamic motion characteristics, and personalized feature information of specific patients of the nasojejunum respectively. The static template is mainly used to describe the basic geometric shape and anatomical features of the nasojejunum, covering contents such as curvature, tube diameter, volume, and boundary. The volume and boundary information of typical anatomical sites, such as the esophageal inlet, gastric fundus, and pylorus, will also be clearly marked for navigation and planning. The static template is stored using triangular mesh models (such as STL / OBJ formats) and point cloud data (PLY format), and this data representation method facilitates subsequent three-dimensional modeling and reconstruction operations. The dynamic template focuses on the time-series motion characteristics of the nasojejunum, recording the frequency, amplitude of its peristalsis, and the direction of deformation. This part of the data is extracted from sources such as dynamic MRI and endoscopic videos and stored using time-series models (such as JSON or HDF5 formats) to ensure that the dynamic change characteristics of the nasojejunum at different time steps can be described. At the same time, by combining the information of the motion vector field, the dynamic behavior of the nasojejunum can be captured more precisely. The personalized template is used to store the individualized nasojejunum model of specific patients, including anatomical shape, motion characteristics, and local features under pathological conditions. This part of the data comes from the actual examination results of patients, and a patient-specific nasojejunum model is generated by combining the static template and the dynamic template. For example, for a patient with gastric ulcer, the template will mark the location of the ulcer and its impact on local gastric peristalsis, providing accurate reference for clinical operations and decision-making. The personalized template can not only guide the navigation operation in complex cases, but also directly support the positioning and path planning during the work of the robot.

[0064] Through such a multi-level design, the multi-source knowledge base of the nasojejunum can not only comprehensively express the anatomical and dynamic characteristics of the nasojejunum, but also meet the individualized operation requirements, providing efficient prior constraints and guidance for robot-assisted nasojejunal surgery and nursing.

[0065] Among them, constructing a dynamic three-dimensional model of the nasojejunum:

[0066] Based on the multi-source knowledge base of the nose and jejunum, static and dynamic three-dimensional models of the nose and jejunum can be constructed. First, the static three-dimensional model of the nose and jejunum is constructed. By fusing multimodal images (such as CT and MRI), these image data are aligned to a unified coordinate system using registration technology, and then the geometric surface structure is reconstructed through surface point cloud generation algorithms (such as Poisson Surface Reconstruction). At the same time, in order to enhance the realism and detail expression of the model, the high-resolution texture collected by the endoscope can be mapped to the surface of the static model. In order to optimize the computational efficiency, a hierarchical modeling method is used to perform local refinement on complex areas such as the fundus and pylorus, while maintaining the overall modeling of other parts of the nose and jejunum. The key to the static model is to accurately describe the overall anatomical morphology of the nose and jejunum, providing a structural basis and boundary constraints for subsequent dynamic modeling.

[0067] Then, the three-dimensional model of nasojejunal motion is constructed. Through ultrasound images or endoscopic videos, the motion characteristics of nasojejunal soft tissue in time series are extracted, and the motion law of nasojejunum is identified by time series analysis method. Then, based on the finite element analysis (FEM) method, the motion of nasojejunal soft tissue, such as peristalsis and deformation, is simulated. By processing the time series data, the displacement field (the displacement change of each point over time), the strain field (reflecting the degree of deformation of the tissue) and other dynamic characteristics (such as peristalsis frequency and period) are extracted to form a parametric model describing the soft tissue motion pattern. In addition, in order to improve the dynamic adaptability of the model, a high-dimensional motion vector field can be introduced to record the complex deformation characteristics of the local area to ensure that the motion model can cover the motion behavior in a variety of clinical scenarios. The motion model can reflect the deformation law of the nasojejunum in different operation scenarios, especially the morphological changes under the action of peristalsis, and provide support for the dynamic adaptability of three-dimensional reconstruction.

[0068] Finally, the static and dynamic three-dimensional models of the nasojejunum are coupled to form a dynamic three-dimensional model of the nasojejunum, which serves as the prior information for the subsequent real-time three-dimensional reconstruction of the nasojejunum, that is, the constraints of the three-dimensional reconstruction. In order to cope with the complex motion characteristics of the soft tissue of the nasojejunum, the static and dynamic models need to be coupled through a rigid-flexible hybrid modeling method. The static model provides the geometric boundary constraints of the nasojejunum, while the dynamic model superimposes the motion characteristics on the static structure through the mapping of the displacement field. At the same time, it is necessary to dynamically adjust the coupling weights of the static and dynamic models based on the distribution of rigid and flexible areas in the actual data to ensure the accuracy of local deformation and the continuity of the overall model. During the coupling process, the Laplace deformation algorithm is used to deform the static model to ensure that the geometric continuity of the model is maintained while the motion characteristics are transmitted. In order to further optimize the coupling effect, deep learning technology (such as LSTM network) can be combined to predict the peristaltic trajectory and deformation pattern, and these prediction results can be superimposed on the dynamic model as constraints to generate a dynamic three-dimensional model of the nasojejunum that can reflect the overall morphology and adapt to the movement of soft tissues.

[0069] Among them, the real-time three-dimensional reconstruction of the nasojejunum combined with prior information:

[0070] During the three-dimensional reconstruction of the nasojejunum, 3D Gaussian Splatting (3D GS) is an efficient method that can directly process point cloud data. To improve the adaptability and accuracy of the 3D GS algorithm in complex dynamic scenarios, a dynamic three-dimensional model is introduced as prior information to optimize the reconstruction process.

[0071] The coupled dynamic three-dimensional model of the nasojejunum provides prior constraint information on geometric shape, motion law, and texture distribution. In the 3D GS algorithm, the reconstruction process is based on point clouds and uses Gaussian kernels to simulate the distribution characteristics of each point. The geometric shape provided by the dynamic model is used to initialize the position of the point cloud, while the motion characteristics (such as displacement field and deformation law) are used to dynamically adjust the parameters of the Gaussian kernel, including the center point position, anisotropic covariance matrix, and intensity value, etc.

[0072] In the nasojejunum scenario, the peristalsis and deformation of tissues make the collected point cloud data prone to inconsistent distribution and uneven density. The motion information of the dynamic model guides the fusion of point clouds and the adjustment of Gaussian kernels by providing displacement field constraints. Specifically, during the point cloud fusion process of 3D GS, the displacement field of the dynamic model is introduced as a constraint term to optimize the spatial update of each point, enabling it to more accurately follow the dynamic deformation of the nasojejunum. The adjustment of the covariance matrix is based on the deformation law of the dynamic model and is used to control the shape and range of the Gaussian kernel to ensure better adaptability to the details of dynamic tissues.

[0073] Dynamic model M dyn provides geometric shape and motion characteristics, including displacement field d(x,t), texture feature T(x), and covariance matrix information Σ(x).

[0074] Geometric initialization:

[0075] x init = x static + d(x,t);

[0076] Among them, x static is the point in the static model, and d(x,t) is the displacement field provided by the dynamic model, representing the dynamic displacement of the point over time.

[0077] Initial parameters of the Gaussian kernel:

[0078] G(x) = {x init , Σ init , w init};

[0079] Among them, x initis the initialized Gaussian kernel center, Σ init is the covariance matrix provided by the dynamic model, w init is the intensity of the Gaussian kernel.

[0080] 3D GS reconstructs by distributing point cloud data in the form of Gaussian kernels in space, and the motion characteristics of the dynamic model are used to optimize the fusion of point clouds.

[0081] Gaussian kernel covariance matrix update:

[0082]

[0083] where R t is the rotation matrix estimated based on the displacement field, Σ noise represents the covariance of the noise, controlling the Gaussian kernel range.

[0084] Fusion weight update:

[0085] w t+1 = αw t +(1 - α)w data ;

[0086] where α is the fusion weight factor, w data is the intensity value of the new point cloud.

[0087] Position update:

[0088] x t+1 = x t + Δd t ;

[0089] where Δd t is the displacement field increment of the current frame.

[0090] Endoscopic images and other sensor data usually have noisy point clouds, which affect the reconstruction accuracy. The texture and geometric feature information of the dynamic model can be used to eliminate point clouds with large errors. For example, through texture matching in the dynamic model, point clouds inconsistent with the actual anatomical features are screened; using the displacement field and motion characteristics provided by the dynamic model, pseudo points inconsistent with the peristalsis law are identified. Such an optimization process can significantly reduce noise interference and improve the fineness of the reconstruction.

[0091] Texture matching:

[0092] ΔT = ||T obs (x) - T model (x)||;

[0093] When ΔT > ò T (texture error threshold), the corresponding points are eliminated.

[0094] Motion consistency:

[0095] Δd = ||d obs (x,t) - d model (x,t)||;

[0096] When Δd > ò d (motion error threshold), eliminate the corresponding points.

[0097] Introduce the energy constraint of prior information to optimize the Gaussian kernel distribution to ensure the accuracy of the model.

[0098] Reconstruction error function:

[0099]

[0100] where x data , Σ model and w model are the observed values and model values of the point cloud position, covariance, and intensity respectively; λ1 and λ2 are weight coefficients.

[0101] Gradient update:

[0102]

[0103] where η is the learning rate.

[0104] The finally reconstructed 3D model is represented by the optimized Gaussian kernel as:

[0105]

[0106] This model integrates static geometric shapes, dynamic motion laws, and real-time point cloud data to provide support for high-precision 3D reconstruction.

[0107] After introducing the dynamic model as a prior, the 3D GS algorithm shows higher accuracy and stability in the complex scenario of nasojejunal soft tissue. The optimized Gaussian kernel distribution can more accurately describe the dynamic deformation of the nasojejunal tissue, and the details of the reconstruction results are richer, providing reliable support for catheter placement path planning. At the same time, it effectively reduces the errors caused by noise and motion, thus improving the accuracy and safety of catheter placement nursing operations.

[0108] S400: Perform dynamic path planning and attitude adjustment, and detect errors in real time.

[0109] Furthermore, in the steps of performing dynamic path planning and attitude adjustment and detecting errors in real time:

[0110] S401: Perform environmental perception and feature extraction based on deep learning;

[0111] S402: Estimate the pose of the robot based on multi-sensor data, and fuse environmental perception and feature extraction to estimate the position and pose of the robot inside the nasojejunum;

[0112] S403: Compare the current pose with the predicted pose, generate targeted correction information based on real-time sensor data, and feedback the error.

[0113] In this embodiment, a multi-threaded computing framework is designed to run 3D reconstruction, error detection, and display separately without interference. The reconstruction results are transmitted to the robot control system in real time through an independent thread for dynamic path planning and pose adjustment. At the same time, real-time error detection is performed to compare the difference between the real-time generated 3D model and the expected anatomical structure and identify possible error sources. When a significant deviation is detected, the model is dynamically corrected through the feature extraction module of the deep learning network to ensure the accuracy and robustness of the reconstruction results.

[0114] Among them, environmental perception and feature extraction based on deep learning:

[0115] Deep learning, especially convolutional neural networks, has significant advantages in extracting structural features in complex environments and can be used to extract key structural information of the nasojejunum from endoscopic images or other sensor data. These structural features, including edges, branch points, narrow areas, etc., are crucial for achieving high-precision pose estimation. By using an annotated medical image dataset (such as CT, MRI, and endoscopic images containing nasojejunum structures), a deep learning model, such as ResNet or U-Net, can be trained to automatically learn and extract the anatomical features of the nasojejunum. Specifically, ResNet, with its residual network design, can effectively capture deep feature information and is suitable for processing complex anatomical shapes; U-Net focuses on image segmentation tasks and can generate high-resolution anatomical label maps to accurately segment the structures of various parts inside the nasojejunum (such as the esophagus, stomach, and intestine). Through convolutional operations, these models can automatically identify the key anatomical elements inside the nasojejunum and convert them into high-quality feature inputs for subsequent algorithms. This deep learning-based feature extraction method not only significantly improves the feature recognition ability in complex scenarios but also provides a solid foundation for subsequent pose estimation.

[0116] Among them, classical pose estimation methods:

[0117] Classic pose estimation methods, such as Kalman filter, Extended Kalman Filter, and Particle Filter, perform excellently in handling pose prediction tasks based on multi-sensor data, especially suitable for robot localization in dynamic and complex environments. These methods can effectively fuse input data from different sensors (such as Inertial Measurement Unit IMU and visual sensors) and the control instructions of the robot itself, comprehensively analyze environmental observation information, and thus calculate the precise position and posture of the robot in the nasojejunum.

[0118] Among them, the Extended Kalman Filter is a state estimation method widely used in nonlinear systems, especially suitable for dealing with the complex anatomical morphology and dynamic motion characteristics of the nasojejunum. In the implementation process of the Kalman filter, first, the pose state is predicted using the robot motion model, and the prediction result is updated and corrected in combination with sensor data (such as the angular velocity and acceleration information provided by the IMU and the key feature points in the endoscopic image). This prediction-update iterative process can significantly improve the accuracy and stability of pose estimation. For example, the acceleration and angular velocity data provided by the IMU sensor can be used to estimate the instantaneous motion state of the robot, while the feature points in the endoscopic image are used as observation values to correct the pose. The state update equation of the Kalman filter is as follows:

[0119] x t|t =x t|t-1 +K t (z t -Hx t|t-1 );

[0120] Among them, x t|t is the corrected state, z t is the observation value, H is the observation model matrix, and K t is the Kalman gain. The Kalman filter dynamically adjusts the gain weight to balance the error between the motion model prediction and sensor observation, thereby achieving continuous and accurate estimation of the robot's pose.

[0121] Among them, the fusion of deep learning and classic pose estimation:

[0122] By using the feature information extracted by the deep learning model as the observation input of the classic pose estimation algorithm, the accuracy of the observation model can be significantly improved. Specifically, the deep learning network can extract key feature points or boundary information from sensor data. These information are used as constraints and input into the Kalman filter, which can effectively correct its state prediction result, thereby helping the model to more accurately estimate the current position and posture of the robot in the nasojejunum. This deep learning-assisted pose estimation method combines perception and mathematical modeling, making up for the deficiencies of single methods in complex scenarios.

[0123] Furthermore, the fusion of deep learning and Kalman filtering also realizes the dynamic optimization of pose estimation by introducing an error feedback mechanism. When a large error occurs in the pose predicted by Kalman filtering in a dynamic environment, the deep learning model can generate targeted correction information based on real-time sensor data and feedback the error into Kalman filtering, thereby continuously optimizing its prediction results. This error correction mechanism improves the robustness and accuracy of the entire system through the complementary advantages of the environmental perception ability of deep learning and the state estimation ability of classical methods, especially when dealing with complex movements in the nasojejunum and the deformation of the robot itself.

[0124] In summary, the present invention proposes a high-precision positioning algorithm that combines deep learning with classical pose estimation methods, providing an innovative solution to the positioning problem of a robot working in the nasojejunum. This algorithm makes full use of the powerful capabilities of deep learning in environmental perception and feature extraction, as well as the mature advantages of classical pose estimation methods in state prediction and dynamic optimization, to construct a pose estimation framework with both real-time performance and high precision. Through the deep learning model, the system can efficiently extract key features in the complex environment of the nasojejunum, providing accurate environmental information support for pose estimation. The classical pose estimation method combines sensor observation data to achieve stable pose prediction in the dynamically changing nasojejunum. The deep integration of the two not only significantly improves the positioning accuracy but also enhances the robustness of the algorithm when facing challenges such as complex movements of nasojejunal soft tissues and deformation of the robot itself. In addition, this algorithm introduces an error feedback and dynamic optimization mechanism, which can adjust and correct the deviation in pose estimation in real time to ensure the positioning stability and accuracy of the robot during fine operations. The combination of deep learning and classical methods achieves effects that are difficult to achieve by a single technology through complementary advantages, providing reliable technical support for the precise navigation and operation of robots in complex biological environments. The real-time performance, accuracy, and robustness of this algorithm make it have broad application potential.

[0125] Corresponding to the foregoing embodiment of the three-dimensional reconstruction method for a nasojejunal intubation robot, the present application also provides an embodiment of a three-dimensional reconstruction system for a nasojejunal intubation robot.

[0126] Figure 7 It is a block diagram of a three-dimensional reconstruction system for a nasojejunal intubation robot shown according to an exemplary embodiment. Referring to Figure 7 , the system may include: a data acquisition module 501, a data parallel processing module 502, a dynamic modeling module 503, and an adjustment module 504; where:

[0127] The data acquisition module 501 is configured to, through an endoscope camera and a data transmission channel, acquire endoscope image data in real time and perform preliminary denoising and enhancement;

[0128] The data parallel processing module 502 is configured to divide, perform denoising and enhancement on the endoscopic image data again, improve the image contrast, and perform distortion correction processing on the endoscopic image data in sequence by adopting a parallel processing architecture;

[0129] The dynamic modeling module 503 is configured to perform dynamic modeling on the newly acquired image data in the nasal jejunum dynamic three-dimensional model;

[0130] The adjustment module 504 is configured to perform dynamic path planning and attitude adjustment, and detect errors in real time.

[0131] In this embodiment, the data acquisition module 501 acquires endoscopic image data in real time through an endoscopic camera and a data transmission channel, and performs preliminary denoising and enhancement; the data parallel processing module 502 adopts a parallel processing architecture to divide, perform denoising and enhancement on the endoscopic image data again, improve the image contrast, and perform distortion correction processing on the endoscopic image data in sequence; the dynamic modeling module 503 performs dynamic modeling on the newly acquired image data in the nasal jejunum dynamic three-dimensional model; the adjustment module 504 performs dynamic path planning and attitude adjustment, and detects errors in real time; through the above method, real-time performance, soft tissue adaptability, and catheter positioning information accuracy can be satisfied, thereby improving the reliability of the nasal jejunum catheterization operation.

[0132] Regarding the system in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0133] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present application. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0134] Correspondingly, the present application further provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional reconstruction method for a transnasal jejunal catheterization robot as described above. As Figure 8 shown, it is a hardware structure diagram of any device with data processing capabilities where the three-dimensional reconstruction system for a transnasal jejunal catheterization robot provided by an embodiment of the present invention is located. Except forFigure 8 In addition to the processor, memory, and network interface shown, any device with data processing capabilities where the device in the embodiment is located usually includes other hardware according to the actual functions of the device with data processing capabilities, which will not be elaborated here.

[0135] Correspondingly, the present application also provides a computer-readable storage medium, on which computer instructions are stored. When the instructions are executed by a processor, the three-dimensional reconstruction method for the nasoenteric intubation robot as described above is implemented. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0136] After considering the specification and practicing the content disclosed herein, those skilled in the art will readily think of other implementation schemes of the present application. The present application aims to cover any variations, uses, or adaptive changes of the present application, and these variations, uses, or adaptive changes follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application.

[0137] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A three-dimensional reconstruction method for a nasojejunal catheterization robot, characterized in that: The steps include: Through the endoscope camera and data transmission channel, the endoscope image data is acquired in real time, and preliminary denoising and enhancement are performed; The parallel processing architecture is used to divide the endoscopic image data, perform image denoising and enhancement, improve image contrast, and perform distortion correction processing in sequence. Dynamic modeling of newly acquired image data in the dynamic three-dimensional model of the nose and jejunum; Perform dynamic path planning and posture adjustment, and detect errors in real time.

2. The three-dimensional reconstruction method for a transnasal jejunal catheterization robot according to claim 1, characterized in that: In the step of dynamically modeling the newly acquired image data in the dynamic three-dimensional model of the nose and jejunum: Construct a multi-source knowledge base of the nose and jejunum, and divide the structure of the multi-source knowledge base into static templates, dynamic templates, and personalized templates; Based on the multi-source knowledge base, static and dynamic three-dimensional models of the nose and jejunum are constructed, and the static and dynamic three-dimensional models are coupled to form a dynamic three-dimensional model of the nose and jejunum. The coupling weights of the static and motion models are dynamically adjusted to perform local deformation of the dynamic three-dimensional model of the nose and jejunum.

3. The three-dimensional reconstruction method for a transnasal jejunal catheterization robot according to claim 2, characterized in that: After dynamically adjusting the coupling weights of the static and motion models and performing the local deformation of the dynamic three-dimensional model of the nose and jejunum: The dynamic three-dimensional model of the nose and jejunum was used as prior information to perform real-time three-dimensional reconstruction of the nose and jejunum.

4. The three-dimensional reconstruction method for a transnasal jejunal catheterization robot according to claim 2, characterized in that: In the step of constructing the nasojejunal multi-source knowledge base and dividing the structure of the multi-source knowledge base into static templates, dynamic templates, and personalized templates: The data of the multi-source knowledge base includes static images, ultrasound data, endoscopic videos, anatomical and physiological data, and path planning reference templates.

5. The three-dimensional reconstruction method for a transnasal jejunal catheterization robot according to claim 2, characterized in that: In the step of constructing the nasojejunal multi-source knowledge base and dividing the structure of the multi-source knowledge base into static templates, dynamic templates, and personalized templates: The static template is used to store the basic anatomy of the nasojejunum; Dynamic templates are used to store dynamic motion characteristics; The personalized template is used to store the patient's characteristic information.

6. The three-dimensional reconstruction method for a transnasal jejunal catheterization robot according to claim 1, characterized in that: In the steps of dynamic path planning and posture adjustment, and real-time error detection: Environmental perception and feature extraction based on deep learning; The robot's position and posture are estimated based on multi-sensor data, and the environmental perception and feature extraction are integrated to estimate the robot's position and posture inside the nasojejunum.

7. The three-dimensional reconstruction method for a transnasal jejunal catheterization robot according to claim 6, characterized in that: After estimating the robot's position and posture based on multi-sensor data, integrating environmental perception and feature extraction, and estimating the robot's position and posture inside the nose and jejunum: Compare the current posture with the predicted posture, generate targeted correction information based on real-time sensor data, and feedback the error.

8. A three-dimensional reconstruction system for a transnasal jejunal catheterization robot, applied to the three-dimensional reconstruction method for a transnasal jejunal catheterization robot as claimed in claim 1, characterized in that: It includes a data acquisition module, a data parallel processing module, a dynamic modeling module and an adjustment module; wherein: The data acquisition module is used to acquire endoscopic image data in real time through an endoscopic camera and a data transmission channel, and to perform preliminary denoising and enhancement; The data parallel processing module is used to use a parallel processing architecture to sequentially divide the endoscopic image data, perform image denoising and enhancement, improve image contrast, and perform distortion correction processing; The dynamic modeling module is used to dynamically model the newly acquired image data in the dynamic three-dimensional model of the nose and jejunum; The adjustment module is used to perform dynamic path planning and posture adjustment, and detect errors in real time.

Images