Endoscope image real-time dynamic three-dimensional reconstruction method and system

Abstract

The application provides an endoscope image real-time dynamic three-dimensional reconstruction method and system, comprising the following steps: acquiring initialization monocular endoscope image data, constructing an endoscope image training data set through a Gaussian splash algorithm and a chaos algorithm, constructing a 3D reconstruction constraint network, pre-training the 3D reconstruction constraint network in combination with the endoscope image training data set and a preset physical loss function, acquiring real-time monocular endoscope image data, generating a dynamic 3D Gaussian scene, performing scene rendering on the endoscope dynamic 3D Gaussian scene based on endoscope visual parameters corresponding to the real-time monocular endoscope image data, generating ideal monocular endoscope image data, acquiring a discrimination update parameter group by using a rendering discrimination update mechanism, and performing feedback optimization on the 3D reconstruction constraint network based on the discrimination update parameter group. The application integrates real-time image processing, dynamic scene modeling and adaptive optimization, and realizes the improvement of the accuracy, speed and robustness of endoscope three-dimensional reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular, provides an endoscope image real-time dynamic three-dimensional reconstruction method and system. BACKGROUND

[0002] As a key tool for minimally invasive surgery and medical diagnosis, endoscopy has been widely used in the examination and treatment of diseases in the digestive tract, respiratory tract and other cavities. With the progress of medical imaging technology, endoscopes have developed from traditional two-dimensional images to three-dimensional visualization to provide more spatial information to assist doctors in precise operations. Among them, the real-time dynamic three-dimensional reconstruction method and system of endoscope images have become a research hotspot, aiming to quickly convert continuous endoscope video streams into three-dimensional models to realize intraoperative real-time navigation and dynamic scene segmentation. However, the existing technology still has significant limitations in the real-time performance, dynamic adaptability and reconstruction accuracy of endoscope images.

[0003] On the one hand, existing three-dimensional reconstruction methods are mostly based on static or offline processing modes, which are difficult to effectively cope with dynamic changes in endoscopic scenes. Traditional methods such as structure light or stereo matching-based techniques usually rely on fixed scene assumptions, but in endoscopic applications, due to organ peristalsis, blood occlusion or rapid movement of the lens, image sequences often change dramatically, resulting in broken or distorted reconstruction models. For example, although methods based on sparse feature point matching reduce computational complexity, they are difficult to capture dynamic details such as soft tissue deformation, causing the model to mismatch the real anatomical structure. In addition, these methods often require post-processing manual correction and cannot achieve real-time feedback.

[0004] On the other hand, existing technologies lack sufficient support for real-time processing capabilities, making it difficult to balance computational efficiency and reconstruction quality. Endoscope image data is large and has a high frame rate, while mainstream reconstruction algorithms such as end-to-end models based on deep learning can improve accuracy, but have large model parameter quantities and high inference delays, making it difficult to run in real time on resource-constrained embedded devices. At the same time, existing methods lack robustness to changes in lighting and noise interference, and are prone to reconstruction errors in complex intracavity environments. For example, methods that use simplified point cloud fusion strategies speed up processing, but sacrifice spatial resolution, resulting in blurred edges in the reconstructed model.

[0005] In addition, existing systems often ignore multi-modal data fusion and adaptive optimization in dynamic scenes. In endoscopic operations, preoperative image data and real-time images need to be registered, but existing registration methods are mostly based on rigid assumptions and cannot adapt to the non-rigid deformation of soft tissues, resulting in a disconnection between the three-dimensional model and the real-time image. At the same time, there is a lack of real-time compensation mechanism for motion artifacts and occlusion problems, further reducing the practicality of reconstruction.

[0006] In summary, due to poor dynamic adaptability, insufficient real-time processing capability and limitations of multi-modal fusion, the existing endoscopic image three-dimensional reconstruction technology is difficult to meet the clinical demand for high reliability and low delay. Therefore, there is an urgent need in the field for a comprehensive method that can integrate real-time image processing, dynamic scene modeling and adaptive optimization to improve the accuracy, speed and robustness of endoscopic three-dimensional reconstruction. SUMMARY

[0007] To solve the above technical problems, the present application provides an endoscopic image real-time dynamic three-dimensional reconstruction method and system.

[0008] The first aspect of the present application provides an endoscopic image real-time dynamic three-dimensional reconstruction method, comprising:

[0009] Obtaining initialization monocular endoscopic image data, and constructing an endoscopic image training data set through a Gaussian splash algorithm and a chaos algorithm, wherein the initialization monocular endoscopic image data includes endoscopic multi-modal data and endoscopic visual parameters;

[0010] Building a 3D reconstruction constraint network based on a multi-layer perception mechanism, and pre-training the 3D reconstruction constraint network in combination with the endoscopic image training data set and a preset physical loss function;

[0011] Obtaining real-time monocular endoscopic image data, inputting the 3D reconstruction constraint network for dynamic reconstruction, and generating a dynamic 3D Gaussian scene;

[0012] Rendering the endoscopic dynamic 3D Gaussian scene based on the endoscopic visual parameters corresponding to the real-time monocular endoscopic image data to generate ideal monocular endoscopic image data;

[0013] Using a rendering discrimination update mechanism to discriminate the real-time and ideal monocular endoscopic image data, obtaining a discrimination update parameter group, and feeding back and optimizing the 3D reconstruction constraint network based on the discrimination update parameter group.

[0014] According to a preferred embodiment, the initialization monocular endoscopic image data is obtained, and an endoscopic image training data set is constructed through a Gaussian splash algorithm and a chaos algorithm, wherein the initialization monocular endoscopic image data includes endoscopic multi-modal data and endoscopic visual parameters, comprising:

[0015] Obtaining a monocular endoscopic initialization sequence, and obtaining initialization endoscopic visual parameters based on the monocular endoscopic initialization sequence using an improved VIO algorithm, wherein the initialization endoscopic visual parameters include camera intrinsic parameters and initialization camera poses;

[0016] Based on the monocular endoscope initialization sequence and the initialization endoscope vision parameter, initialization endoscope multi-modal data is generated, including initialization RGB images, initialization depth maps, and initialization semantic segmentation maps;

[0017] Based on the initialization RGB images, a static 3D Gaussian scene is constructed using a Gaussian splatting algorithm, based on the initialization depth maps and the initialization semantic segmentation maps, an endoscope Gaussian density control strategy is defined, and based on the endoscope Gaussian density control strategy, the static 3D Gaussian scene is refined.

[0018] Based on the initialization semantic segmentation maps, a main semantic label and a corresponding confidence are assigned to each Gaussian point in the static 3D Gaussian scene through back projection and a voting mechanism.

[0019] A dynamic disturbance condition vector is generated using a chaotic algorithm, the dynamic disturbance condition vector is introduced into the static 3D Gaussian scene, and a disturbance Gaussian point attribute adjustment amount is obtained through back optimization, and the dynamic disturbance condition vector and the disturbance Gaussian point attribute adjustment amount are corresponded and combined into an endoscope image training dataset.

[0020] According to a preferred embodiment, a 3D reconstruction constraint network is constructed based on a multi-layer perception mechanism, and the 3D reconstruction constraint network is pre-trained in combination with an endoscope image training dataset and a preset physical loss function, including:

[0021] A multi-layer perception mechanism including a condition modulation block is used as the 3D reconstruction constraint network, and the condition modulation block is used to embed the dynamic disturbance condition vector into the intermediate layer features;

[0022] A preset physical loss function is obtained, and the preset physical loss function is specifically a weighted sum of an isometric deformation loss term, a motion smoothness loss term, a periodic motion loss term, and an instrument rigid motion loss term.

[0023] The dynamic disturbance condition vector in the endoscope image training dataset is taken as input, and the corresponding disturbance Gaussian point attribute adjustment amount is taken as a supervision label, and a network prediction value is calculated through forward propagation;

[0024] A predicted training total loss is calculated based on the network prediction value and in combination with the preset physical loss function, and a constraint weight parameter of a linear transformation layer of the 3D reconstruction constraint network is iterated based on the predicted training total loss until the predicted training total loss converges, and the pre-training of the 3D reconstruction constraint network is completed.

[0025] According to a preferred embodiment, a preset physical loss function is obtained, and the preset physical loss function is specifically a weighted sum of an isometric deformation loss term, a motion smoothness loss term, a periodic motion loss term, and an instrument rigid motion loss term, including:

[0026] The equidistance deformation loss term is used to constrain the local area preservation characteristics in the tissue deformation process;

[0027] The motion smoothness loss term is used to constrain the continuity of adjacent Gaussian points on the motion vector;

[0028] The periodic motion loss term is used to constrain the physiological periodic motion prior as a regularization constraint;

[0029] Based on the initialization semantic segmentation map in the endoscopic multi-modal data, an instrument rigid motion loss term is obtained, and the instrument rigid motion loss term is used to constrain the Gaussian point cloud identified as a medical instrument to follow the rigid motion rule.

[0030] According to a preferred embodiment, real-time monocular endoscopic image data is obtained, input into a 3D reconstruction constraint network for dynamic reconstruction, and a dynamic 3D Gaussian scene is generated, including:

[0031] Real-time endoscopic visual parameters and real-time endoscopic multi-modal data are obtained, and the real-time endoscopic visual parameters and the real-time endoscopic multi-modal data are packaged as real-time monocular endoscopic image data;

[0032] Based on the real-time endoscopic visual parameters and through a medical instrument tracking model, instrument state information is obtained, and the instrument state information is encoded as an instrument state vector;

[0033] The real-time semantic segmentation map in the real-time endoscopic multi-modal data is encoded as a semantic feature vector through global statistics, and the real-time RGB image and the real-time depth image in the real-time endoscopic multi-modal data are encoded as a physiological signal vector;

[0034] The instrument state vector, the semantic feature vector, and the physiological signal vector are subjected to time series filtering and standardization, and are packaged as a programming condition vector;

[0035] The programming condition vector is input into a 3D reconstruction constraint network, a real-time Gaussian point attribute adjustment amount is obtained, the real-time Gaussian point attribute adjustment amount is applied to a static 3D Gaussian scene, and a dynamic 3D Gaussian scene is obtained in real time.

[0036] According to a preferred embodiment, the endoscopic dynamic 3D Gaussian scene is rendered based on the endoscopic visual parameters corresponding to the real-time monocular endoscopic image data, and ideal monocular endoscopic image data is generated, including:

[0037] Specifically, a differentiable renderer based on Z-Buffer depth sorting and Alpha blending is used to render the endoscopic dynamic 3D Gaussian scene according to the real-time camera pose into an endoscopic two-dimensional image arranged in time series, and as ideal monocular endoscopic image data.

[0038] According to a preferred embodiment, the residual discrimination is performed on real-time ideal monocular endoscope image data by using a rendering discrimination update mechanism to obtain a discrimination update parameter group, and the 3D reconstruction constraint network is feedback optimized based on the discrimination update parameter group, including:

[0039] The rendering discrimination update mechanism includes a rendering-perception consistency verification module and a dynamic scene quality evaluation module, the perception parameter correction signal is generated based on the rendering-perception consistency verification module, the model optimization signal is generated based on the dynamic scene quality evaluation module, and the perception parameter correction signal and the model optimization signal are combined into the discrimination update parameter group;

[0040] The medical instrument tracking model parameters, the global statistical parameters, and the frequency domain analysis and signal extraction parameters are iteratively updated based on the perception parameter correction signal;

[0041] The weight parameters of the linear transformation layer of the 3D reconstruction constraint network are iteratively updated based on the model optimization signal by using a back propagation algorithm.

[0042] According to a preferred embodiment, the rendering discrimination update mechanism includes a rendering-perception consistency verification module and a dynamic scene quality evaluation module, including:

[0043] The rendering-perception consistency verification module is configured to obtain the reprojection error and the optical flow difference between the ideal monocular endoscope image data and the real-time monocular endoscope image data by using a multi-scale feature matching and semantic consistency evaluation algorithm, and to generate a perception parameter correction signal;

[0044] The dynamic scene quality evaluation module is configured to obtain the time series stability and the geometric rationality of the ideal monocular endoscope image data by using a time series filtering anomaly detection and surface normal consistency verification algorithm based on a dynamic convolution kernel, and to quantify the time series stability and the geometric rationality into a model optimization signal.

[0045] According to a preferred embodiment, the medical instrument tracking model parameters, the global statistical parameters, and the frequency domain analysis and signal extraction parameters are iteratively updated based on the perception parameter correction signal, including:

[0046] The confidence weight arranged in time series is obtained based on the model optimization signal, and the original loss of each time series frame is calculated based on the ideal monocular endoscope image data and the real-time monocular endoscope image data;

[0047] The global rendering quality loss is obtained based on the confidence weight of each time series frame, and the global rendering quality loss gradient is obtained by using a back propagation algorithm;

[0048] The historical high-confidence frames are obtained by a screening operation, and the dominant direction is obtained by using principal component analysis based on the original loss of the historical high-confidence frames, and is used as the network main feature direction.

[0049] The global rendering quality loss gradient is projected to the network main feature direction to obtain an updated feature direction, and a preset safety threshold is used to constrain the feature direction angle between the updated feature direction and the network main feature direction.

[0050] According to the feature direction angle, the 3D reconstruction constraint network is updated and adjusted based on historical high-confidence frames or based on the global rendering quality loss gradient.

[0051] The second aspect of the present application also provides an endoscopic image real-time dynamic three-dimensional reconstruction system, comprising:

[0052] A data acquisition preprocessing module is configured to acquire initial monocular endoscopic image data and construct an endoscopic image training dataset by using a Gaussian splash algorithm and a chaos algorithm.

[0053] A 3D reconstruction constraint network module is configured to construct a 3D reconstruction constraint network based on a multi-layer perception mechanism, and pre-train the 3D reconstruction constraint network in combination with the endoscopic image training dataset and a preset physical loss function.

[0054] A dynamic scene generation and rendering module is configured to acquire real-time monocular endoscopic image data, input the 3D reconstruction constraint network for dynamic reconstruction, generate a dynamic 3D Gaussian scene, and perform scene rendering on the endoscopic dynamic 3D Gaussian scene based on the corresponding endoscopic visual parameters of the real-time monocular endoscopic image data to generate ideal monocular endoscopic image data.

[0055] A rendering discrimination updating module is configured to use a rendering discrimination updating mechanism to perform residual discrimination on real-time and ideal monocular endoscopic image data, acquire a discrimination updating parameter group, and perform feedback optimization on the 3D reconstruction constraint network based on the discrimination updating parameter group.

[0056] Based on the above aspects, the embodiment of the application obtains the initialization monocular endoscope image data, and constructs the endoscope image training data set by using the Gaussian splash algorithm and the chaos algorithm, constructs the 3D reconstruction constraint network based on the multilayer perception mechanism, and pre-trains by combining the physical loss function, inputs the real-time monocular endoscope image data into the network for dynamic reconstruction to generate a dynamic 3D Gaussian scene, renders the dynamic scene based on the endoscope visual parameters to generate ideal monocular endoscope image data, and uses the rendering discrimination update mechanism for residual error discrimination and feedback optimization. The above steps first construct a static 3D Gaussian scene by using the Gaussian splash algorithm, and generate a dynamic disturbance condition vector by using the chaos algorithm, simulate the pseudo-random characteristics of the dynamic changes such as tissue peristalsis and instrument movement in the endoscope scene, provide a real and diversified dynamic prior data basis for training, fundamentally overcome the limitations that the traditional method cannot capture the details of soft tissue deformation based on static assumptions or sparse feature matching, and solve the problem of poor dynamic adaptability.

[0057] On the other hand, by constructing a multilayer perception mechanism containing a condition modulation block as a 3D reconstruction constraint network, and introducing a physical loss function weighted by an isometric deformation loss, a motion smoothness loss, a periodic motion loss and an instrument rigid motion loss, the physical laws such as tissue deformation preservation, motion continuity, physiological cycle prior and instrument rigidity constraint are embedded in the pre-training, the physical rationality of dynamic reconstruction is ensured, the robustness of the model in the texture missing or occlusion scene is improved, and further, in the real-time reconstruction stage, the instrument state vector, the semantic feature vector and the physiological signal vector are packaged as a programming condition vector to drive the network to predict the Gaussian point attribute adjustment amount, realize the real-time conversion from a static scene to a dynamic scene, and combine the differentiable renderer to quickly generate ideal image data, effectively balance the calculation efficiency and reconstruction accuracy, and overcome the defects of high delay and large resource consumption of the existing deep learning model.

[0058] In addition, through the rendering-perception consistency verification module and the dynamic scene quality evaluation module in the rendering discrimination update mechanism, multi-scale feature matching, optical flow difference analysis and time sequence stability detection are performed on the real-time and ideal image data to generate perception parameter correction signals and model optimization signals, and based on this, the medical instrument tracking model, the multi-modal statistical parameters and the network weights are iteratively updated, realizing adaptive optimization and closed-loop feedback of the reconstruction process, and solving the problems of insufficient multi-modal data fusion and motion artifact compensation. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 An execution flowchart of the endoscope image real-time dynamic three-dimensional reconstruction method of the application is shown.

[0060] Figure 2 An acquisition flowchart of ideal monocular endoscope image data in the endoscope image real-time dynamic three-dimensional reconstruction method of the application is shown.

[0061] Figure 3 A schematic view of an endoscope image real-time dynamic three-dimensional reconstruction system is shown. DETAILED DESCRIPTION

[0062] In order to enable a more clear understanding of the above-mentioned objects, features and advantages of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0063] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0064] As shown in Figure 1 , Figure 2 , Figure 3 ,

[0065] The first aspect of the present application provides an endoscope image real-time dynamic three-dimensional reconstruction method, comprising:

[0066] Step S1: obtaining initialization monocular endoscope image data, and constructing an endoscope image training data set through a Gaussian splash algorithm and a chaos algorithm, wherein the initialization monocular endoscope image data comprises endoscope multi-modal data and endoscope visual parameters.

[0067] In step S1, the following steps are included:

[0068] Step S11: obtaining a monocular endoscope initialization sequence, obtaining camera intrinsic parameters based on the monocular endoscope initialization sequence by using an improved VIO algorithm, and encapsulating the camera intrinsic parameters and the initialization camera pose as initialization endoscope visual parameters.

[0069] Specifically, initialization sequence images are collected by a monocular endoscope device such as a digestive tract endoscope or a bronchial endoscope, and are used as a monocular endoscope initialization sequence. The monocular endoscope device needs to be equipped with a high frame rate camera such as a global shutter CMOS sensor to ensure that clear and continuous images are captured when tissue movement or lens movement occurs. The monocular endoscope initialization sequence needs to include continuous frame images of the detection site. For example, assuming that a digestive tract endoscope is used for medical examination, the obtained monocular endoscope initialization sequence needs to include continuous frame images of the esophagus, stomach and other parts. The frame rate needs to meet the Nyquist sampling theorem, i.e. higher than 2 times the main frequency of tissue movement. Since the tissue movement frequency in the endoscope application scenario is usually lower than 10 Hz, the sampling frame rate of the monocular endoscope initialization sequence can be set to 30 fps to balance the detail capture and calculation efficiency.

[0070] Further, the camera intrinsic parameters are obtained by offline calibration, and the camera intrinsic parameters at least include a camera focal length, a principal point coordinate, and a distortion coefficient, and the like. For example, assuming that for the digestive tract endoscope A, the Zhang Zhengyou calibration method is adopted, and the camera focal length is obtained as fx=500 pixels, fy=500 pixels, the principal point coordinate is obtained as cx=320 pixels, cy=240 pixels, and the distortion coefficient is obtained as the radial distortion k1, k2 and the tangential distortion p1, p2.

[0071] It can be understood that the improved VIO algorithm introduces photometric consistency constraints and semantic feature tracking on the basis of the traditional VIO algorithm to improve the robustness in the case of texture missing or dynamic scene. For example, assuming that the digestive tract endoscope A performs a gastroscopy, the improved VIO algorithm preferentially tracks stable features such as gastric wall folds, and compensates for drift caused by rapid camera motion by using IMU data.

[0072] Further, the camera optical center of the first frame in the monocular endoscope initialization sequence can be set as the origin, the direction of the camera optical axis of the scene can be set as the Z axis, a reference coordinate system can be set, the initialization RGB image can be extracted based on the monocular endoscope initialization sequence, the ORB feature points can be extracted from the initialization RGB image as image feature points, the IMU data can be collected in real time by the integrated inertial measurement unit of the endoscope device, the IMU data includes angular velocity and acceleration, photometric consistency constraints and semantic feature tracking are introduced, the photometric consistency constraints are obtained by calculating the photometric error of the image feature points in the fixed neighborhood window, such as 5*5 pixels, between adjacent frames, and a re-projection error function is constructed to optimize the camera pose, the semantic feature tracking preferentially selects stable image feature points such as gastric wall folds and blood vessel bifurcations for tracking, and avoids tracking failure caused by tissue movement.

[0073] It can be understood that based on the image feature points and the IMU data, a tightly coupled nonlinear optimization method based on key frames can be adopted to establish a common optimization target of visual re-projection error items and IMU pre-integration error items within a sliding window, to balance the contribution degree of the data of the two sensors through a weighted fusion manner, to solve the camera pose minimizing the comprehensive error by using an iterative optimization algorithm, and to take the 6-degree-of-freedom pose of each frame in the reference coordinate system. For example, assuming that in the gastroscopy scene, the monocular endoscope initialization sequence contains 300 continuous images with a time length of 10 seconds, the improved VIO algorithm first obtains a sliding window containing 5 key frames through the rule of selecting a key frame every 5 frames, and then optimizes the sliding window. Within the sliding window, the re-projection error of the visual feature points and the pre-integration error of the IMU are jointly optimized, wherein the semantic feature points are given a weight of 1.5 times, the weight of the luminosity consistency constraint is increased for the texture missing area such as the gastric juice covered area, the stability of the pose estimation is ensured, the 6-degree-of-freedom pose of each frame of the camera in the reference coordinate system is obtained through iterative optimization, all the 6-degree-of-freedom poses are arranged and combined in time sequence as the initialization camera pose, and the initialization camera pose and the camera intrinsic parameter are encapsulated as the initialization endoscope visual parameter.

[0074] Step S12: generating initialization endoscope multi-modal data based on the monocular endoscope initialization sequence and the initialization endoscope visual parameter, the initialization endoscope multi-modal data including initialization RGB images, initialization depth maps, and initialization semantic segmentation maps.

[0075] Specifically, the initialization RGB image is extracted based on the monocular endoscope initialization sequence, the initialization RGB image can be denoised using a non-local mean filtering algorithm, and color correction is performed based on white balance color offset correction, and based on the initialization RGB image and the initialization camera pose, a photometric consistency depth estimation algorithm such as Monodepth2 algorithm is used to obtain the initialization depth map. For example, assuming that the Monodepth2 algorithm is used, the depth value of each pixel is predicted by a convolutional neural network using sequence multi-view geometry constraints. The depth map resolution is consistent with the initialization RGB image, such as 640x480 pixels, and the depth range is set according to the working distance of the endoscope, for example, 50mm to 150mm. A semantic segmentation network such as U-Net, DeepLabv3+ pre-trained on medical image datasets such as Kvasir-SEG is used to perform pixel-level semantic segmentation on the initialization RGB image, and an initialization semantic segmentation map containing tissue types and medical devices is output. The tissue types are, for example, mucosa, blood vessels, etc., and the medical devices are, for example, forceps, catheters, etc. The initialization semantic segmentation map provides a pixel class label and a corresponding pixel confidence for each pixel. The initialization depth map and the initialization semantic segmentation map are both projected into the reference coordinate system according to the initialization camera pose, aligning the initialization endoscope multi-modal data with the initialization endoscope visual parameters and ensuring spatial consistency.

[0076] Step S13: Based on the initialization RGB image, a static 3D Gaussian scene is constructed using a Gaussian splatting algorithm, an endoscope Gaussian density control strategy is defined based on the initialization depth map and the initialization semantic segmentation map, and the static 3D Gaussian scene is refined based on the endoscope Gaussian density control strategy.

[0077] Specifically, the SIFT algorithm can be used to extract the initialization image feature points of the initialization RGB image, and the initialization image feature points are back-projected to three-dimensional space to obtain an initialization Gaussian point set. The Gaussian point parameters are optimized by a stochastic gradient descent algorithm, and the Gaussian point parameters include at least position, covariance, RGB color value and opacity. The optimization target is to minimize the photometric error between the two-dimensional rendered image and the real image, and the number of iterations is set to 1000 times. Based on the optimized Gaussian point set, an anisotropic Gaussian distribution is constructed, and a static 3D Gaussian scene is formed.

[0078] Further, based on the initialized depth map, the density of Gaussian points is increased in the near-range area to enhance details, and the density of Gaussian points is thinned in the far-range area to reduce the computational load, the near-range area is specifically an area with small depth values, and the far-range area is specifically an area with large depth values, for example, a depth threshold can be set, the area with a depth less than 80 mm is set to 10 points per cubic millimeter, and the area with a depth greater than or equal to 80 mm is set to 5 points per cubic millimeter, the pixel class labels in the initialized semantic segmentation map are used to identify key areas, and the density enhancement amplitude is weighted based on the pixel confidence of the key areas, the key areas are lesion tissues or instrument contact points, etc., for example, for the key area identified as a lesion tissue, the basic density is increased by 50%, if the average pixel-level confidence of the key area is higher than 0.8, it is additionally increased by 20%, and if the average pixel-level confidence is lower than 0.6, it is only increased by 30%, to balance the enhancement effect and the risk brought by segmentation uncertainty, the static 3D Gaussian scene is iteratively optimized based on the initialized depth map and the initialized semantic segmentation map, the rendering quality is re-evaluated after each optimization, and the static 3D Gaussian scene is iteratively optimized until the static rendering refinement convergence criterion is met, and the static rendering refinement convergence criterion can be set to a peak signal-to-noise ratio (PSNR) of the static 3D Gaussian scene greater than 30 dB.

[0079] Step S14: Based on the initialized semantic segmentation map, a main semantic label and a corresponding confidence are assigned to each Gaussian point in the static 3D Gaussian scene through back projection and voting mechanism.

[0080] Specifically, each Gaussian point in the static 3D Gaussian scene is projected onto the initialized semantic segmentation map to obtain the semantic projection position and the corresponding semantic label of each Gaussian point in each frame of image, for each Gaussian point, the semantic projection position and the corresponding semantic label in different frames are counted, the majority voting method is used to determine the main semantic label, for example, the semantic label with the highest frequency is taken as the main semantic label, and the confidence is calculated as a measure of voting consistency, for example, the confidence is defined as the proportion of the highest number of votes to the total number of votes, if the confidence is lower than a voting threshold, the voting threshold can be set to 0.6, the Gaussian point is marked as an unknown class, for example, assuming that a Gaussian point is identified as mucosa 8 times in 10 projection frames, the main semantic label of the Gaussian point is mucosa, and the confidence is 0.8.

[0081] Step S15: A dynamic disturbance condition vector is generated by using a chaotic algorithm, the dynamic disturbance condition vector is introduced into the static 3D Gaussian scene, the attribute adjustment amount of the disturbed Gaussian point is obtained by reverse optimization, and the dynamic disturbance condition vector and the attribute adjustment amount of the disturbed Gaussian point are correspondingly combined to form an endoscopic image training data set.

[0082] It can be understood that in the endoscope use scenario, there are dynamic changes such as tissue movement and instrument movement, which have quasi-periodicity and uncertainty. The chaotic algorithm can effectively simulate the pseudo-random characteristics of such motion, and provide real dynamic prior data for the training data.

[0083] Specifically, the Logistic mapping can be used as a chaotic source to generate a one-dimensional chaotic sequence through iteration, and the one-dimensional chaotic sequence can be encoded as a multi-dimensional vector as a dynamic disturbance condition vector. The dynamic disturbance condition vector simulates the pseudo-random characteristics of physiological movements such as breathing and heartbeat. The dynamic disturbance condition vector is used as input, and a lightweight neural network such as a 3-layer fully connected network is used to predict the adjustment amount of each Gaussian point in position, rotation, scale, etc. The training set optimization target is to minimize the difference between the two-dimensional rendering image of the disturbed scene and the expected dynamic image. The training set loss function includes training set luminosity loss and motion smoothness and other physical constraints. Based on the training set optimization target and the training set loss function, the Adam optimizer is used for reverse optimization. Finally, the dynamic disturbance condition vector after reverse optimization is paired with the corresponding Gaussian point attribute adjustment amount. The dynamic disturbance condition vector is used as the input vector, and the corresponding Gaussian point attribute adjustment amount is used as the adjustment amount label. Based on all input vectors and their corresponding adjustment amount labels, an endoscope image training data set is formed.

[0084] Step S2: Building a 3D reconstruction constraint network based on a multi-layer perception mechanism, and pre-training the 3D reconstruction constraint network in combination with the endoscope image training data set and a preset physical loss function.

[0085] In step S2, the following steps are included:

[0086] Step S21: A multi-layer perception mechanism including a condition modulation block is used as the 3D reconstruction constraint network, and the condition modulation block is used to embed the dynamic disturbance condition vector into the intermediate layer features.

[0087] It should be noted that the multi-layer perception mechanism is used as the main structure of the 3D reconstruction constraint network. If the number of fully connected layers of the multi-layer perception mechanism is too small, the network capacity will be insufficient, and it will be difficult to learn complex dynamic mapping relationships. If the number of fully connected layers is too large, the training difficulty will be increased and overfitting will be easily caused. Therefore, the number of fully connected layers can be set to 8. The dimension of the input layer in the 3D reconstruction constraint network is set to be consistent with the dynamic disturbance condition vector. The dimension of the output layer is set according to the dimension of the Gaussian point attribute adjustment amount. For example, the Gaussian point attribute adjustment amount includes a 3-dimensional position adjustment amount, a 4-dimensional quaternion rotation adjustment amount, a 3-dimensional size adjustment amount, and a 3-dimensional RGB color adjustment amount. Therefore, the dimension of the output layer is 13.

[0088] It can be understood that the 3D reconstruction constraint network inserts a condition modulation block in the middle layer, allowing the 3D reconstruction constraint network to dynamically adjust the extracted feature vector according to the dynamic disturbance condition vector. For example, during the training process, a plurality of dynamic disturbance condition vectors are generated by a chaotic algorithm, each of which uniquely corresponds to a possible dynamic scene, such as large amplitude peristalsis. When the dynamic disturbance condition vector input into the 3D reconstruction constraint network indicates large amplitude peristalsis, the modulation block can amplify the feature channels related to deformation.

[0089] In some possible embodiments, assuming that the dynamic disturbance condition vector is 64-dimensional, the 3D reconstruction constraint network is composed of an input layer, 8 fully connected layers [FC1, FC2, FC3, …, FC8] and an output layer, a condition modulation block A is inserted between FC3 and FC4, and a condition modulation block B is inserted between FC5 and FC6. The dynamic disturbance condition vector is input into the input layer as a feature vector, FC1 and FC2 can use the ReLU activation function to raise the 64-dimensional feature vector to a 128-dimensional feature vector and then to a 256-dimensional feature vector, thereby obtaining feature information from low-dimensional feature vectors and realizing encoding and feature expansion of the feature vector. FC3 keeps the feature vector dimension unchanged and consolidates the feature vector. Then, the modulation block A conditionally modulates the feature vector output by FC3 according to the original feature vector of the input layer. FC4 maps the feature vector to the highest dimension of 512 dimensions to construct the most abstract and global feature representation, so as to encode complex dynamic relationships such as the spatiotemporal continuity of tissue deformation. FC5 keeps the feature vector dimension again, allowing the 3D reconstruction constraint network to perform sufficient calculation on the highest dimension feature vector. Then, the modulation block B conditionally modulates the feature vector output by FC5 according to the original feature vector of the input layer. FC6 and FC7 reduce the 512-dimensional feature vector to a 256-dimensional feature vector and then to a 128-dimensional feature vector, thereby extracting a low-dimensional feature vector more relevant to the final task from the highest dimension feature vector containing rich information but having redundancy. FC8 maps the 128-dimensional feature vector to specific 13-dimensional control instructions and outputs the network prediction value, i.e., the position adjustment amount, the quaternion rotation adjustment amount, the size adjustment amount and the RGB color adjustment amount, via the output layer.

[0090] Step S22: obtaining a preset physical loss function, and the preset physical loss function is specifically a weighted sum of an isometric deformation loss term, a motion smoothness loss term, a periodic motion loss term and an instrument rigid motion loss term.

[0091] Specifically, the preset physical loss function is composed of four loss terms with clear physical meaning and is obtained by weighted summation, which is used to introduce physical prior constraints in the training process to ensure that the network prediction result conforms to the physical laws of the real world and the clinical scene.

[0092] The isometric deformation loss term is used to constrain the local area preservation characteristics in the tissue deformation process. The isometric deformation loss term can be obtained by calculating the square sum of the area changes of the triangular facets formed by adjacent Gaussian points in a static 3D Gaussian scene before and after deformation. For example, for a triangular facet formed by three Gaussian points, the area change rate before and after deformation is calculated, and the square sum of the area change rates of all triangular facets is taken as the isometric deformation loss value.

[0093] The motion smoothness loss term is used to constrain the continuity of adjacent Gaussian points in the motion vector. The motion smoothness loss term can be obtained by calculating the L2 norm of the motion vector difference between each Gaussian point and its k nearest neighbors. K can be set to 8. This loss term ensures that spatially adjacent Gaussian points have similar motion trends and avoids unreasonable motion discontinuities.

[0094] The periodic motion loss term is used to constrain the physiological periodic motion prior as a regularization constraint. The periodic motion loss term can be obtained by performing a fast Fourier transform on the motion trajectories of the Gaussian points in consecutive time series frames, extracting the dominant frequency components of the motion signal, and calculating the square sum of the differences between the dominant frequency components and the expected physiological frequencies, such as the respiratory frequency 0.1-0.3 Hz and the heartbeat frequency 1-2 Hz.

[0095] The instrument rigid motion loss term is obtained based on the initialization semantic segmentation of the endoscopic multi-modal data. The instrument rigid motion loss term is used to constrain the Gaussian point cloud identified as a medical instrument to follow the rigid motion law. The instrument rigid motion loss term can be obtained by calculating the square sum of the relative distance change rates of the Gaussian point cloud corresponding to the medical instrument before and after deformation.

[0096] Further, the weight distribution of the isometric deformation loss term, the motion smoothness loss term, the periodic motion loss term, and the instrument rigid motion loss term can be set based on the importance in the clinical scene. For example, the isometric deformation loss term weight can be set to 0.4, the motion smoothness loss term weight can be set to 0.3, the periodic motion loss term weight can be set to 0.2, and the instrument rigid motion loss term weight can be set to 0.1. This weight distribution ensures the dominance of the tissue deformation constraint while taking into account other physical constraints.

[0097] Step S23: The dynamic disturbance condition vector in the endoscopic image training dataset is input, and the corresponding disturbance Gaussian point attribute adjustment amount is taken as the supervision label. The network prediction value is calculated by forward propagation.

[0098] Specifically, the endoscope image training dataset is randomly divided into a training set and a validation set in a ratio of 8:2, wherein the training set is used for actual updating and optimization of the 3D reconstruction constraint network parameters, and the validation set is used for monitoring the training process and evaluating the model generalization capability. The 3D reconstruction constraint network is trained based on the training set, each training batch can contain 128 samples, each sample includes a 64-dimensional dynamic disturbance condition vector as the network input, and a corresponding 13-dimensional disturbance Gaussian point attribute adjustment amount as the supervision label.

[0099] Further, the forward propagation process is performed according to the 3D reconstruction constraint network structure described in step S21. The dynamic disturbance condition vector is input into the 3D reconstruction constraint network, and the network prediction value is output. In the training process, a stochastic gradient descent strategy is adopted. After completing a batch of training, the 3D reconstruction constraint network parameters are updated. In order to ensure the stability of the training, the dynamic disturbance condition vector is standardized, so that the mean value of each dimension is 0 and the standard deviation is 1.

[0100] Step S24: Based on the network prediction value and combined with the preset physical loss function, the predicted training total loss is calculated. The constraint weight parameters of the linear transformation layer of the 3D reconstruction constraint network are iterated based on the predicted training total loss until the predicted training total loss converges, and the pre-training of the 3D reconstruction constraint network is completed.

[0101] Specifically, the mean square error of the network prediction value and the corresponding adjustment amount label is taken as the supervision loss. The physical constraint loss is obtained based on the preset physical loss function. The weighted sum of the supervision loss and the physical constraint loss is taken as the predicted training total loss. The predicted training total loss can be set as supervision loss + 0.5 x physical constraint loss. In the training process, a learning rate decay strategy can be adopted, for example, the learning rate is multiplied by a decay coefficient of 0.8 every 50 training batches.

[0102] Further, based on the predicted training total loss, the Adam optimizer can be used to iteratively update the weight parameters of all the fully connected layers in the 3D reconstruction constraint network. After the end of each training batch, the gradient of the predicted training total loss with respect to the weight parameters of all the fully connected layers is calculated. Then, the Adam optimizer dynamically calculates the update amount corresponding to each fully connected layer weight parameter based on the gradient direction and historical gradient information, i.e. the first moment and the second moment, and performs the update. After the end of each training iteration period, the current 3D reconstruction constraint network is forward propagated using the validation set to obtain the validation set loss, which is used to monitor the generalization performance of the model on unseen data. The above steps are repeatedly implemented until the predicted training total loss converges, so that the weight parameters of all the fully connected layers in the 3D reconstruction constraint network are adjusted in the direction of smaller loss.

[0103] It should be noted that when the verification set loss decreases by less than the preset verification threshold, such as 0.1%, for 20 consecutive training batches, it is determined to be converged, and the training is immediately terminated to prevent overfitting. If the total number of training batches reaches 500 and the verification threshold condition is not triggered, the training is forcibly terminated. After the training is completed, the network weight parameters that minimize the verification set loss are saved, and the pre-training of the 3D reconstruction constraint network is completed.

[0104] Step S3: acquiring real-time monocular endoscope image data, inputting the 3D reconstruction constraint network for dynamic reconstruction, and generating a dynamic 3D Gaussian scene.

[0105] In step S3, the following steps are included:

[0106] Step S31: acquiring real-time endoscope visual parameters and real-time endoscope multi-modal data, and encapsulating the real-time endoscope visual parameters and the real-time endoscope multi-modal data as real-time monocular endoscope image data.

[0107] Specifically, the same data acquisition and processing process as in step S11 is used to collect real-time sequence images through a monocular endoscope device, and the real-time sequence images are used as monocular endoscope real-time sequences. An improved VIO algorithm is used to obtain the real-time camera pose of each frame, and the pre-calibrated camera intrinsic parameters are combined to form real-time endoscope visual parameters.

[0108] Further, real-time RGB images are extracted from the monocular endoscope real-time sequences, and are processed in combination with the real-time camera pose to obtain real-time depth maps. Pixel-level semantic segmentation is performed on the real-time RGB images to obtain real-time semantic segmentation maps. The real-time RGB images, the real-time depth maps, and the real-time semantic segmentation maps are encapsulated as real-time endoscope multi-modal data. The real-time endoscope visual parameters and the real-time endoscope multi-modal data are aligned and encapsulated according to the time stamp to form real-time monocular endoscope image data.

[0109] Step S32: obtaining instrument state information based on the real-time endoscope visual parameters and through a medical instrument tracking model, and encoding the instrument state information into an instrument state vector.

[0110] It should be noted that the medical instrument tracking model is used to detect and track medical instruments in an endoscope scene, and can be constructed based on an improved YOLOv7 framework. For example, in view of the characteristics of the medical instruments in the endoscope scene, such as variable shapes and frequent occlusions, an angle regression branch can be added to the detection head to directly output the instrument attitude angle. A time sequence tracking module can be introduced to predict the instrument motion trajectory based on a Kalman filter, thereby improving the tracking stability.

[0111] Specifically, the medical instrument tracking model receives real-time RGB images and real-time semantic segmentation maps as inputs, and outputs spatial position, attitude information and motion state of the medical instrument. The spatial position includes two-dimensional coordinates and depth value of the tip of the medical instrument in the reference coordinate system. The depth value is obtained based on the real-time depth map through interpolation methods such as bilinear interpolation. For example, the depth value can be obtained by taking the depth values of the surrounding four pixels as the center of the image coordinates of the tip of the medical instrument and performing weighted average. The attitude information includes three direction angles of the instrument relative to the reference coordinate system. The motion state includes the motion velocity and acceleration components of the medical instrument.

[0112] Further, the spatial position, attitude information and motion state of the medical instrument are encoded into a 13-dimensional instrument state vector, which specifically includes three-dimensional spatial coordinates (x, y, z), three direction angles (θx, θy, θz), three velocity components (vx, vy, vz) and three acceleration components (ax, ay, az). For example, when the tip of the forceps instrument detects two-dimensional image coordinates (320, 240), the depth value 150 mm is obtained based on the real-time depth map through bilinear interpolation, the attitude angle is (10°, 5°, 0°), the motion velocity is (2 mm / s, 1 mm / s, 0.5 mm / s), and the acceleration is (0.1 mm / s², 0.05 mm / s², 0.02 mm / s²), then the instrument state vector is encoded as [320, 240, 150, 10, 5, 0, 2, 1, 0.5, 0.1, 0.05, 0.02].

[0113] In step S33, the real-time semantic segmentation map in the real-time endoscope multi-modal data is encoded into a semantic feature vector through global statistics, and the real-time RGB image and the real-time depth map in the real-time endoscope multi-modal data are encoded into a physiological signal vector.

[0114] Specifically, the real-time semantic segmentation map is subjected to connected component analysis, and the pixel number proportion of each class, such as mucosa, blood vessels, lesion tissue and medical instrument, is counted. These classes are the main anatomical structures. The spatial distribution characteristics of the main anatomical structures, such as the centroid position and distribution range, are calculated, and finally a 16-dimensional real-time semantic feature vector is generated.

[0115] For example, assuming that in the gastroscope image, the mucosa region accounts for 60%, the blood vessels account for 15%, the instrument accounts for 5%, and the other tissues account for 20%. The centroid positions and distribution variances of the regions are calculated, such as the mucosa centroid being located at the image center (0.5, 0.5) and the blood vessel distribution variance being 0.1. These features are normalized and encoded, and a 16-dimensional real-time semantic feature vector [0.6, 0.15, 0.05, 0.2, 0.5, 0.5, 0.1, …] is output. The real-time semantic feature vector completely represents the semantic composition and spatial distribution characteristics of the current scene.

[0116] Further, the photometric variation signal of the tissue surface is extracted from the real-time RGB image, the physiological rhythm is extracted by 0.1-2Hz band-pass filtering, the deformation frequency feature of the tissue surface is calculated from the real-time depth image, and finally encoded into an 8-dimensional physiological signal vector.

[0117] For example, assuming in the gastroscope scene, by analyzing the photometric variation of the stomach wall surface, the respiratory rhythm signal with a main frequency of 0.2Hz and an amplitude of 0.3 is extracted, the heartbeat rhythm signal with a main frequency of 1.5Hz and an amplitude of 0.1 is extracted, and the main frequency of 0.25Hz and the amplitude of 0.2 of the tissue deformation are extracted from the depth variation, these features are encoded into an 8-dimensional physiological signal vector [0.2, 0.3, 1.5, 0.1, 0.25, 0.2, …], which accurately represents the physiological motion features.

[0118] Step S34, the instrument state vector, the semantic feature vector and the physiological signal vector are time series filtered and standardized, and packaged into a programming condition vector.

[0119] Specifically, Kalman filtering can be used to time series smooth the instrument state vector to eliminate detection jitter, sliding window mean filtering can be used for the semantic feature vector and the physiological signal vector, and the window size is set to five frames, z-score can be used to standardize the instrument state vector, the semantic feature vector and the physiological signal vector after time series filtering, and the standardized instrument state vector, the semantic feature vector and the physiological signal vector are spliced into a programming condition vector. The dimension of the programming condition vector is consistent with the dimension of the dynamic disturbance condition vector used in the training stage, so as to ensure the compatibility with the pre-trained 3D reconstruction constraint network.

[0120] Step S35, input the programming condition vector into the 3D reconstruction constraint network to obtain a real-time Gaussian point attribute adjustment amount, apply the real-time Gaussian point attribute adjustment amount to the static 3D Gaussian scene to obtain a dynamic 3D Gaussian scene in real time.

[0121] Specifically, the programming condition vector is input into the pre-trained 3D reconstruction constraint network for forward propagation, and a 13-dimensional real-time Gaussian point attribute adjustment amount is output. The application of the real-time Gaussian point attribute adjustment amount includes position adjustment, deformation adjustment and appearance adjustment. The position adjustment is specifically updating the spatial coordinates of the Gaussian point according to the three-dimensional position adjustment amount, the deformation adjustment is specifically updating the shape parameters of the Gaussian point according to the four-dimensional rotation adjustment amount and the three-dimensional scale adjustment amount, and the appearance adjustment is specifically updating the color attribute of the Gaussian point according to the three-dimensional color adjustment amount.

[0122] Adjust all Gaussian points in the static 3D Gaussian scene based on the real-time Gaussian point attribute adjustment amount to obtain a dynamic 3D Gaussian scene. For example, assuming in a gastroscope scene, the static 3D Gaussian scene represents the initial static state of the stomach anatomy, which is manifested as a flat stomach wall and fixed mucosal fold morphology, and the dynamic 3D Gaussian scene simulates the real-time deformation process of the stomach under the action of respiration and peristalsis, the stomach wall presents a periodic fluctuation motion with an amplitude of 2-3 mm and a frequency of 0.2 Hz, the mucosal folds change in waveform with peristaltic waves, the instrument contact area produces a local concave deformation with a depth of 1-2 mm, and the tissue color changes slightly in brightness with blood flow, and these dynamic characteristics are presented by applying the real-time Gaussian point attribute adjustment amount.

[0123] In step S4, the endoscope dynamic 3D Gaussian scene is rendered based on the endoscope visual parameters corresponding to the real-time monocular endoscope image data to generate ideal monocular endoscope image data.

[0124] It can be understood that the real-time monocular endoscope image data represents a two-dimensional video sequence collected by a monocular endoscope device, which contains various noises and disturbances. The real-time monocular endoscope image data reflects the real scene with noises and disturbances. The ideal monocular endoscope image data represents a two-dimensional video sequence generated by a differentiable renderer based on the dynamic 3D Gaussian scene and physical constraints of the present application, which is consistent with the perspective of the real-time data. The ideal monocular endoscope image data is obtained based on a 3D reconstruction constraint network and follows the preset physical laws such as smooth deformation and instrument rigidity, and excludes noises and disturbances in the real world. The ideal monocular endoscope image data provides a benchmark for evaluating and optimizing the quality of three-dimensional reconstruction. By comparing the ideal monocular endoscope image data with the real-time monocular endoscope image data, the error of the current dynamic 3D Gaussian scene can be quantified.

[0125] It should be noted that the differentiable renderer based on Z-Buffer depth sorting and Alpha blending renders the endoscope dynamic 3D Gaussian scene according to the real-time camera pose to obtain a time-sequenced endoscope two-dimensional image, which is used as the ideal monocular endoscope image data.

[0126] Specifically, based on the real-time camera pose and camera intrinsic parameters, each Gaussian point in the dynamic 3D Gaussian scene obtained in step S35 is projected onto a two-dimensional image plane. Z-Buffer depth sorting is used to sort the multiple Gaussian points overlapping on the same pixel from far to near according to their depth values. Alpha blending is used to sequentially mix the RGB color values of each Gaussian point according to its opacity. Finally, the final color of the pixel is calculated. The above process simulates the occlusion and semi-transparency of light, ensuring the geometric rationality of the two-dimensional rendered image.

[0127] Further, for each real-time camera pose at each time, the above process is repeated to obtain a two-dimensional rendered image that is completely consistent with the perspective of the real-time monocular endoscope image data at each time, and all two-dimensional rendered images are organized in time sequence as ideal monocular endoscope image data. For example, at a certain time, the real-time monocular endoscope image data shows that the stomach wall is blurred and has reflection due to respiration, while the ideal monocular endoscope image data rendered by step S4 presents a clear image of the stomach wall generated by the 3D reconstruction constraint network, with smooth deformation and flawless reflection. The difference between the two, such as edge alignment error and texture difference, is the basis for feedback optimization of the 3D reconstruction constraint network in subsequent steps.

[0128] Step S5, using a rendering discrimination update mechanism to perform residual discrimination on the real-time and ideal monocular endoscope image data, obtaining a discrimination update parameter group, and feeding back and optimizing the 3D reconstruction constraint network based on the discrimination update parameter group.

[0129] Step S5 includes:

[0130] Step S51, the rendering discrimination update mechanism includes a rendering-perception consistency verification module and a dynamic scene quality evaluation module, a perception parameter correction signal is generated based on the rendering-perception consistency verification module, a model optimization signal is generated based on the dynamic scene quality evaluation module, and the perception parameter correction signal and the model optimization signal are combined into a discrimination update parameter group.

[0131] It can be understood that the rendering-perception consistency verification module is used to obtain the reprojection error and the optical flow difference between the ideal monocular endoscope image data and the real-time monocular endoscope image data by using a multi-scale feature matching and semantic consistency evaluation algorithm, and to generate a perception parameter correction signal.

[0132] Specifically, a multi-scale pyramid feature matching algorithm can be used to calculate the local feature matching error between the ideal monocular endoscope image data and the real-time monocular endoscope image data at different levels of the image pyramid as the reprojection error, and a dense optical flow algorithm such as Farneback optical flow can be used to calculate the pixel-level motion vector difference between the ideal monocular endoscope image data and the real-time monocular endoscope image data as the optical flow difference. The reprojection error reflects the accuracy of three-dimensional geometric projection, and the optical flow difference reflects the continuity of motion estimation. The reprojection error and the optical flow difference are fused by normalization, weighted summation, etc. to generate a perception parameter correction signal.

[0133] It can be understood that the dynamic scene quality evaluation module is used to obtain the time sequence stability and geometric reasonableness of the ideal monocular endoscope image data by using a time sequence filtering anomaly detection and surface normal consistency verification algorithm based on a dynamic convolution kernel, and to quantify the time sequence stability and geometric reasonableness into a model optimization signal.

[0134] Specifically, based on the ideal monocular endoscope image data, a time series convolution network can be used to evaluate the time series stability of the ideal monocular endoscope image data by detecting whether there is unreasonable mutation or jitter between frames, and taking the jitter amplitude as the time series stability score. Based on the endoscope dynamic 3D Gaussian scene, the surface normal map of the scene is generated by calculating the spatial gradient of adjacent Gaussian points. On the surface normal map, for each pixel point, the cosine value of the normal direction angle with the neighborhood pixels is calculated. The neighborhood pixels can use eight-neighbor. If the cosine value is lower than a certain threshold, such as 0.8, it is determined that the neighborhood is inconsistent with the normal. The proportion of the normal inconsistency area is taken as the geometric rationality score to evaluate its geometric rationality. The time series stability score and the geometric rationality score are fused by normalization, weighted summation and other ways to generate a model optimization signal.

[0135] Step S52, based on the perception parameter correction signal, iteratively updating the medical instrument tracking model parameters, global statistical parameters and frequency domain analysis and signal extraction parameters.

[0136] It can be understood that the perception parameter correction signal, as a quantitative indicator of multi-source error, is used to guide the adaptive adjustment of the medical instrument tracking model parameters, global statistical parameters and frequency domain analysis and signal extraction parameters to ensure that their output is consistent with the dynamic 3D Gaussian scene.

[0137] Specifically, based on the re-projection error in the perception parameter correction signal, the medical instrument re-projection error is obtained. Frames with medical instrument re-projection error continuously higher than a threshold are identified as medical instrument tracking difficult cases, and the weight of the medical instrument tracking difficult cases in the medical instrument tracking model loss function is increased. For example, when the forceps instrument has a re-projection error exceeding 5 pixels in 5 consecutive frames, these frames are marked as medical instrument tracking difficult cases. During model training, the loss weight of these medical instrument tracking difficult cases is set to twice that of normal frames, so that the model pays more attention to these difficult-to-accurately-track situations, and improves its robustness in complex scenes such as tissue occlusion and reflection.

[0138] Further, the global statistical parameters used to generate the semantic feature vector, such as the sliding average of the class proportion, the historical mean and variance of the centroid position, need to adapt to scene changes. The perception parameter correction signal is used as a global statistical adaptive factor to optimize the global statistical parameters. For example, when the optical flow difference shows that the scene is moving rapidly, the system will dynamically reduce the size of the sliding window used for statistics, such as from 30 frames to 10 frames, so that the statistical features can respond more quickly to scene changes. When the re-projection error is small, a larger window is restored to maintain stability.

[0139] It can be understood that the accuracy of the physiological signal vector extraction depends on the frequency domain analysis and signal extraction parameters, the motion continuity information can be corrected based on the perception parameters, and the frequency domain analysis and signal extraction parameters can be optimized based on the motion continuity information. For example, it is assumed that the optical flow difference indicates that there is too much high-frequency noise in the extracted physiological signal vector, such as a fast moving artifact of the device, and the upper limit frequency of the band-pass filter is adaptively adjusted to be more strictly limited in the physiological frequency range, such as 0.1-2Hz, so as to suppress the noise.

[0140] In step S53, the weight parameters of the linear transformation layer of the 3D reconstruction constraint network are iteratively updated based on the model optimization signal using a back propagation algorithm.

[0141] In step S53, the weight parameters of the linear transformation layer of the 3D reconstruction constraint network are iteratively updated based on the model optimization signal using a back propagation algorithm.

[0142] In step S53, the weight parameters of the linear transformation layer of the 3D reconstruction constraint network are iteratively updated based on the model optimization signal using a back propagation algorithm.

[0143] Specifically, the model optimization signal includes a time sequence stability score and a geometric rationality score of each frame. The time sequence stability score and the geometric rationality score are multiplied and normalized to obtain the confidence weight of each frame. The higher the confidence weight, the more reliable the dynamic 3D Gaussian scene quality corresponding to the frame.

[0144] Further, for each time sequence frame, the multi-scale structural similarity loss MS-SSIM and the L1 photometric loss between the ideal monocular endoscope image data and the real-time monocular endoscope image data are calculated, and the MS-SSIM and the L1 photometric loss are added to obtain the original loss of the frame. For example, in the t-th frame, the MS-SSIM is 0.2 and the L1 photometric loss is 0.15, so the original loss of the frame is 0.35. At the same time, the time sequence stability score is 0.9 and the geometric rationality score is 0.8, so the confidence weight is 0.72. MS-SSIM focuses on evaluating the similarity of image structure, and L1 photometric loss directly measures the difference in pixel intensity. The combination of MS-SSIM and L1 photometric loss can comprehensively evaluate the difference between the rendered image and the real image.

[0145] In step S532, the global rendering quality loss is obtained based on the confidence weight of each time sequence frame, and the global rendering quality loss gradient is obtained by a back propagation algorithm.

[0146] Specifically, the original losses of all frames in a continuous period of time, for example, in a window containing 10 time sequence frames, are weighted and summed according to the corresponding confidence weights to obtain a global rendering quality loss, which represents the overall rendering performance of the 3D reconstruction constraint network on multiple frames. Then, the gradient of the global rendering quality loss with respect to the weight parameters of all fully connected layers in the 3D reconstruction constraint network, i.e., the global rendering quality loss gradient, is calculated by a back propagation algorithm. The global rendering quality loss gradient indicates the direction and magnitude in which the weight parameters of all fully connected layers in the 3D reconstruction constraint network should be adjusted to reduce the global rendering quality loss.

[0147] In step S533, a historical high-confidence frame is obtained through a screening operation, and a principal component analysis is performed on the original loss of the historical high-confidence frame to obtain a dominant direction as the network main feature direction.

[0148] Specifically, a historical buffer is established to store the original losses and corresponding confidence weights of recently processed frames, for example, the original losses and corresponding confidence weights of the last 100 frames. The historical high-confidence frames with the highest confidence weights are periodically screened from the historical buffer. For example, after processing 10 frames, the 20 frames with the highest confidence weights are screened from the historical buffer as the historical high-confidence frames. The original losses of the historical high-confidence frames are combined into an original loss matrix. A principal component analysis is performed on the original loss matrix, and the direction defined by the first principal component is extracted. This direction represents the core optimization path followed by the 3D reconstruction constraint network when it achieves the best performance in a historical period. The core optimization path is defined as the network main feature direction.

[0149] In step S534, the global rendering quality loss gradient is projected onto the network main feature direction to obtain an updated feature direction. The feature direction angle between the updated feature direction and the network main feature direction is constrained based on a preset safety threshold.

[0150] Specifically, the global rendering quality loss gradient obtained in step S532 is projected onto the network main feature direction obtained in step S533 to obtain the updated feature direction of the current iteration. The angle between the updated feature direction and the network main feature direction, i.e., the feature direction angle, is calculated. A safety threshold, for example, 15 degrees, is preset to determine the reliability of the current updated feature direction. In step S535, the 3D reconstruction constraint network is updated and adjusted based on the historical high-confidence frames or the global rendering quality loss gradient according to the feature direction angle.

[0151] Specifically, when the feature direction angle is less than or equal to the safety threshold, it indicates that the current updated feature direction is basically consistent with the historical successful experience, and the update risk is low. At this time, the current updated feature direction can be directly applied, and the weight parameters of all linear transformation layers of the 3D reconstruction constraint network are iteratively updated by a back propagation algorithm.

[0152] Further, when the feature direction included angle is greater than the safety threshold value, it indicates that the current updated feature direction deviates greatly from the historical main feature direction, which may be caused by the dramatic change of the scene or the temporary abnormality of the input data, and directly applying the current updated feature direction may cause the overall rendering performance of the 3D reconstruction constraint network to be unstable, at this time, the system enables a conservative update strategy, ignores the current global rendering quality loss gradient, recalculates a conservative loss based on the historical high confidence frame, such as the average value of the original loss corresponding to the historical high confidence frame, and guides the 3D reconstruction constraint network to make a small amplitude parameter adjustment along the network main feature direction, the conservative update strategy preferentially ensures the stability of the 3D reconstruction constraint network.

[0153] The second aspect of the application provides a real-time dynamic three-dimensional reconstruction system of an endoscope image, comprising:

[0154] A data acquisition preprocessing module is configured to acquire initial monocular endoscope image data and construct an endoscope image training data set by using a Gaussian splash algorithm and a chaos algorithm.

[0155] A 3D reconstruction constraint network module is configured to construct a 3D reconstruction constraint network based on a multi-layer perception mechanism, and pre-train the 3D reconstruction constraint network in combination with an endoscope image training data set and a preset physical loss function.

[0156] A dynamic scene generation and rendering module is configured to acquire real-time monocular endoscope image data, input the 3D reconstruction constraint network for dynamic reconstruction, generate a dynamic 3D Gaussian scene, perform scene rendering on the endoscope dynamic 3D Gaussian scene based on endoscope visual parameters corresponding to the real-time monocular endoscope image data, and generate ideal monocular endoscope image data.

[0157] A rendering discrimination update module is configured to use a rendering discrimination update mechanism to perform residual discrimination on real-time and ideal monocular endoscope image data, acquire a discrimination update parameter group, and perform feedback optimization on the 3D reconstruction constraint network based on the discrimination update parameter group.

[0158] The specific use mode and effect of the application are described as follows:

[0159] The embodiment of the application obtains initialization monocular endoscope image data, constructs an endoscope image training data set by using a Gaussian splash algorithm and a chaos algorithm, constructs a 3D reconstruction constraint network based on a multilayer perception mechanism, pre-trains the network in combination with a physical loss function, obtains real-time monocular endoscope image data, inputs the data into the network for dynamic reconstruction to generate a dynamic 3D Gaussian scene, renders the dynamic scene based on endoscope visual parameters to generate ideal monocular endoscope image data, and uses a rendering discrimination update mechanism to perform residual discrimination and feedback optimization. The above steps first construct a static 3D Gaussian scene by using a Gaussian splash algorithm and generate a dynamic disturbance condition vector by using a chaos algorithm, simulate the pseudo-random characteristics of dynamic changes such as tissue peristalsis and instrument movement in the endoscope scene, provide a real and diversified dynamic prior data basis for training, fundamentally overcome the limitations of traditional methods based on static assumptions or sparse feature matching that cannot capture the details of soft tissue deformation, and solve the problem of poor dynamic adaptability.

[0160] On the other hand, by constructing a multilayer perception mechanism containing a condition modulation block as a 3D reconstruction constraint network, and introducing a physical loss function weighted by an isometric deformation loss, a motion smoothness loss, a periodic motion loss and an instrument rigid motion loss, the physical laws such as tissue deformation preservation, motion continuity, physiological cycle prior and instrument rigidity constraint are embedded in the pre-training, ensuring the physical rationality of dynamic reconstruction, improving the robustness of the model in texture missing or occluded scenes, and further, in the real-time reconstruction stage, by encapsulating the instrument state vector, the semantic feature vector and the physiological signal vector into a programming condition vector, the network is driven to predict the Gaussian point attribute adjustment amount, realizing real-time conversion from a static scene to a dynamic scene, combining a differentiable renderer to quickly generate ideal image data, effectively balancing the calculation efficiency and reconstruction accuracy, and overcoming the defects of high delay and large resource consumption of existing deep learning models.

[0161] In addition, by using the rendering-perception consistency verification module and the dynamic scene quality evaluation module in the rendering discrimination update mechanism, the real-time and ideal image data are subjected to multi-scale feature matching, optical flow difference analysis and time sequence stability detection, the perception parameter correction signal and the model optimization signal are generated, and based on this, the medical instrument tracking model, the multi-modal statistical parameters and the network weights are iteratively updated, realizing adaptive optimization and closed-loop feedback of the reconstruction process, and solving the problems of insufficient multi-modal data fusion and motion artifact compensation.

Claims

1. An endoscope image real-time dynamic three-dimensional reconstruction method, characterized in that, The method comprises: acquiring initialization monocular endoscope image data, and constructing endoscope image training data set through Gaussian splash algorithm and chaos algorithm, wherein the initialization monocular endoscope image data comprises endoscope multi-modal data and endoscope visual parameters; constructing a 3D reconstruction constraint network based on a multi-layer perception mechanism, and pre-training the 3D reconstruction constraint network in combination with the endoscope image training data set and a preset physical loss function; acquiring real-time monocular endoscope image data, inputting the 3D reconstruction constraint network for dynamic reconstruction, and generating a dynamic 3D Gaussian scene; based on the endoscope visual parameters corresponding to the real-time monocular endoscope image data, performing scene rendering on the endoscope dynamic 3D Gaussian scene to generate ideal monocular endoscope image data; using a rendering discrimination update mechanism to perform residual error discrimination on the real-time and ideal monocular endoscope image data, and acquiring a discrimination update parameter group, and based on the discrimination update parameter group, performing feedback optimization on the 3D reconstruction constraint network; the rendering discrimination update mechanism comprises a rendering-perception consistency verification module and a dynamic scene quality evaluation module, a perception parameter correction signal is generated based on the rendering-perception consistency verification module, a model optimization signal is generated based on the dynamic scene quality evaluation module, and the perception parameter correction signal and the model optimization signal are combined into the discrimination update parameter group; based on the perception parameter correction signal, iteratively updating medical instrument tracking model parameters, global statistical parameters, and frequency domain analysis and signal extraction parameters; based on the model optimization signal, iteratively updating the weight parameters of the linear transformation layer of the 3D reconstruction constraint network using a back propagation algorithm.

2. The method of claim 1, wherein, The method comprises: acquiring initialization monocular endoscope image data, and constructing endoscope image training data set through Gaussian splash algorithm and chaos algorithm, wherein the initialization monocular endoscope image data comprises endoscope multi-modal data and endoscope visual parameters, comprising: acquiring a monocular endoscope initialization sequence, and acquiring initialization endoscope visual parameters based on the monocular endoscope initialization sequence using an improved VIO algorithm, wherein the initialization endoscope visual parameters comprise camera intrinsic parameters and initialization camera pose; based on the monocular endoscope initialization sequence and the initialization endoscope visual parameters, generating initialization endoscope multi-modal data, wherein the initialization endoscope multi-modal data comprises initialization RGB images, initialization depth maps, and initialization semantic segmentation maps; based on the initialization RGB images, constructing a static 3D Gaussian scene using a Gaussian splash algorithm, defining an endoscope Gaussian density control strategy based on the initialization depth maps and the initialization semantic segmentation maps, and refining the static 3D Gaussian scene based on the endoscope Gaussian density control strategy; based on the initialization semantic segmentation maps, assigning a main semantic label and a corresponding confidence to each Gaussian point in the static 3D Gaussian scene through a back projection and a voting mechanism; using a chaos algorithm to generate a dynamic disturbance condition vector, importing the dynamic disturbance condition vector into the static 3D Gaussian scene, and obtaining a disturbance Gaussian point attribute adjustment amount through reverse optimization, and corresponding and combining the dynamic disturbance condition vector and the disturbance Gaussian point attribute adjustment amount into the endoscope image training data set.

3. The method of claim 1, wherein, The 3D reconstruction constraint network is constructed based on a multi-layer perception mechanism, and the 3D reconstruction constraint network is pre-trained in combination with an endoscope image training data set and a preset physical loss function, including: A multi-layer perception mechanism including a conditional modulation block is taken as the 3D reconstruction constraint network, and the conditional modulation block is used for embedding a dynamic disturbance condition vector into intermediate layer features; A preset physical loss function is obtained, and the preset physical loss function is specifically a weighted sum of an equidistance deformation loss term, a motion smoothness loss term, a periodic motion loss term and an instrument rigid motion loss term; A dynamic disturbance condition vector in an endoscope image training data set is taken as input, and a corresponding disturbance Gaussian point attribute adjustment amount is taken as a supervision label, and a network prediction value is calculated through forward propagation; A prediction training total loss is calculated based on the network prediction value and in combination with the preset physical loss function, and constraint weight parameters of a linear transformation layer of the 3D reconstruction constraint network are iterated based on the prediction training total loss until the prediction training total loss converges, and the pre-training of the 3D reconstruction constraint network is completed.

4. The method of claim 3, wherein, A preset physical loss function is obtained, and the preset physical loss function is specifically a weighted sum of an equidistance deformation loss term, a motion smoothness loss term, a periodic motion loss term and an instrument rigid motion loss term, including: The equidistance deformation loss term is used for constraining local area preservation characteristics in a tissue deformation process; The motion smoothness loss term is used for constraining continuity of adjacent Gaussian points on a motion vector; The periodic motion loss term is used for taking a physiological periodic motion prior as a regularization constraint; The instrument rigid motion loss term is obtained based on an initialization semantic segmentation map in endoscope multi-modal data, and the instrument rigid motion loss term is used for constraining a Gaussian point cloud identified as a medical instrument to follow a rigid motion rule.

5. The method of claim 1, wherein, Real-time monocular endoscope image data is obtained, and is input into the 3D reconstruction constraint network for dynamic reconstruction to generate a dynamic 3D Gaussian scene, including: Real-time endoscope visual parameters and real-time endoscope multi-modal data are obtained, and the real-time endoscope visual parameters and the real-time endoscope multi-modal data are packaged as the real-time monocular endoscope image data; Instrument state information is obtained based on the real-time endoscope visual parameters and through a medical instrument tracking model, and the instrument state information is encoded as an instrument state vector; Real-time semantic segmentation maps in the real-time endoscope multi-modal data are encoded as semantic feature vectors through global statistics, and real-time RGB images and real-time depth maps in the real-time endoscope multi-modal data are encoded as physiological signal vectors; The instrument state vector, the semantic feature vector and the physiological signal vector are subjected to time series filtering and standardization, and are packaged as a programming condition vector; The programming condition vector is input into the 3D reconstruction constraint network, real-time Gaussian point attribute adjustment amounts are obtained, the real-time Gaussian point attribute adjustment amounts are applied to a static 3D Gaussian scene, and a dynamic 3D Gaussian scene is obtained in real time.

6. The method of claim 1, wherein, An endoscope dynamic 3D Gaussian scene is rendered based on endoscope visual parameters corresponding to real-time monocular endoscope image data to generate ideal monocular endoscope image data, including: Specifically, a differentiable renderer based on Z-Buffer depth sorting and Alpha blending is adopted to render the endoscopic dynamic 3D Gaussian scene into time-sequenced endoscopic two-dimensional images according to real-time camera poses, and the time-sequenced endoscopic two-dimensional images are taken as ideal monocular endoscopic image data.

7. The method of claim 1, wherein, The rendering discrimination updating mechanism comprises a rendering-perception consistency checking module and a dynamic scene quality evaluation module, and comprises: The rendering-perception consistency checking module is configured to obtain reprojection error and optical flow difference between the ideal monocular endoscopic image data and the real-time monocular endoscopic image data by using a multi-scale feature matching and semantic consistency evaluation algorithm, and to generate a perception parameter correction signal. The dynamic scene quality evaluation module is configured to obtain time sequence stability and geometric reasonableness of the ideal monocular endoscopic image data by using a time sequence filtering anomaly detection and surface normal consistency verification algorithm based on a dynamic convolution kernel, and to quantify the time sequence stability and the geometric reasonableness into a model optimization signal.

8. The method of claim 1, wherein, The perception parameter correction signal is used to iteratively update medical instrument tracking model parameters, global statistical parameters, and frequency domain analysis and signal extraction parameters, comprising: Based on the model optimization signal, a confidence weight arranged in time sequence is obtained, and based on the ideal monocular endoscopic image data and the real-time monocular endoscopic image data, an original loss of each time sequence frame is calculated. Based on the confidence weight of each time sequence frame, a global rendering quality loss is obtained, and a global rendering quality loss gradient is obtained through a back propagation algorithm. The historical high-confidence frames are obtained through a screening operation, the dominant direction is obtained by using principal component analysis based on the original loss of the historical high-confidence frames, and the dominant direction is taken as the network main feature direction. The global rendering quality loss gradient is projected to the network main feature direction to obtain an updated feature direction, and the feature direction included angle between the updated feature direction and the network main feature direction is constrained based on a preset safety threshold. According to the feature direction included angle, the 3D reconstruction constraint network is updated and adjusted based on the historical high-confidence frames or the global rendering quality loss gradient.

9. An endoscopic image real-time dynamic 3D reconstruction system applied to the method of any one of claims 1 to 8, characterized in that, Comprise: The data acquisition preprocessing module is configured to obtain initialization monocular endoscopic image data, and construct an endoscopic image training dataset by using a Gaussian splashing algorithm and a chaos algorithm; The 3D reconstruction constraint network module is configured to construct a 3D reconstruction constraint network based on a multi-layer perception mechanism, and pre-train the 3D reconstruction constraint network in combination with the endoscopic image training dataset and a preset physical loss function; The dynamic scene generation and rendering module is configured to obtain real-time monocular endoscopic image data, input the 3D reconstruction constraint network for dynamic reconstruction, generate a dynamic 3D Gaussian scene, and perform scene rendering on the endoscopic dynamic 3D Gaussian scene based on endoscopic visual parameters corresponding to the real-time monocular endoscopic image data to generate ideal monocular endoscopic image data; The rendering discrimination updating module is configured to use the rendering discrimination updating mechanism to perform residual discrimination on the real-time and ideal monocular endoscopic image data, obtain a discrimination updating parameter group, and feed back and optimize the 3D reconstruction constraint network based on the discrimination updating parameter group.

Images