A 3D Reconstruction Method and System for Binocular Videos of Minimally Invasive Partial Nephrectomy

Abstract

The present invention provides a three-dimensional reconstruction method and system for binocular videos of minimally invasive partial nephrectomy, which relates to the technical field of medical image processing. The three-dimensional reconstruction method for binocular videos of minimally invasive partial nephrectomy includes the following steps: image data acquisition, artifact detection and evaluation, image registration evaluation, and comprehensive evaluation and judgment. By obtaining the preliminary three-dimensional reconstruction result and the intraoperative real-time image data, performing image processing on the real-time image data to obtain enhanced image data, then detecting and evaluating artifacts in the enhanced image data and determining whether to perform image registration, and then evaluating the registration quality of the image registration to obtain a comprehensive evaluation result and determining whether to perform dynamic three-dimensional reconstruction, the present invention achieves the effect of improving the accuracy of real-time dynamic three-dimensional image reconstruction during laparoscopic partial nephrectomy, and solves the problem of low accuracy of real-time dynamic three-dimensional image reconstruction during laparoscopic partial nephrectomy in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and particularly to a three-dimensional reconstruction method and system for binocular videos of minimally invasive partial nephrectomy. Background Art

[0002] Laparoscopic partial nephrectomy is a common minimally invasive surgical method mainly used for treating localized renal tumors. Compared with traditional open surgery, laparoscopic surgery has the advantages of less trauma, faster recovery, and fewer complications. However, due to the complexity and delicacy of the surgery, high-quality image guidance is required during the operation to ensure the accuracy and safety of the surgery.

[0003] Existing dynamic three-dimensional reconstructions for laparoscopic partial nephrectomy images acquire image data within the patient's abdominal cavity simultaneously or alternately through multiple imaging devices (such as ultrasound), and use high-frame-rate cameras or sensors to capture possible deformations and movements of the kidney and its surrounding tissues during the operation. Based on multi-modal image data, three-dimensional reconstruction is performed using computer vision and graphics algorithms to form an accurate model of the kidney and its surrounding tissues. By updating the three-dimensional model in real time, the changes in tissues during the operation are reflected, enabling doctors to intuitively observe the dynamic changes in the surgical area.

[0004] Dynamic three-dimensional reconstruction involves multi-modal image data acquisition, high-performance computing, and advanced graphics processors. The real-time update of the three-dimensional model requires extremely high computing resources and efficiency. Especially when dealing with large-scale data or complex surgical scenarios, delays or a decrease in accuracy may occur.

[0005] In the prior art, dynamic three-dimensional reconstruction generates a three-dimensional model of an organ or structure from continuously acquired image data. In medical images, non-real signals or distortions introduced for various reasons (such as metal implants, etc.) may affect the clarity and accuracy of the images, thereby affecting the accuracy of dynamic three-dimensional reconstruction during surgery. During laparoscopic surgery, especially in partial nephrectomy, there is a problem of low accuracy in real-time dynamic three-dimensional image reconstruction for laparoscopic partial nephrectomy. Summary of the Invention

[0006] The present invention provides a three-dimensional reconstruction method and system for binocular videos of minimally invasive partial nephrectomy, which solves the problem of low accuracy in real-time dynamic three-dimensional image reconstruction for laparoscopic partial nephrectomy in the prior art, and realizes an improvement in the accuracy of real-time dynamic three-dimensional image reconstruction for laparoscopic partial nephrectomy. The technical solution is as follows:

[0007] A three-dimensional reconstruction method for minimally invasive partial nephrectomy binocular video, comprising the following steps: S1, obtaining a preliminary three-dimensional reconstruction result based on the original image data, obtaining real-time image data within a preset time interval during the operation, performing image processing on the real-time image data to obtain enhanced image data, and the original image data is obtained from the binocular video; S2, performing artifact detection and evaluation on the enhanced image data to obtain an artifact detection effect index, and determining whether to perform image registration based on the artifact detection effect index and a preset detection threshold range. If not, a feedback reminder for artifact detection is given to a preset person to re-obtain the real-time image data, and the artifact detection effect index is used to quantify the influence degree of artifacts on the enhanced image data; S3, after performing image registration, obtaining the registered image data after image registration of the enhanced image data, and evaluating the registration quality of the image registration according to the enhanced image data and the registered image data to obtain a registration quality evaluation index, and the registration quality evaluation index is used to describe the quality alignment degree of the registered image data during the registration process; S4, obtaining a comprehensive evaluation result based on the registration quality evaluation index and a preset registration threshold range, and determining whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result, and the comprehensive evaluation result is used to determine whether to perform dynamic three-dimensional reconstruction.

[0008] Optionally, before obtaining the preliminary three-dimensional reconstruction result based on the original image data, the following steps are further included: obtaining the preoperative image data of the patient and importing it into three-dimensional reconstruction software, and the preoperative image data is stored in DICOM format; identifying the kidney and its surrounding structures by processing the preoperative image data using an automatic segmentation algorithm to obtain initial image data, and performing Gaussian filtering processing on the initial image data to obtain the original image data.

[0009] Optionally, the specific steps for obtaining the enhanced image data are as follows: performing smoothing processing on the real-time image data through Gaussian filtering to obtain smoothed image data, and performing edge processing on the smoothed image data to obtain the first enhanced image data; performing high-frequency component enhancement processing on the first enhanced image data to obtain the second enhanced image data; performing histogram equalization processing on the second enhanced image data to obtain the enhanced image data.

[0010] Optionally, the specific method for obtaining the artifact detection effect index is as follows: Identify the artifact regions in the enhanced image data through edge detection and number them, obtain the number of pixels and the corresponding pixel brightness values within the artifact regions corresponding to the numbers; perform statistical analysis on the pixel brightness values to obtain the brightness-related values of the artifact regions, obtain the directional gradients of the enhanced image data, where the directional gradients include horizontal directional gradients and vertical directional gradients, and the brightness-related values include the maximum pixel brightness, the minimum pixel brightness, and the average pixel brightness; combine the number of pixels, pixel brightness values, brightness-related values, and directional gradients of the enhanced image data in the artifact regions with the evaluation weights obtained from a preset database to obtain the artifact detection effect index, where the evaluation weights include artifact intensity weights, artifact contrast weights, and edge sharpness weights.

[0011] Optionally, the artifact detection effect index is calculated using the following formula:

[0012]

[0013] In the formula, W represents the artifact detection effect index, i represents the pixel number within the artifact region, i = 1, 2,..., N, N represents the total number of pixels within the artifact region, j represents the artifact region number of the enhanced image data, j = 1, 2,..., M, M represents the total number of artifact regions of the enhanced image data, represents the pixel brightness value of the i-th pixel within the j-th artifact region, I max represents the maximum pixel brightness, I min represents the minimum pixel brightness, I a represents the average pixel brightness, I(x, y) represents the pixel value of the enhanced image data at the coordinate (x, y), represents the horizontal directional gradient, represents the vertical directional gradient, c1 represents the artifact intensity weight, c2 represents the artifact contrast weight, c3 represents the edge sharpness weight, ΔA represents the pixel brightness reference value, ΔB represents the artifact contrast reference value, and ΔC represents the edge sharpness reference value.

[0014] Optionally, the specific steps for performing image registration on the enhanced image data are as follows: Obtain the enhanced image data and adjacent enhanced image data within a preset time interval and perform feature extraction to obtain invariant feature points, where the adjacent enhanced image data is used to describe the enhanced image data of the adjacent preset time interval, and the invariant feature points include enhanced invariant feature points and adjacent invariant feature points; perform feature matching on the invariant feature points to obtain matching feature points, obtain a transformation matrix by processing the matching feature points using the least squares method, and perform image alignment on the transformation matrix to obtain aligned image data; perform image fusion on the aligned image data and the enhanced image data to obtain registered image data.

[0015] Optionally, the specific method for obtaining the registration quality evaluation index is as follows: Obtain the total number of pixels of the registered image data and the gray values of the image pixels, where the gray values of the image pixels include the gray values of the registered image data and the enhanced image data; obtain the average gray value of the image based on the gray values of the registered image data and the enhanced image data at corresponding pixels, where the average gray value of the image includes the average gray value of the registered image data and the average gray value of the enhanced image data; combine the total number of pixels of the registered image data, the gray values of the image pixels, the average gray value of the image with the reference gray values and evaluation weights obtained from a preset database to obtain the registration quality evaluation index, where the evaluation weights include the weight of the degree of difference and the weight of the degree of cross-correlation, and the reference gray values include the reference gray value of the registered image data and the reference gray value of the enhanced image data.

[0016] Optionally, the registration quality evaluation index is calculated using the following formula:

[0017]

[0018] In the formula, Z represents the registration quality evaluation index, q represents the pixel number of the registered image data, q = 1, 2,..., Q, and Q represents the total number of pixels of the registered image data. represents the gray value of the registered image data of the q-th pixel. represents the gray value of the enhanced image data of the q-th pixel. represents the average gray value of the registered image data. represents the average gray value of the enhanced image data, Δp represents the reference gray value of the registered image data, Δh represents the reference gray value of the enhanced image data, α represents the weight of the degree of difference, β represents the weight of the degree of cross-correlation, ΔM represents the reference value of the registration difference, and ΔN represents the reference value of the registration cross-degree.

[0019] Optionally, the specific steps for determining whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result are as follows: Obtain the registration quality evaluation index and the preset registration threshold range, compare the registration quality evaluation index with the preset registration threshold range. If the registration quality evaluation index is within the preset registration threshold range, perform dynamic three-dimensional reconstruction; otherwise, feedback the registration quality evaluation index to remind the preset personnel to replace the registration technology and perform the registration quality evaluation of the image again to obtain the registration quality evaluation index.

[0020] The present invention provides a three-dimensional reconstruction system for minimally invasive partial nephrectomy binocular video, comprising: an image data acquisition module, an artifact detection and evaluation module, an image registration evaluation module, and a comprehensive evaluation and judgment module; wherein, the image data acquisition module is used to obtain a preliminary three-dimensional reconstruction result based on the original image data, obtain real-time image data within a preset time interval during the operation, perform image processing on the real-time image data to obtain enhanced image data, and the original image data is obtained from the binocular video; the artifact detection and evaluation module is used to perform artifact detection and evaluation on the enhanced image data to obtain an artifact detection effect index, and judge whether to perform image registration based on the artifact detection effect index and a preset detection threshold range. If not, a preset person is reminded through artifact detection feedback to re-obtain the real-time image data, and the artifact detection effect index is used to quantify the influence degree of artifacts on the enhanced image data; the image registration evaluation module is used to obtain the registered image data after image registration of the enhanced image data after performing image registration, and evaluate the registration quality of the image registration according to the enhanced image data and the registered image data to obtain a registration quality evaluation index, and the registration quality evaluation index is used to describe the quality alignment degree of the registered image data during the registration process; the comprehensive evaluation and judgment module is used to perform an evaluation based on the registration quality evaluation index and a preset registration threshold range to obtain a comprehensive evaluation result, and judge whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result, and the comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

[0021] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0022] 1. By performing image processing on the real-time image data to obtain enhanced image data, then performing artifact detection and evaluation on the enhanced image data, and judging whether to perform image registration based on the artifact detection effect index and a preset detection threshold range. If image registration is performed, the registration quality of the image registration is evaluated to obtain a comprehensive evaluation result. Finally, it is judged whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result, thereby realizing the artifact detection of intraoperative real-time image data and the evaluation of image registration, and further realizing the improvement of the accuracy of intraoperative real-time dynamic three-dimensional image reconstruction for partial nephrectomy of the kidney in the abdominal cavity, effectively solving the problem of low accuracy of intraoperative real-time dynamic three-dimensional image reconstruction for partial nephrectomy of the kidney in the abdominal cavity existing in the prior art.

[0023] 2. Identify the artifact regions in the enhanced image data through edge detection and number them. Obtain the number of pixels and the corresponding pixel brightness values within the artifact regions corresponding to the numbers. Conduct statistical analysis on the pixel brightness values to obtain the brightness-related values of the artifact regions and obtain the directional gradient of the enhanced image data. Combine the number of pixels, pixel brightness values, brightness-related values, and directional gradient of the enhanced image data with the evaluation weights obtained from the preset database to obtain the artifact detection effect index, thereby realizing the quantification of the influence degree of artifacts on the enhanced image data, and further realizing a more accurate evaluation of the influence degree of artifacts on the enhanced image data.

[0024] 3. Obtain the total number of pixels and the pixel gray values of the registered image data, and obtain the average image gray value based on the gray values of the registered image data and the enhanced image data at the corresponding pixels. Then, combine the total number of pixels, pixel gray values, and average image gray value of the registered image data with the reference gray value and evaluation weights obtained from the preset database to obtain the registration quality evaluation index, thereby realizing the quantification of the quality alignment degree of the registered image data during the registration process, and further realizing a more accurate evaluation of the image registration of the enhanced image data. Brief Description of the Drawings

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following-described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0026] Figure 1 It is a flowchart of a three-dimensional reconstruction method for a binocular video of minimally invasive partial nephrectomy provided by an embodiment of the present application;

[0027] Figure 2 It is a schematic diagram of the change of the artifact detection effect index provided by an embodiment of the present application;

[0028] Figure 3 It is a schematic structural diagram of a three-dimensional reconstruction system for a binocular video of minimally invasive partial nephrectomy provided by an embodiment of the present application. Detailed Embodiments

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions of the embodiments of the present invention in conjunction with the drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the described embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0030] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the ordinary meanings as understood by those of ordinary skill in the field to which the present invention pertains. The terms "first", "second" and similar terms used in the present invention do not denote any order, quantity or importance, but are only used to distinguish different components. Similarly, terms such as "a", "an" or "the" do not denote a limitation of quantity, but mean that there is at least one. Terms such as "comprising" or "including" mean that the elements or items appearing before the term cover the elements or items listed after the term and their equivalents, without excluding other elements or items. Terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0031] It should be noted that the terms "upper", "lower", "left", "right", "front", "rear", etc. used in the present invention are only used to represent relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0032] In view of the problem of low accuracy in real-time dynamic three-dimensional image reconstruction in laparoscopic partial nephrectomy in the prior art, the present invention provides a three-dimensional reconstruction method and system for binocular video of minimally invasive partial nephrectomy, achieving an improvement in the accuracy of real-time dynamic three-dimensional image reconstruction in laparoscopic partial nephrectomy.

[0033] Such as Figure 1As shown in the figure, it is a flowchart of a three-dimensional reconstruction method for binocular videos of minimally invasive partial nephrectomy provided by the present invention. A three-dimensional reconstruction method for binocular videos of minimally invasive partial nephrectomy provided by the present invention includes the following steps: S1, obtaining a preliminary three-dimensional reconstruction result based on the original image data, obtaining real-time image data within a preset time interval during the operation, performing image processing on the real-time image data to obtain enhanced image data, and the original image data is obtained from the binocular video; S2, performing artifact detection and evaluation on the enhanced image data to obtain an artifact detection effect index, and judging whether to perform image registration based on the artifact detection effect index and a preset detection threshold range. If not, a reminder for artifact detection feedback is given to a preset person to re-obtain the real-time image data. The artifact detection effect index is used to quantify the influence degree of artifacts on the enhanced image data; S3, after performing image registration, obtaining the registered image data after image registration of the enhanced image data, and evaluating the registration quality of the image registration based on the enhanced image data and the registered image data to obtain a registration quality evaluation index. The registration quality evaluation index is used to describe the quality alignment degree of the registered image data during the registration process; S4, obtaining a comprehensive evaluation result based on the registration quality evaluation index and a preset registration threshold range, and judging whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result. The comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

[0034] In this embodiment, the original image data is obtained from the binocular video through a video processing library. The binocular video refers to two videos of the same scene captured simultaneously by two cameras (usually referred to as the left camera and the right camera); judging whether to perform image registration based on the artifact detection effect index and a preset detection threshold range. Specifically, when the artifact detection effect index exceeds the preset detection threshold range, it is not performed. If the artifact detection effect index is within the preset detection threshold range, it is performed; the original image data includes computed tomography (CT) image data, magnetic resonance imaging (MRI) image data, and ultrasound image data. The preliminary three-dimensional reconstruction result is used to describe the three-dimensional shape of the patient's kidney and its surrounding structures before surgery. The real-time image data includes the patient's CT image data, MRI image data, and ultrasound image data during the operation. The image processing includes edge detection and enhancement, high-frequency component enhancement, and histogram equalization. Specifically, for edge detection and enhancement, the real-time image data is first converted to grayscale, and the edges are determined through an edge detection algorithm; high-frequency component enhancement converts the image from the spatial domain to the frequency domain using the discrete Fourier transform and applies a mask to the frequency-domain image to enhance the high-frequency components. Histogram equalization refers to mapping the pixels to the equalized gray levels, thus improving the reliability of the images during minimally invasive partial nephrectomy in the abdominal cavity.

[0035] Among them, to obtain the preliminary 3D reconstruction result based on the original image data, the following steps are also included before: obtaining the preoperative image data of the patient and importing it into 3D reconstruction software (such as RealityCapture), and the preoperative image data is stored in DICOM format; by using an automatic segmentation algorithm to process the preoperative image data to identify the kidney and its surrounding structures to obtain the initial image data, and performing Gaussian filtering on the initial image data to obtain the original image data.

[0036] In this embodiment, it should be understood that the image data is stored in DICOM (Digital Imaging and Communications in Medicine) format, and the DICOM format is the standard format in the field of medical imaging; among them, performing Gaussian filtering on the initial image data makes the sharp edges of the image smoother through convolution operations, thereby reducing the influence of noise; by using an automatic segmentation algorithm to process the preoperative image data to identify the kidney and its surrounding structures, for example, common automatic segmentation algorithms include threshold-based segmentation; the quality of the original image data is improved.

[0037] Among them, the specific steps for obtaining the enhanced image data are as follows: performing smoothing processing on the real-time image data through Gaussian filtering to obtain the smoothed image data, and performing edge processing on the smoothed image data to obtain the first enhanced image data; performing high-frequency component enhancement processing on the first enhanced image data to obtain the second enhanced image data; performing histogram equalization processing on the second enhanced image data to obtain the enhanced image data.

[0038] In this embodiment, performing smoothing processing on the real-time image data through Gaussian filtering, and performing convolution operations on the Gaussian kernel and the image data. The weights of the Gaussian kernel are distributed according to the distance from the center, and the closer to the center, the greater the weight, thus producing a smoothing effect; among them, common edge detection algorithms include Sobel operator, Canny edge detection, etc.; among them, high-frequency component enhancement processing is to strengthen the performance of the details in the image, especially the edges of small structures. Common techniques are Laplacian operator enhancement or non-linear filtering; among them, histogram equalization is used to adjust the contrast of the image, and by redistributing the gray levels of the image pixels, the pixel distribution of different gray values becomes more uniform; the reliability of the enhanced image data is improved.

[0039] Among them, the specific method for obtaining the artifact detection effect index is as follows: Identify the artifact area of the enhanced image data through edge detection and number it, obtain the number of pixels and the corresponding pixel brightness values within the artifact area corresponding to the number; perform statistical analysis on the pixel brightness values to obtain the brightness-related value of the artifact area, obtain the direction gradient of the enhanced image data, where the direction gradient includes the horizontal direction gradient and the vertical direction gradient, and the brightness-related values include the maximum pixel brightness, the minimum pixel brightness, and the average pixel brightness; combine the number of pixels, pixel brightness values, brightness-related values, and direction gradient of the enhanced image data with the evaluation weights obtained from the preset database to obtain the artifact detection effect index, and the evaluation weights include the artifact intensity weight, the artifact contrast weight, and the edge sharpness weight.

[0040] In this embodiment, the edges in the image are detected by the Sobel operator or the Canny algorithm for edge detection, and image morphological operations (such as dilation and erosion) are used to enhance the edges, and each identified contour is numbered to obtain the number of the artifact area; the direction gradient of the enhanced image data is obtained by the gradient method; among them, for statistical analysis of the pixel brightness values, the maximum pixel brightness can be calculated by using the max function of numpy, the minimum pixel brightness can be calculated by using the min function of numpy, and the average pixel brightness can be calculated by using the mean function of numpy.

[0041] Among them, the artifact intensity weight is used to describe the influence degree of the artifact intensity on the artifact detection effect index. Specifically, the artifact intensity weight is the weight corresponding to the artifact intensity in the preset artifact detection in the preset database, which represents the numerical value of the influence degree of the artifact intensity on the artifact detection effect index. When used, the preset weight corresponding to the artifact intensity weight can be directly obtained from the preset database, and its corresponding relationship can be a pre-set mapping relationship. For example, the artifact intensity in the artifact detection and the weight corresponding to the preset artifact intensity in the preset database form a mapping set, and the artifact intensity in the real-time artifact detection is input into the mapping set to obtain the corresponding weight, and the mapping relationship therein can be a one-to-one or many-to-one relationship. In this example, the value range is [0, 1].

[0042] The artifact contrast weight is used to describe the influence degree of the artifact contrast on the artifact detection effect index. Specifically, the artifact contrast is the weight corresponding to the artifact contrast in the preset artifact detection in the preset database, which represents the numerical value of the influence degree of the artifact contrast in the artifact detection on the artifact detection effect index. When in use, the weight corresponding to the preset artifact contrast can be directly obtained from the preset database, and its corresponding relationship can be a preset mapping relationship. For example, the artifact contrast in the artifact detection and the weight corresponding to the preset artifact contrast in the preset database form a mapping set, and the artifact contrast in the real-time artifact detection is input into the mapping set to obtain the corresponding weight. The mapping relationship therein can be a one-to-one or many-to-one relationship. In this example, the value range is [0, 1].

[0043] The edge sharpness weight is used to describe the influence degree of the edge sharpness on the artifact detection effect index. In this example, the sum of the artifact intensity weight, the artifact contrast weight, and the edge sharpness weight is 1, which realizes a more accurate evaluation of the artifact detection.

[0044] It should be understood that the artifact detection effect index is calculated using the following formula:

[0045]

[0046] In the formula, W represents the artifact detection effect index, i represents the pixel number within the artifact region, i = 1, 2,..., N, N represents the total number of pixels within the artifact region, j represents the artifact region number of the enhanced image data, j = 1, 2,..., M, and M represents the total number of artifact regions of the enhanced image data. represents the pixel brightness value of the i-th pixel in the j-th artifact region, I max represents the maximum pixel brightness, I min represents the minimum pixel brightness, I a represents the average pixel brightness, I(x, y) represents the pixel value of the enhanced image data at the coordinate (x, y). represents the horizontal direction gradient, represents the vertical direction gradient, c1 represents the artifact intensity weight, c2 represents the artifact contrast weight, c3 represents the edge sharpness weight, ΔA represents the pixel brightness reference value, ΔB represents the artifact contrast reference value, and ΔC represents the edge sharpness reference value.

[0047] Such as Figure 2As shown in the figure, it is a schematic diagram of the change of the artifact detection effect index provided by the embodiment of the present application. Among them, the pixel brightness difference is the difference between the maximum pixel brightness and the minimum pixel brightness. It is set that: the total number of pixels in the artifact area is 5, the total number of artifact areas in the enhanced image data is 3, the artifact intensity weight is 0.5, the artifact contrast weight is 0.3, the edge sharpness weight is 0.2, the horizontal gradient is 2, the vertical gradient is 2, and the pixel brightness values in all artifact areas are 10 candela per square meter. The pixel brightness reference value, the artifact contrast reference value, and the edge sharpness reference value are from a preset database; the difference between the maximum pixel brightness and the minimum pixel brightness in the artifact area can reflect the contrast of the artifact area. As can be seen from the figure, as the difference between the maximum pixel brightness and the minimum pixel brightness increases, the artifact detection effect index increases accordingly, and the curve shows a gradually rising trend, especially in the direction where the difference between the maximum pixel brightness and the minimum pixel brightness increases, and there will be a downward trend in the direction where the average pixel brightness increases.

[0048] Through the correlation and mutual influence between the artifact detection effect index and the difference between the maximum pixel brightness and the minimum pixel brightness and the average pixel brightness, the artifact detection effect index is comprehensively analyzed, and the influence degree of the artifact on the enhanced image data can be accurately quantified; the artifact detection effect index within the preset detection threshold range indicates that the effect of artifact detection on the enhanced image data is relatively good; the pixel brightness reference value, the artifact contrast reference value, and the edge sharpness reference value are used as the denominator. When the ratio of the numerator of the above formula to the corresponding reference value tends to 1, the artifact detection effect index reaches the optimal value; among them, the artifact detection effect index includes parameters in multiple aspects, and there is a connection between the parameters, and they do not exist independently, and cannot be obtained by simple combination addition; for example, the edge sharpness will affect the effect of artifact detection, but it does not necessarily mean that the artifact detection effect index is high. At this time, the artifact contrast situation also needs to be considered; among them, when the average pixel brightness is a certain value, as the difference between the maximum pixel brightness and the minimum pixel brightness increases, the artifact detection effect index increases accordingly; high contrast can enhance the visibility of the artifact in the image, make the boundary of the artifact more obvious, and thus make it easier to detect and analyze. At the same time, as the pixel brightness value increases, the artifact detection effect index increases accordingly, realizing the numerical evaluation of the influence degree of the artifact on the enhanced image data.

[0049] Among them, the specific steps for obtaining enhanced image data for image registration are as follows: Obtain enhanced image data and adjacent enhanced image data within a preset time interval and perform feature extraction to obtain invariant feature points. The adjacent enhanced image data is used to describe the enhanced image data of adjacent preset time intervals. The invariant feature points include enhanced invariant feature points and adjacent invariant feature points; perform feature matching on the invariant feature points through fast approximate nearest neighbor search to obtain matching feature points, process the matching feature points through the least squares method to obtain a transformation matrix, and perform image alignment on the transformation matrix to obtain aligned image data; perform image fusion on the aligned image data and the enhanced image data to obtain registered image data.

[0050] In this embodiment, scale-invariant feature transform is used to perform feature extraction to obtain invariant feature points. Specifically, in order to make the feature points have rotational invariance, obtain the gradient direction histogram within the neighborhood of each feature point; the invariant feature points are used to describe the feature points that are invariant in scale and rotation for both the enhanced image data and the adjacent enhanced image data; among them, the matching feature points are processed through the least squares method to obtain a transformation matrix. Specifically, construct a linear equation system for the matching feature points and write it in matrix form and solve it to obtain the transformation matrix; among them, perform image alignment on the transformation matrix through the transformation function in the image processing library to obtain aligned image data. Specifically, use the transformation function (such as the affine transformation function) to align the enhanced image data to the coordinate system of the adjacent enhanced image data; among them, perform image fusion on the aligned image data and the enhanced image data through an image fusion algorithm to obtain registered image data. Specifically, the pixel value of the corresponding registered image data pixel point can be obtained by selecting the maximum or minimum value of the pixels in the enhanced image data and the adjacent enhanced image data for each pixel point and obtaining the average value of the maximum and minimum values of the pixels; the improvement of the image registration quality of the enhanced image data is realized.

[0051] Among them, the specific method for obtaining the registration quality evaluation index is as follows: Obtain the total number of pixels of the registered image data and the image pixel gray values. The image pixel gray values include the gray values of the registered image data and the enhanced image data; obtain the average image gray value according to the gray values of the registered image data and the enhanced image data at the corresponding pixels. The average image gray value includes the average gray value of the registered image data and the average gray value of the enhanced image data; combine the total number of pixels of the registered image data, the image pixel gray values, the average image gray value with the reference gray values and evaluation weights obtained from the preset database to obtain the registration quality evaluation index. The evaluation weights include the difference degree weight and the cross-correlation degree weight, and the reference gray values include the reference gray value of the registered image data and the reference gray value of the enhanced image data.

[0052] In this embodiment, the result of summing and averaging the gray values of the collected historical registered image data represents the reference gray value of the registered image data, and the result of summing and averaging the gray values of the collected historical enhanced image data represents the reference gray value of the enhanced image data.

[0053] Among them, the difference degree weight is used to describe the influence degree of the difference degree between the enhanced image data and the registered image data on the registration quality evaluation index. Specifically, the difference degree weight is the weight corresponding to the difference degree between the enhanced image data and the registered image data preset in the preset database, which represents the numerical value of the influence degree of the difference degree between the enhanced image data and the registered image data on the registration quality evaluation index. When in use, the weight corresponding to the difference degree between the enhanced image data and the registered image data preset in the preset database can be directly obtained. The corresponding relationship can be a preset mapping relationship. For example, the difference degree between the enhanced image data and the registered image data and the weight corresponding to the difference degree between the enhanced image data and the registered image data preset in the preset database form a mapping set. The difference degree between the real-time enhanced image data and the registered image data is input into the mapping set to obtain the corresponding weight. The mapping relationship therein can be a one-to-one or many-to-one relationship. In this example, the value range is [0, 1].

[0054] The cross-correlation degree weight is used to describe the influence degree of the cross-correlation degree between the enhanced image data and the registered image data on the registration quality evaluation index. In this example, the sum of the difference degree weight and the cross-correlation degree weight is 1, realizing the accurate evaluation of the quality alignment degree of the registered image data during the registration process.

[0055] Among them, the registration quality evaluation index is calculated using the following formula:

[0056]

[0057] In the formula, Z represents the registration quality evaluation index, q represents the pixel number of the registered image data, q = 1, 2,..., Q, and Q represents the total number of pixels of the registered image data. represents the gray value of the registered image data of the qth pixel. represents the gray value of the enhanced image data of the qth pixel. represents the average gray value of the registered image data. represents the average gray value of the enhanced image data, Δp represents the reference gray value of the registered image data, Δh represents the reference gray value of the enhanced image data, α represents the difference degree weight, β represents the cross-correlation degree weight, ΔM represents the reference value of the registration difference, and ΔN represents the reference value of the registration cross degree.

[0058] Specific assumptions: The total number of pixels in the registered image data is 1, the average gray value of the registered image data is 10, the average gray value of the enhanced image data is 20, the reference gray value of the registered image data is 10, the reference gray value of the enhanced image data is 20, the weight of the difference degree is 0.5, the weight of the cross-correlation degree is 0.5, define The reference value of the registration difference is 1, and the reference value of the registration cross degree is 1. Among them, I C is the square term of the gray value difference. The change statistical table of the registration quality evaluation index is shown in Table 1:

[0059] Table 1 Change Statistical Table of Registration Quality Evaluation Index

[0060]

[0061]

[0062] The registration quality evaluation index is used to evaluate the quality alignment degree of the registered image data during the registration process. It can be seen from Table 1 that as the square of the difference between the gray value of the registered image data and the gray value of the enhanced image data increases, the impact on the registration quality evaluation index also increases. Therefore, the registration quality evaluation index within the registration threshold range indicates that the quality alignment degree of the registered image data during the registration process is relatively good, and the registration quality evaluation index outside the range indicates that there are abnormalities in the registered image data during the registration process. The range of the preset registration threshold is set by professionals according to the standards in the field. Among them, the reference value of the registration difference and the reference value of the registration cross degree are used as the denominators. When the ratio of the numerator of the above formula to the corresponding reference value tends to 1, the artifact detection effect index reaches the optimal value. Among them, the registration quality evaluation index combines the difference and correlation of the image gray value to quantify the registration effect, and realizes the numerical evaluation of the quality alignment degree of the registered image data during the registration process.

[0063] Among them, the specific steps to judge whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation results are as follows: Obtain the registration quality evaluation index and the preset registration threshold range, compare the registration quality evaluation index with the preset registration threshold range. If the registration quality evaluation index is within the preset registration threshold range, perform dynamic three-dimensional reconstruction; otherwise, feedback the registration quality evaluation index to remind the preset personnel to replace the registration technology and perform the registration quality evaluation of the image again to obtain the registration quality evaluation index.

[0064] In this embodiment, the process of comprehensive evaluation according to the registration quality evaluation index and the preset registration threshold range is as follows: compare the registration quality evaluation index with the preset registration threshold range. If the registration quality evaluation index is within the preset registration threshold range, perform dynamic three-dimensional reconstruction. If the registration quality evaluation index exceeds the preset registration threshold range, trigger the warning mechanism system to give feedback through a light warning. After the preset personnel receive the feedback and take measures, they need to re-evaluate the registration quality of the image. If the registration quality evaluation index obtained by the re-registration is within the preset registration threshold range, dynamic three-dimensional reconstruction can continue; if it still does not meet the preset registration threshold range, trigger the warning mechanism again until the registration quality evaluation index is within the preset registration threshold range, thereby improving the reliability of the real-time dynamic three-dimensional image reconstruction result.

[0065] As Figure 3 shown, it is a schematic structural diagram of a three-dimensional reconstruction system for minimally invasive partial nephrectomy binocular video provided by an embodiment of the present application. The three-dimensional reconstruction system for minimally invasive partial nephrectomy binocular video provided by an embodiment of the present application includes: an image data acquisition module, an artifact detection and evaluation module, an image registration evaluation module, and a comprehensive evaluation and judgment module. Among them, the image data acquisition module is used to obtain a preliminary three-dimensional reconstruction result based on the original image data, obtain real-time image data within a preset time interval during the operation, and perform image processing on the real-time image data to obtain enhanced image data. The original image data is obtained from the binocular video. The artifact detection and evaluation module is used to perform artifact detection and evaluation on the enhanced image data to obtain an artifact detection effect index, and judge whether to perform image registration based on the artifact detection effect index and the preset detection threshold range. If not, give an artifact detection feedback reminder to the preset personnel to re-obtain the real-time image data. The artifact detection effect index is used to quantify the influence degree of artifacts on the enhanced image data. The image registration evaluation module is used to obtain the registered image data after image registration of the enhanced image data, and evaluate the registration quality of the image registration according to the enhanced image data and the registered image data to obtain a registration quality evaluation index. The registration quality evaluation index is used to describe the quality alignment degree of the registered image data during the registration process. The comprehensive evaluation and judgment module is used to perform an evaluation according to the registration quality evaluation index and the preset registration threshold range to obtain a comprehensive evaluation result, and judge whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result. The comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

[0066] In this embodiment, the image data acquisition module includes obtaining a preliminary three-dimensional reconstruction result, obtaining real-time image data, and image processing to meet the requirements for subsequent evaluation; the artifact detection and evaluation module includes generating an artifact detection effect index through artifact detection and evaluation and determining whether to perform image registration to detect and evaluate artifacts in the image; the image registration evaluation module includes obtaining registered image data and generating a registration quality evaluation index to evaluate the quality of the registered image; the comprehensive evaluation and determination module is used to generate a comprehensive evaluation result and determine whether to perform a final dynamic three-dimensional reconstruction; the quality of real-time image data during the operation is improved, thereby improving the accuracy of performing dynamic three-dimensional reconstruction.

[0067] The following points need to be explained:

[0068] (1) The drawings of the embodiments of the present invention only relate to the structures involved in the embodiments of the present invention, and other structures can refer to the general design.

[0069] (2) For clarity, in the drawings used to describe the embodiments of the present invention, the thickness of the layer or region is enlarged or reduced, that is, these drawings are not drawn according to the actual scale. It can be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element can be "directly" on or under the other element or there can be an intermediate element.

[0070] (3) Without conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0071] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A three-dimensional reconstruction method for binocular video of minimally invasive partial nephrectomy, characterized in that: The following steps are involved: S1, obtaining a preliminary three-dimensional reconstruction result based on the original image data, obtaining real-time image data within a preset time interval during the operation, and performing image processing on the real-time image data to obtain enhanced image data, wherein the original image data is obtained from the binocular video; S2, performing artifact detection evaluation on the enhanced image data to obtain an artifact detection effect index, and judging whether to perform image registration based on the artifact detection effect index and a preset detection threshold range, and if not, performing artifact detection feedback to remind the preset personnel to reacquire real-time image data, wherein the artifact detection effect index is used to quantify the influence of artifacts on the enhanced image data; The artifact detection effect index is calculated using the following formula: ； In the formula, Represents the artifact detection effect index, represents the pixel number within the artifact area, , represents the total number of pixels in the artifact area, Indicates the artifact area number of the enhanced image data, , Represents the total number of artifact areas in the enhanced image data, Indicates The pixel brightness value of the i-th pixel in the artifact area, Indicates the maximum pixel brightness. Indicates the minimum pixel brightness. represents the average pixel brightness. Indicates the enhanced image data at coordinates The pixel value at represents the horizontal gradient, represents the vertical gradient, represents the artifact intensity weight, represents the artifact contrast weight, represents the edge sharpness weight, Indicates the pixel brightness reference value, represents the artifact contrast reference value, Indicates the edge sharpness reference value; S3, after performing image registration, obtaining registered image data after image registration of the enhanced image data, evaluating the registration quality of the image registration according to the enhanced image data and the registered image data to obtain a registration quality evaluation index, wherein the registration quality evaluation index is used to describe the quality alignment degree of the registered image data during the registration process; S4, performing evaluation according to the registration quality evaluation index and the preset registration threshold range to obtain a comprehensive evaluation result, and judging whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result, wherein the comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

2. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The obtaining of the preliminary three-dimensional reconstruction result based on the original image data also includes the following steps: Acquire the patient's preoperative imaging data and import it into the three-dimensional reconstruction software, wherein the preoperative imaging data is stored in DICOM format; The kidney and its surrounding structures are identified by using an automatic segmentation algorithm to process the preoperative image data to obtain initial image data, and the initial image data is processed by Gaussian filtering to obtain original image data.

3. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The specific steps of obtaining the enhanced image data are as follows: The real-time image data is smoothed by Gaussian filtering to obtain smoothed image data, and the smoothed image data is edge processed to obtain first enhanced image data; Performing high frequency component enhancement processing on the first enhanced image data to obtain second enhanced image data; The second enhanced image data is subjected to histogram equalization processing to obtain enhanced image data.

4. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The specific method for obtaining the artifact detection effect index is as follows: Identify and number the artifact area of ​​the enhanced image data through edge detection, and obtain the number of pixels in the artifact area with the corresponding number and the corresponding pixel brightness value; Performing statistical analysis on the pixel brightness values ​​to obtain brightness correlation values ​​of the artifact area, and obtaining directional gradients of the enhanced image data, wherein the directional gradients include horizontal directional gradients and vertical directional gradients, and the brightness correlation values ​​include maximum pixel brightness, minimum pixel brightness, and average pixel brightness; The number of pixels in the artifact area, the pixel brightness value, the brightness correlation value, the directional gradient of the enhanced image data are combined with the evaluation weights obtained from the preset database to obtain the artifact detection effect index, wherein the evaluation weights include the artifact intensity weight, the artifact contrast weight and the edge sharpness weight.

5. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The specific steps of obtaining enhanced image data for image registration are as follows: Acquire enhanced image data and adjacent enhanced image data within a preset time interval and perform feature extraction to obtain invariant feature points, wherein the adjacent enhanced image data is used to describe enhanced image data of adjacent preset time intervals, and the invariant feature points include enhanced invariant feature points and adjacent invariant feature points; Perform feature matching on the invariant feature points to obtain matching feature points, process the matching feature points by the least square method to obtain a transformation matrix, and perform image alignment on the transformation matrix to obtain aligned image data; The aligned image data and the enhanced image data are fused to obtain the registered image data.

6. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 5, characterized in that: The specific method for obtaining the registration quality evaluation index is as follows: Acquire the total number of pixels of the registered image data and the grayscale value of the image pixels, wherein the grayscale value of the image pixels includes the grayscale value of the registered image data and the grayscale value of the enhanced image data; Obtaining an average grayscale value of the image according to the grayscale values ​​of the registered image data and the enhanced image data at corresponding pixels, wherein the average grayscale value of the image includes the average grayscale value of the registered image data and the average grayscale value of the enhanced image data; The total number of pixels of the registered image data, the image pixel grayscale value, the image average grayscale value, the reference grayscale value obtained from the preset database and the evaluation weight are combined to obtain the registration quality evaluation index, wherein the evaluation weight includes the difference degree weight and the cross-correlation degree weight, and the reference grayscale value includes the reference grayscale value of the registered image data and the reference grayscale value of the enhanced image data.

7. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 6, characterized in that: The registration quality evaluation index is calculated using the following formula: ； In the formula, represents the registration quality assessment index, represents the pixel number of the registered image data, , Represents the total number of pixels of the registered image data, represents the gray value of the registered image data of the qth pixel, represents the gray value of the enhanced image data of the qth pixel, represents the average gray value of the registered image data, represents the average gray value of enhanced image data, Represents the reference gray value of the registered image data, Represents the reference gray value of enhanced image data, represents the difference degree weight, represents the cross-correlation weight, represents the registration difference reference value, Indicates the reference value of the registration cross degree.

8. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The specific steps of determining whether to perform dynamic 3D reconstruction based on the comprehensive evaluation results are as follows: Obtaining a registration quality evaluation index and a preset registration threshold range, comparing the registration quality evaluation index with the preset registration threshold range, and performing dynamic three-dimensional reconstruction if the registration quality evaluation index is within the preset registration threshold range; Otherwise, the registration quality assessment index is fed back to remind the preset personnel to change the registration technology and perform the image registration quality assessment again to obtain the registration quality assessment index.

9. A three-dimensional reconstruction system for binocular video of minimally invasive partial nephrectomy, the system being used to implement the method according to any one of claims 1 to 8, characterized in that: The system comprises: an image data acquisition module, an artifact detection and evaluation module, an image registration and evaluation module, and a comprehensive evaluation and judgment module; The image data acquisition module is used to acquire a preliminary three-dimensional reconstruction result based on the original image data, acquire real-time image data within a preset time interval during the operation, and perform image processing on the real-time image data to obtain enhanced image data, wherein the original image data is acquired from the binocular video; The artifact detection and evaluation module is used to perform artifact detection and evaluation on the enhanced image data to obtain an artifact detection effect index, and determine whether to perform image registration based on the artifact detection effect index and a preset detection threshold range. If not, artifact detection feedback is performed to remind the preset personnel to re-acquire real-time image data. The artifact detection effect index is used to quantify the degree of influence of artifacts on the enhanced image data; The image registration evaluation module is used to obtain the registered image data after the enhanced image data is registered after the image registration is performed, and to evaluate the registration quality of the image registration according to the enhanced image data and the registered image data to obtain a registration quality evaluation index, wherein the registration quality evaluation index is used to describe the quality alignment degree of the registered image data during the registration process; The comprehensive evaluation judgment module is used to obtain a comprehensive evaluation result based on the alignment quality evaluation index and the preset alignment threshold range, and to judge whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result. The comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

Images