Three-dimensional reconstruction method and system for minimally invasive partial kidney resection binocular video

Abstract

The invention provides a three-dimensional reconstruction method and system for a minimally invasive partial kidney resection binocular video, and relates to the technical field of medical image processing. The three-dimensional reconstruction method for the minimally invasive partial kidney resection binocular video comprises the following steps of image data acquisition, artifact detection evaluation, image registration evaluation and comprehensive evaluation judgment. According to the method, the preliminary three-dimensional reconstruction result and the real-time image data in the operation are obtained, the real-time image data are subjected to image processing to obtain the enhanced image data, then the enhanced image data are subjected to artifact detection evaluation, and whether image registration is executed or not is judged; then the registration quality of image registration is evaluated to obtain a comprehensive evaluation result, and whether dynamic three-dimensional reconstruction is carried out or not is judged, so that the effect of improving the real-time dynamic three-dimensional image reconstruction precision in the peritoneal kidney partial resection is achieved; the problem that in the prior art, real-time dynamic three-dimensional image reconstruction precision is not high in the peritoneal kidney partial resection is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a three-dimensional reconstruction method and system for binocular video of minimally invasive partial nephrectomy. Background Art

[0002] Laparoscopic partial nephrectomy is a common minimally invasive surgical method, mainly used to treat localized renal tumors. Compared with traditional open surgery, laparoscopic surgery has the advantages of less trauma, faster recovery, and fewer complications. However, due to the complexity and delicacy of the operation, the surgical process requires the help of high-quality image guidance to ensure the accuracy and safety of the operation.

[0003] The existing dynamic 3D reconstruction for abdominal nephrectomy images uses multiple imaging devices (such as ultrasound) to simultaneously or alternately obtain the image data of the patient's abdominal cavity, and uses high-frame-rate cameras or sensors to capture the possible deformation and movement of the kidney and its surrounding tissues during the operation. Based on the multimodal image data, computer vision and graphics algorithms are used to perform 3D reconstruction to form an accurate model of the kidney and surrounding tissues. By updating the 3D model in real time, the changes in tissues during the operation are reflected, allowing doctors to intuitively see the dynamic changes in the surgical area.

[0004] Dynamic 3D reconstruction involves multimodal image data acquisition, high-performance computing, and advanced graphics processors. Real-time updating of 3D models requires extremely high computing resources and efficiency. Especially when processing large-scale data or complex surgical scenarios, delays or reduced accuracy may occur.

[0005] In the prior art, dynamic 3D reconstruction generates a 3D model of an organ or structure through continuously acquired image data. Non-real signals or distortion introduced in medical images due to various reasons (such as metal implants, etc.) may affect the clarity and accuracy of the image, thereby affecting the accuracy of dynamic 3D reconstruction during surgery. During laparoscopic surgery, especially partial nephrectomy, there is a problem of low accuracy of real-time dynamic 3D image reconstruction for abdominal partial nephrectomy. Summary of the invention

[0006] The present invention provides a three-dimensional reconstruction method and system for binocular video of minimally invasive partial nephrectomy, which solves the problem of low accuracy of real-time dynamic three-dimensional image reconstruction in abdominal partial nephrectomy in the prior art, and improves the accuracy of real-time dynamic three-dimensional image reconstruction in abdominal partial nephrectomy. The technical solution is as follows:

[0007] A three-dimensional reconstruction method for binocular video of minimally invasive partial nephrectomy, comprising the following steps: S1, obtaining a preliminary three-dimensional reconstruction result based on original image data, obtaining real-time image data within a preset time interval during surgery, performing image processing on the real-time image data to obtain enhanced image data, wherein the original image data is obtained from the binocular video; S2, performing artifact detection evaluation on the enhanced image data to obtain an artifact detection effect index, judging whether to perform image registration based on the artifact detection effect index and a preset detection threshold range, and if not, performing artifact detection feedback to remind a preset person to re-acquire real-time image data, wherein the artifact detection effect index The number is used to quantify the influence of the artifact on the enhanced image data; S3, after performing image registration, the registered image data after the enhanced image data is registered is obtained, and the registration quality of the image registration is evaluated according to the enhanced image data and the registered image data to obtain a registration quality evaluation index, and the registration quality evaluation index is used to describe the quality alignment of the registered image data during the registration process; S4, according to the registration quality evaluation index and the preset registration threshold range, an evaluation is performed to obtain a comprehensive evaluation result, and based on the comprehensive evaluation result, it is determined whether to perform dynamic three-dimensional reconstruction, and the comprehensive evaluation result is used to determine whether to perform dynamic three-dimensional reconstruction.

[0008] Optionally, the method of obtaining a preliminary three-dimensional reconstruction result based on the original image data also includes the following steps: obtaining the patient's preoperative image data and importing it into the three-dimensional reconstruction software, wherein the preoperative image data is stored in a DI COM format; identifying the kidney and its surrounding structures by processing the preoperative image data using an automatic segmentation algorithm to obtain initial image data, and performing Gaussian filtering on the initial image data to obtain the original image data.

[0009] Optionally, the specific steps for acquiring the enhanced image data are as follows: smoothing the real-time image data through Gaussian filtering to obtain smoothed image data, performing edge processing on the smoothed image data to obtain first enhanced image data; performing high-frequency component enhancement processing on the first enhanced image data to obtain second enhanced image data; and performing histogram equalization processing on the second enhanced image data to obtain enhanced image data.

[0010] Optionally, a specific method for obtaining the artifact detection effect index is as follows: identifying the artifact area of ​​the enhanced image data through edge detection and numbering it, obtaining the number of pixels in the corresponding numbered artifact area and the corresponding pixel brightness value; performing statistical analysis on the pixel brightness values ​​to obtain the brightness-related values ​​of the artifact area, and obtaining the directional gradient of the enhanced image data, the directional gradient includes the horizontal gradient and the vertical gradient, and the brightness-related values ​​include the maximum pixel brightness, the minimum pixel brightness and the average pixel brightness; combining the number of pixels in the artifact area, the pixel brightness value, the brightness-related value, and the directional gradient of the enhanced image data with the evaluation weights obtained from a preset database to obtain the artifact detection effect index, the evaluation weights include the artifact intensity weight, the artifact contrast weight and the edge sharpness weight.

[0011] Optionally, the artifact detection effect index is calculated using the following formula:

[0012]

[0013] Where W represents the artifact detection effect index, i represents the pixel number in the artifact area, i = 1, 2, ..., N, N represents the total number of pixels in the artifact area, j represents the artifact area number of the enhanced image data, j = 1, 2, ..., M, M represents the total number of artifact areas of the enhanced image data, represents the pixel brightness value of the i-th pixel in the j-th artifact area, I max Indicates the maximum pixel brightness, I min Indicates the minimum pixel brightness, I a represents the average pixel brightness, I(x,y) represents the pixel value of the enhanced image data at the coordinate (x,y), represents the horizontal gradient, represents the vertical gradient, c1 represents the artifact intensity weight, c2 represents the artifact contrast weight, c3 represents the edge sharpness weight, ΔA represents the pixel brightness reference value, ΔB represents the artifact contrast reference value, and ΔC represents the edge sharpness reference value.

[0014] Optionally, the specific steps of obtaining enhanced image data for image registration are as follows: obtaining enhanced image data and adjacent enhanced image data within a preset time interval and performing feature extraction to obtain invariant feature points, wherein the adjacent enhanced image data is used to describe the enhanced image data of adjacent preset time intervals, and the invariant feature points include enhanced invariant feature points and adjacent invariant feature points; performing feature matching on the invariant feature points to obtain matching feature points, processing the matching feature points through the least squares method to obtain a transformation matrix, and performing image alignment on the transformation matrix to obtain aligned image data; performing image fusion on the aligned image data and the enhanced image data to obtain registered image data.

[0015] Optionally, the specific method for obtaining the registration quality evaluation index is as follows: obtain the total number of pixels and image pixel grayscale values ​​of the registered image data, the image pixel grayscale values ​​include the registered image data grayscale values ​​and the enhanced image data grayscale values; obtain the image average grayscale value according to the grayscale values ​​of the registered image data and the enhanced image data at corresponding pixels, the image average grayscale value includes the registered image data average grayscale value and the enhanced image data average grayscale value; combine the total number of pixels of the registered image data, the image pixel grayscale values, the image average grayscale value with the reference grayscale value and evaluation weights obtained from a preset database to obtain the registration quality evaluation index, the evaluation weights include the difference degree weights and the cross-correlation degree weights, and the reference grayscale values ​​include the registered image data reference grayscale values ​​and the enhanced image data reference grayscale values.

[0016] Optionally, the registration quality evaluation index is calculated using the following formula:

[0017]

[0018] Where Z represents the registration quality evaluation index, q represents the pixel number of the registered image data, q = 1, 2, ..., Q, Q represents the total number of pixels of the registered image data, represents the gray value of the registered image data of the qth pixel, represents the gray value of the enhanced image data of the qth pixel, represents the average gray value of the registered image data, represents the average gray value of enhanced image data, Δp represents the reference gray value of registered image data, Δh represents the reference gray value of enhanced image data, α represents the difference degree weight, β represents the cross-correlation degree weight, ΔM represents the registration difference reference value, and ΔN represents the registration cross degree reference value.

[0019] Optionally, the specific steps for determining whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation results are as follows: obtain a registration quality evaluation index and a preset registration threshold range, compare the registration quality evaluation index with the preset registration threshold range, and if the registration quality evaluation index is within the preset registration threshold range, perform dynamic three-dimensional reconstruction; otherwise, feed back the registration quality evaluation index to remind the preset personnel to change the registration technology and perform image registration quality evaluation again to obtain the registration quality evaluation index.

[0020] The present invention provides a three-dimensional reconstruction system for binocular video of minimally invasive partial nephrectomy, comprising: an image data acquisition module, an artifact detection and evaluation module, an image registration evaluation module, and a comprehensive evaluation and judgment module; wherein the image data acquisition module is used to acquire a preliminary three-dimensional reconstruction result based on original image data, acquire real-time image data within a preset time interval during surgery, and perform image processing on the real-time image data to obtain enhanced image data, wherein the original image data is acquired from the binocular video; the artifact detection and evaluation module is used to perform artifact detection and evaluation on the enhanced image data to obtain an artifact detection effect index, and judge whether to perform image registration based on the artifact detection effect index and a preset detection threshold range, and if not, perform artifact detection feedback to remind preset personnel Reacquire real-time image data, and the artifact detection effect index is used to quantify the influence of artifacts on enhanced image data; the image registration evaluation module is used to obtain the registered image data after the enhanced image data is registered after image registration, and evaluate the registration quality of the image registration according to the enhanced image data and the registered image data to obtain the registration quality evaluation index, and the registration quality evaluation index is used to describe the quality alignment of the registered image data during the registration process; the comprehensive evaluation judgment module is used to evaluate according to the registration quality evaluation index and the preset registration threshold range to obtain a comprehensive evaluation result, and judge whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result, and the comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

[0021] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0022] 1. Enhanced image data is obtained by performing image processing on real-time image data, and then artifact detection and evaluation are performed on the enhanced image data, and whether to perform image registration is determined based on the artifact detection effect index and the preset detection threshold range. If the image registration criteria are executed, the image registration quality is evaluated to obtain a comprehensive evaluation result, and finally, based on the comprehensive evaluation result, it is determined whether to perform dynamic three-dimensional reconstruction, thereby realizing artifact detection and image registration evaluation of real-time image data during surgery, and further realizing the improvement of the accuracy of real-time dynamic three-dimensional image reconstruction during abdominal partial nephrectomy, and effectively solving the problem of low accuracy of real-time dynamic three-dimensional image reconstruction during abdominal partial nephrectomy in the prior art.

[0023] 2. Identify and number the artifact areas of the enhanced image data through edge detection, obtain the number of pixels in the corresponding numbered artifact area and the corresponding pixel brightness value, perform statistical analysis on the pixel brightness value to obtain the brightness-related value of the artifact area and obtain the directional gradient of the enhanced image data, combine the number of pixels in the artifact area, the pixel brightness value, the brightness-related value, and the directional gradient of the enhanced image data with the evaluation weight obtained from the preset database to obtain the artifact detection effect index, thereby realizing the quantification of the degree of influence of the artifact on the enhanced data, and further realizing a more accurate evaluation of the degree of influence of the artifact on the enhanced image data.

[0024] 3. By obtaining the total number of pixels and image pixel grayscale values ​​of the registered image data, and obtaining the image average grayscale value according to the grayscale values ​​of the registered image data and the enhanced image data at the corresponding pixels, the total number of pixels of the registered image data, the image pixel grayscale value, the image average grayscale value and the reference grayscale value and evaluation weight obtained from the preset database are combined to obtain the registration quality evaluation index, thereby realizing the digitization of the quality alignment degree of the registered image data during the registration process, and then realizing a more accurate evaluation of the enhanced image data for image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flowchart of a three-dimensional reconstruction method for binocular video of minimally invasive partial nephrectomy provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of changes in the artifact detection effect index provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the structure of a three-dimensional reconstruction system of binocular video for minimally invasive partial nephrectomy provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "one" or "the" do not indicate a quantitative limitation, but indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0031] It should be noted that the terms "up", "down", "left", "right", "front", "back", etc. used in the present invention are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0032] In view of the problem in the prior art that the accuracy of real-time dynamic three-dimensional image reconstruction in abdominal partial nephrectomy is low, the present invention provides a three-dimensional reconstruction method and system for binocular video of minimally invasive partial nephrectomy, thereby improving the accuracy of real-time dynamic three-dimensional image reconstruction in abdominal partial nephrectomy.

[0033] like Figure 1As shown, it is a flow chart of a 3D reconstruction method for binocular video of minimally invasive partial nephrectomy provided by the present invention. The 3D reconstruction method for binocular video of minimally invasive partial nephrectomy provided by the present invention comprises the following steps: S1, obtaining a preliminary 3D reconstruction result based on original image data, obtaining real-time image data within a preset time interval during the operation, performing image processing on the real-time image data to obtain enhanced image data, and the original image data is obtained from the binocular video; S2, performing artifact detection and evaluation on the enhanced image data to obtain an artifact detection effect index, judging whether to perform image registration based on the artifact detection effect index and a preset detection threshold range, and if not, performing artifact detection feedback to remind the preset Suppose personnel reacquire the real-time image data, and the artifact detection effect index is used to quantify the influence of the artifact on the enhanced image data; S3, after performing image registration, obtain the registered image data after the enhanced image data is registered, and evaluate the registration quality of the image registration according to the enhanced image data and the registered image data to obtain the registration quality evaluation index, and the registration quality evaluation index is used to describe the quality alignment of the registered image data during the registration process; S4, evaluate according to the registration quality evaluation index and the preset registration threshold range to obtain a comprehensive evaluation result, and judge whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result, and the comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

[0034] In this embodiment, raw image data is obtained from binocular video through a video processing library. Binocular video refers to two channels of video of the same scene captured simultaneously by two cameras (usually referred to as a left-eye camera and a right-eye camera); whether to perform image registration is determined based on an artifact detection effect index and a preset detection threshold range. Specifically, when the artifact detection effect index exceeds the preset detection threshold range, it is not performed; if the artifact detection effect index is within the preset detection threshold range, it is performed; the raw image data includes computed tomography image data, magnetic resonance imaging image data, and ultrasound image data. The preliminary three-dimensional reconstruction results are used to describe the patient's kidneys and surrounding structures before surgery. The real-time image data includes computed tomography image data, magnetic resonance imaging image data and ultrasound image data of the patient during surgery. Image processing includes edge detection and enhancement, high-frequency component enhancement and histogram equalization. Specifically, edge detection and enhancement first convert the real-time image data into grayscale, and determine the edge through the edge detection algorithm; high-frequency component enhancement converts the image from the spatial domain to the frequency domain by using discrete Fourier transform, and applies a mask to the frequency domain image to enhance the high-frequency component. Histogram equalization refers to mapping pixels to the equalized grayscale; the reliability of images during partial abdominal nephrectomy is improved.

[0035] Among them, obtaining preliminary three-dimensional reconstruction results based on the original image data also includes the following steps: obtaining the patient's preoperative image data and importing it into the three-dimensional reconstruction software (such as RealityCapture), and the preoperative image data is stored in DI COM format; the preoperative image data is processed using an automatic segmentation algorithm to identify the kidneys and their surrounding structures to obtain initial image data, and the initial image data is processed by Gaussian filtering to obtain the original image data.

[0036] In this embodiment, it should be understood that the image data is stored in the DI COM (Digital Imaging and Communications in Medicine) format, which is a standard format in the field of medical imaging; the initial image data is subjected to Gaussian filtering, and the sharp edges of the image are smoothed by a convolution operation, thereby reducing the impact of noise; the kidneys and their surrounding structures are identified by processing the preoperative image data using an automatic segmentation algorithm, for example, commonly used automatic segmentation algorithms include threshold-based segmentation; and the quality of the original image data is improved.

[0037] Among them, the specific steps for obtaining enhanced image data are as follows: smoothing the real-time image data through Gaussian filtering to obtain smoothed image data, edge processing is performed on the smoothed image data to obtain first enhanced image data; high-frequency component enhancement processing is performed on the first enhanced image data to obtain second enhanced image data; and histogram equalization processing is performed on the second enhanced image data to obtain enhanced image data.

[0038] In this embodiment, the real-time image data is smoothed by Gaussian filtering, and the Gaussian kernel is convolved with the image data. The weight of the Gaussian kernel is allocated according to the distance from the center. The closer the distance to the center, the greater the weight, thereby producing a smoothing effect; wherein, commonly used edge detection algorithms include Sobel operator, Canny edge detection, etc.; wherein, high-frequency component enhancement processing is to enhance the performance of the details in the image, especially the edges of small structures. Commonly used techniques are Laplace operator enhancement or nonlinear filtering; wherein, histogram equalization is used to adjust the contrast of the image, and by redistributing the grayscale levels of the image pixels, the distribution of pixels of different grayscale values ​​is made more uniform; and the reliability of the enhanced image data is improved.

[0039] Among them, the specific method for obtaining the artifact detection effect index is as follows: identify the artifact area of ​​the enhanced image data through edge detection and number it, obtain the number of pixels in the corresponding numbered artifact area and the corresponding pixel brightness value; perform statistical analysis on the pixel brightness value to obtain the brightness related value of the artifact area, and obtain the directional gradient of the enhanced image data, the directional gradient includes the horizontal gradient and the vertical gradient, and the brightness related value includes the maximum pixel brightness, the minimum pixel brightness and the average pixel brightness; combine the number of pixels in the artifact area, the pixel brightness value, the brightness related value, and the directional gradient of the enhanced image data with the evaluation weights obtained from the preset database to obtain the artifact detection effect index, and the evaluation weights include the artifact intensity weight, the artifact contrast weight and the edge sharpness weight.

[0040] In this embodiment, the edge in the image is detected by using the edge detection Sobel operator or the Canny algorithm, and the edge is enhanced by using image morphological operations (such as dilation and erosion). Each identified contour is numbered to obtain the number of the artifact area; the directional gradient of the enhanced image data is obtained by the gradient method; wherein, the pixel brightness values ​​are statistically analyzed, the maximum pixel brightness value can be calculated by using the numpy max function, the minimum pixel brightness value can be calculated by using the numpy min function, and the average pixel brightness value can be calculated by using the numpy mean function.

[0041] Among them, the artifact intensity weight is used to describe the influence of the artifact intensity on the artifact detection effect index. Specifically, the artifact intensity weight is the weight corresponding to the artifact intensity in the artifact detection preset in the preset database, which represents the numerical value of the influence of the artifact intensity in the artifact detection on the artifact detection effect index. When used, the weight corresponding to the preset artifact intensity weight can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the artifact intensity in the artifact detection and the weight corresponding to the artifact intensity in the artifact detection preset in the preset database form a mapping set, and the artifact intensity in the real-time artifact detection is input into the mapping set to obtain the corresponding weight. The mapping relationship can be one-to-one or many-to-one. In this example, the value range is [0, 1].

[0042] The artifact contrast weight is used to describe the influence of the artifact contrast on the artifact detection effect index. Specifically, the artifact contrast is the weight corresponding to the artifact contrast in the artifact detection preset in the preset database, which represents the numerical value of the influence of the artifact contrast in the artifact detection on the artifact detection effect index. When used, the weight corresponding to the preset artifact contrast can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the artifact contrast in the artifact detection and the weight corresponding to the artifact contrast in the artifact detection preset in the preset database form a mapping set, and the artifact contrast in the real-time artifact detection is input into the mapping set to obtain the corresponding weight. The mapping relationship can be one-to-one or many-to-one. In this example, the value range is [0, 1].

[0043] The edge sharpness weight is used to describe the influence of edge sharpness on the artifact detection effect index. In this example, the sum of the artifact intensity weight, the artifact contrast weight, and the edge sharpness weight is 1, which achieves a more accurate evaluation of artifact detection.

[0044] It is important to understand that the artifact detection effectiveness index is calculated using the following formula:

[0045]

[0046] Where W represents the artifact detection effect index, i represents the pixel number in the artifact area, i = 1, 2, ..., N, N represents the total number of pixels in the artifact area, j represents the artifact area number of the enhanced image data, j = 1, 2, ..., M, M represents the total number of artifact areas of the enhanced image data, represents the pixel brightness value of the i-th pixel in the j-th artifact area, I max Indicates the maximum pixel brightness, I min Indicates the minimum pixel brightness, I a represents the average pixel brightness, I(x,y) represents the pixel value of the enhanced image data at the coordinate (x,y), represents the horizontal gradient, represents the vertical gradient, c1 represents the artifact intensity weight, c2 represents the artifact contrast weight, c3 represents the edge sharpness weight, ΔA represents the pixel brightness reference value, ΔB represents the artifact contrast reference value, and ΔC represents the edge sharpness reference value.

[0047] like Figure 2As shown, it is a schematic diagram of the change of the artifact detection effect index provided by the embodiment of the present application, wherein the pixel brightness difference is the difference between the maximum pixel brightness and the minimum pixel brightness, wherein it is set that: the total number of pixels in the artifact area is 5, the total number of artifact areas of the enhanced image data is 3, the artifact intensity weight is 0.5, the artifact contrast weight is 0.3, the edge sharpness weight is 0.2, the horizontal gradient is 2, the vertical gradient is 2, the pixel brightness values ​​in all artifact areas are 10 candelas per square meter, the pixel brightness reference value, the artifact contrast reference value and the edge sharpness reference value are from a preset database; the difference between the maximum pixel brightness and the minimum pixel brightness in the artifact area can reflect the contrast of the artifact area. It can be seen from the figure that as the difference between the maximum pixel brightness and the minimum pixel brightness increases, the artifact detection effect index increases accordingly, and the surface shows a gradually rising trend, especially in the direction where the difference between the maximum pixel brightness and the minimum pixel brightness increases, and it will show a downward trend in the direction where the average pixel brightness increases.

[0048] Through the correlation and mutual influence between the artifact detection effect index and the difference between the maximum pixel brightness and the minimum pixel brightness and the average pixel brightness, the artifact detection effect index is obtained through comprehensive analysis, which can accurately quantify the influence of artifacts on enhanced image data; the artifact detection effect index within the preset detection threshold range indicates that the effect of artifact detection on enhanced image data is relatively good; the pixel brightness reference value, the artifact contrast reference value and the edge sharpness reference value are used as the denominator. When the ratio of the numerator of the above formula to the corresponding reference value tends to 1, the artifact detection effect index reaches the optimal value; among them, the artifact detection effect index includes multiple parameters, and the various parameters are related to each other. , does not exist independently, and cannot be obtained by simple combination and addition; for example, edge sharpness affects the effect of artifact detection, which does not necessarily mean that the artifact detection effect index is high. At this time, the artifact contrast situation must also be considered; among them, when the average pixel brightness is a certain value, as the difference between the maximum pixel brightness and the minimum pixel brightness increases, the artifact detection effect index increases accordingly; high contrast can enhance the visibility of artifacts in the image, making the boundaries of artifacts more obvious, making it easier to detect and analyze. At the same time, as the pixel brightness value increases, the artifact detection effect index increases accordingly, realizing the numerical evaluation of the degree of influence of artifacts on enhanced image data.

[0049] Among them, the specific steps of obtaining enhanced image data for image registration are as follows: obtaining enhanced image data and adjacent enhanced image data within a preset time interval and performing feature extraction to obtain invariant feature points, the adjacent enhanced image data is used to describe the enhanced image data of adjacent preset time intervals, and the invariant feature points include enhanced invariant feature points and adjacent invariant feature points; performing feature matching on the invariant feature points through fast approximate nearest neighbor search to obtain matching feature points, processing the matching feature points through the least squares method to obtain a transformation matrix, and performing image alignment on the transformation matrix to obtain aligned image data; performing image fusion on the aligned image data and the enhanced image data to obtain registered image data.

[0050] In this embodiment, feature extraction is performed through scale-invariant feature transformation to obtain invariant feature points. Specifically, in order to make the feature points have rotation invariance, its gradient direction histogram is obtained in the neighborhood of each feature point; the invariant feature points are used to describe the feature points of the enhanced image data and the adjacent enhanced image data whose scale and rotation are invariant; wherein the matching feature points are processed by the least square method to obtain a transformation matrix, specifically, a linear equation system is constructed for the matching feature points and written in matrix form and solved to obtain the transformation matrix; wherein the transformation matrix is ​​image aligned by the transformation function in the image processing library to obtain the aligned image data, specifically, the enhanced image data is aligned to the coordinate system of the adjacent enhanced image data using the transformation function (such as the affine transformation function); wherein the aligned image data and the enhanced image data are image fused by the image fusion algorithm to obtain the registered image data, specifically, the pixel value of the corresponding registered image data pixel point can be obtained by selecting the maximum value or minimum value of the pixel in the enhanced image data and the adjacent enhanced image data for each pixel point and obtaining the average value of the maximum value and the minimum value of the pixel; thereby improving the image registration quality of the enhanced image data.

[0051] Among them, the specific method for obtaining the registration quality evaluation index is as follows: obtain the total number of pixels and image pixel grayscale values ​​of the registered image data, the image pixel grayscale values ​​include the registered image data grayscale values ​​and the enhanced image data grayscale values; obtain the image average grayscale value according to the grayscale values ​​of the registered image data and the enhanced image data at corresponding pixels, the image average grayscale value includes the registered image data average grayscale value and the enhanced image data average grayscale value; combine the total number of pixels of the registered image data, the image pixel grayscale value, the image average grayscale value with the reference grayscale value and evaluation weight obtained from the preset database to obtain the registration quality evaluation index, the evaluation weight includes the difference degree weight and the cross-correlation degree weight, and the reference grayscale value includes the registered image data reference grayscale value and the enhanced image data reference grayscale value.

[0052] In this embodiment, the result of summing and averaging the grayscale values ​​of the collected historical registered image data represents the reference grayscale value of the registered image data, and the result of summing and averaging the grayscale values ​​of the collected historical enhanced image data represents the reference grayscale value of the enhanced image data.

[0053] Among them, the difference degree weight is used to describe the influence of the difference degree between the enhanced image data and the registered image data on the registration quality evaluation index. Specifically, the difference degree weight is the weight corresponding to the difference degree between the enhanced image data and the registered image data preset in the preset database, which represents the numerical value of the influence degree of the difference degree between the enhanced image data and the registered image data on the registration quality evaluation index. When used, the weight corresponding to the difference degree between the preset enhanced image data and the registered image data can be directly obtained from the preset database, and the corresponding relationship can be a preset mapping relationship. For example, the difference degree between the enhanced image data and the registered image data and the weight corresponding to the difference degree between the enhanced image data and the registered image data preset in the preset database form a mapping set, and the difference degree between the real-time enhanced image data and the registered image data is input into the mapping set to obtain the corresponding weight. The mapping relationship can be a one-to-one correspondence or a many-to-one relationship. In this example, the value range is [0, 1].

[0054] The cross-correlation weight is used to describe the influence of the cross-correlation degree between the enhanced image data and the registered image data on the registration quality evaluation index. In this example, the sum of the difference degree weight and the cross-correlation degree weight is 1, which realizes the accurate evaluation of the quality alignment degree of the registered image data during the registration process.

[0055] Among them, the registration quality evaluation index is calculated using the following formula:

[0056]

[0057] Where Z represents the registration quality evaluation index, q represents the pixel number of the registered image data, q = 1, 2, ..., Q, Q represents the total number of pixels of the registered image data, represents the gray value of the registered image data of the qth pixel, represents the gray value of the enhanced image data of the qth pixel, represents the average gray value of the registered image data, represents the average gray value of enhanced image data, Δp represents the reference gray value of registered image data, Δh represents the reference gray value of enhanced image data, α represents the difference degree weight, β represents the cross-correlation degree weight, ΔM represents the registration difference reference value, and ΔN represents the registration cross degree reference value.

[0058] Specific assumptions: the total number of pixels in the registered image data is 1, the average grayscale value of the registered image data is 10, the average grayscale value of the enhanced image data is 20, the reference grayscale value of the registered image data is 10, the reference grayscale value of the enhanced image data is 20, the difference degree weight is 0.5, the cross-correlation degree weight is 0.5, and the definition is The reference value of the registration difference is 1, and the reference value of the registration cross degree is 1, where I C is the square term of gray value difference, and the statistical table of changes in the registration quality evaluation index is shown in Table 1:

[0059] Table 1. Statistics of changes in registration quality evaluation index

[0060]

[0061]

[0062] The registration quality assessment index is used to evaluate the quality alignment of the registered image data during the registration process. As shown in Table 1, as the square of the difference between the grayscale value of the registered image data and the grayscale value of the enhanced image data increases, the impact on the registration quality assessment index also increases. Therefore, the registration quality assessment index within the registration threshold range indicates that the quality alignment of the registered image data during the registration process is relatively good, and the registration quality assessment index beyond the range indicates that the registered image data has an abnormality during the registration process. The range of the preset registration threshold is set by professionals according to the standards in the field, among which the registration difference reference value and the registration cross degree reference value are used as the denominator. When the ratio of the numerator of the above formula to the corresponding reference value approaches 1, the artifact detection effect index reaches the optimal value. Among them, the registration quality assessment index combines the difference and correlation of the image grayscale values ​​to quantify the effect of registration, and realizes the numerical evaluation of the quality alignment of the registered image data during the registration process.

[0063] Among them, the specific steps for judging whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation results are as follows: obtain the registration quality evaluation index and the preset registration threshold range, compare the registration quality evaluation index with the preset registration threshold range, and if the registration quality evaluation index is within the preset registration threshold range, perform dynamic three-dimensional reconstruction; otherwise, feed back the registration quality evaluation index to remind the preset personnel to change the registration technology and perform the image registration quality evaluation again to obtain the registration quality evaluation index.

[0064] In this embodiment, the process of comprehensive evaluation based on the registration quality evaluation index and the preset registration threshold range is to compare the registration quality evaluation index with the preset registration threshold range. If the registration quality evaluation index is within the preset registration threshold range, dynamic three-dimensional reconstruction is performed. If the registration quality evaluation index exceeds the preset registration threshold range, the early warning mechanism system is triggered to provide feedback through a light warning. After receiving the feedback and taking measures, the preset personnel need to re-evaluate the image registration quality. If the registration quality evaluation index obtained by the re-registration is within the preset registration threshold range, dynamic three-dimensional reconstruction can continue. If the preset registration threshold range is still not met, the early warning mechanism is triggered again until the registration quality evaluation index is within the preset registration threshold range, thereby improving the reliability of the real-time dynamic three-dimensional image reconstruction results.

[0065] like Figure 3 As shown, it is a structural schematic diagram of a 3D reconstruction system for binocular video of minimally invasive partial nephrectomy provided by an embodiment of the present application. The 3D reconstruction system for binocular video of minimally invasive partial nephrectomy provided by an embodiment of the present application comprises: an image data acquisition module, an artifact detection and evaluation module, an image registration evaluation module, and a comprehensive evaluation and judgment module; wherein the image data acquisition module is used to acquire a preliminary 3D reconstruction result based on the original image data, acquire real-time image data within a preset time interval during the operation, perform image processing on the real-time image data to obtain enhanced image data, and the original image data is acquired from the binocular video; the artifact detection and evaluation module is used to perform artifact detection and evaluation on the enhanced image data to obtain an artifact detection effect index, and judge whether to execute the artifact detection based on the artifact detection effect index and a preset detection threshold range. Image registration. If it is not performed, artifact detection feedback is performed to remind the preset personnel to re-acquire real-time image data. The artifact detection effect index is used to quantify the influence of artifacts on enhanced image data; the image registration evaluation module is used to obtain the registered image data after the enhanced image data is performed for image registration, and evaluate the registration quality of the image registration according to the enhanced image data and the registered image data to obtain the registration quality evaluation index. The registration quality evaluation index is used to describe the quality alignment of the registered image data during the registration process; the comprehensive evaluation judgment module is used to evaluate according to the registration quality evaluation index and the preset registration threshold range to obtain a comprehensive evaluation result, and judge whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result. The comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

[0066] In this embodiment, the image data acquisition module includes obtaining preliminary three-dimensional reconstruction results, obtaining real-time image data and image processing to meet the needs of subsequent evaluation; the artifact detection and evaluation module includes generating an artifact detection effect index through artifact detection and evaluation and judging whether to perform image registration to detect and evaluate artifacts in the image; the image registration evaluation module includes obtaining registration image data and generating a registration quality evaluation index to perform quality evaluation on the registered image; the comprehensive evaluation judgment module is used to generate a comprehensive evaluation result and judge whether to perform the final dynamic three-dimensional reconstruction; the quality of real-time image data during the operation is improved, thereby improving the accuracy of dynamic three-dimensional reconstruction.

[0067] There are a few points to note:

[0068] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention, and other structures may refer to the general design.

[0069] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of the layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0070] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0071] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A three-dimensional reconstruction method for binocular video of minimally invasive partial nephrectomy, characterized in that: The following steps are involved: S1, obtaining a preliminary three-dimensional reconstruction result based on the original image data, obtaining real-time image data within a preset time interval during the operation, and performing image processing on the real-time image data to obtain enhanced image data, wherein the original image data is obtained from the binocular video; S2, performing artifact detection evaluation on the enhanced image data to obtain an artifact detection effect index, and judging whether to perform image registration based on the artifact detection effect index and a preset detection threshold range, and if not, performing artifact detection feedback to remind the preset personnel to reacquire real-time image data, wherein the artifact detection effect index is used to quantify the influence of artifacts on the enhanced image data; S3, after performing image registration, obtaining registered image data after image registration of the enhanced image data, evaluating the registration quality of the image registration according to the enhanced image data and the registered image data to obtain a registration quality evaluation index, wherein the registration quality evaluation index is used to describe the quality alignment degree of the registered image data during the registration process; S4, performing evaluation according to the registration quality evaluation index and the preset registration threshold range to obtain a comprehensive evaluation result, and judging whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result, wherein the comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

2. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The obtaining of the preliminary three-dimensional reconstruction result based on the original image data also includes the following steps: Acquire the patient's preoperative imaging data and import it into the three-dimensional reconstruction software, wherein the preoperative imaging data is stored in DICOM format; The kidney and its surrounding structures are identified by using an automatic segmentation algorithm to process the preoperative image data to obtain initial image data, and the initial image data is processed by Gaussian filtering to obtain original image data.

3. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The specific steps of obtaining the enhanced image data are as follows: The real-time image data is smoothed by Gaussian filtering to obtain smoothed image data, and the smoothed image data is edge processed to obtain first enhanced image data; Performing high frequency component enhancement processing on the first enhanced image data to obtain second enhanced image data; The second enhanced image data is subjected to histogram equalization processing to obtain enhanced image data.

4. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The specific method for obtaining the artifact detection effect index is as follows: Identify and number the artifact area of ​​the enhanced image data through edge detection, and obtain the number of pixels in the artifact area with the corresponding number and the corresponding pixel brightness value; Performing statistical analysis on the pixel brightness values ​​to obtain brightness correlation values ​​of the artifact area, and obtaining directional gradients of the enhanced image data, wherein the directional gradients include horizontal directional gradients and vertical directional gradients, and the brightness correlation values ​​include maximum pixel brightness, minimum pixel brightness, and average pixel brightness; The number of pixels in the artifact area, the pixel brightness value, the brightness correlation value, the directional gradient of the enhanced image data are combined with the evaluation weights obtained from the preset database to obtain the artifact detection effect index, wherein the evaluation weights include the artifact intensity weight, the artifact contrast weight and the edge sharpness weight.

5. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 4, characterized in that: The artifact detection effect index is calculated using the following formula: Where W represents the artifact detection effect index, i represents the pixel number in the artifact area, i = 1, 2, ..., N, N represents the total number of pixels in the artifact area, j represents the artifact area number of the enhanced image data, j = 1, 2, ..., M, M represents the total number of artifact areas of the enhanced image data, represents the pixel brightness value of the i-th pixel in the j-th artifact area, I max Indicates the maximum pixel brightness, I min Indicates the minimum pixel brightness, I a represents the average pixel brightness, I(x,y) represents the pixel value of the enhanced image data at the coordinate (x,y), represents the horizontal gradient, represents the vertical gradient, c1 represents the artifact intensity weight, c2 represents the artifact contrast weight, c3 represents the edge sharpness weight, ΔA represents the pixel brightness reference value, ΔB represents the artifact contrast reference value, and ΔC represents the edge sharpness reference value.

6. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The specific steps of obtaining enhanced image data for image registration are as follows: Acquire enhanced image data and adjacent enhanced image data within a preset time interval and perform feature extraction to obtain invariant feature points, wherein the adjacent enhanced image data is used to describe enhanced image data of adjacent preset time intervals, and the invariant feature points include enhanced invariant feature points and adjacent invariant feature points; Perform feature matching on the invariant feature points to obtain matching feature points, process the matching feature points by the least square method to obtain a transformation matrix, and perform image alignment on the transformation matrix to obtain aligned image data; The aligned image data and the enhanced image data are fused to obtain the registered image data.

7. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 6, characterized in that: The specific method for obtaining the registration quality evaluation index is as follows: Acquire the total number of pixels of the registered image data and the grayscale value of the image pixels, wherein the grayscale value of the image pixels includes the grayscale value of the registered image data and the grayscale value of the enhanced image data; Obtaining an average grayscale value of the image according to the grayscale values ​​of the registered image data and the enhanced image data at corresponding pixels, wherein the average grayscale value of the image includes the average grayscale value of the registered image data and the average grayscale value of the enhanced image data; The total number of pixels of the registered image data, the image pixel grayscale value, the image average grayscale value, the reference grayscale value obtained from the preset database and the evaluation weight are combined to obtain the registration quality evaluation index, wherein the evaluation weight includes the difference degree weight and the cross-correlation degree weight, and the reference grayscale value includes the reference grayscale value of the registered image data and the reference grayscale value of the enhanced image data.

8. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 7, characterized in that: The registration quality evaluation index is calculated using the following formula: Where Z represents the registration quality evaluation index, q represents the pixel number of the registered image data, q = 1, 2, ..., Q, Q represents the total number of pixels of the registered image data, represents the gray value of the registered image data of the qth pixel, represents the gray value of the enhanced image data of the qth pixel, represents the average gray value of the registered image data, represents the average gray value of enhanced image data, Δp represents the reference gray value of registered image data, Δh represents the reference gray value of enhanced image data, α represents the difference degree weight, β represents the cross-correlation degree weight, ΔM represents the registration difference reference value, and ΔN represents the registration cross degree reference value.

9. The three-dimensional reconstruction method of binocular video for minimally invasive partial nephrectomy according to claim 1, characterized in that: The specific steps of determining whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation results are as follows: Obtaining a registration quality evaluation index and a preset registration threshold range, comparing the registration quality evaluation index with the preset registration threshold range, and performing dynamic three-dimensional reconstruction if the registration quality evaluation index is within the preset registration threshold range; Otherwise, the registration quality assessment index is fed back to remind the preset personnel to change the registration technology and perform the image registration quality assessment again to obtain the registration quality assessment index.

10. A 3D reconstruction system for binocular video of minimally invasive partial nephrectomy, characterized in that: include: Image data acquisition module, artifact detection and evaluation module, image registration evaluation module, and comprehensive evaluation and judgment module; The image data acquisition module is used to acquire a preliminary three-dimensional reconstruction result based on the original image data, acquire real-time image data within a preset time interval during the operation, and perform image processing on the real-time image data to obtain enhanced image data, wherein the original image data is acquired from the binocular video; The artifact detection and evaluation module is used to perform artifact detection and evaluation on the enhanced image data to obtain an artifact detection effect index, and determine whether to perform image registration based on the artifact detection effect index and a preset detection threshold range. If not, artifact detection feedback is performed to remind the preset personnel to re-acquire real-time image data. The artifact detection effect index is used to quantify the degree of influence of artifacts on the enhanced image data; The image registration evaluation module is used to obtain the registered image data after the enhanced image data is registered after the image registration is performed, and to evaluate the registration quality of the image registration according to the enhanced image data and the registered image data to obtain a registration quality evaluation index, wherein the registration quality evaluation index is used to describe the quality alignment degree of the registered image data during the registration process; The comprehensive evaluation judgment module is used to obtain a comprehensive evaluation result based on the alignment quality evaluation index and the preset alignment threshold range, and to judge whether to perform dynamic three-dimensional reconstruction based on the comprehensive evaluation result. The comprehensive evaluation result is used to judge whether to perform dynamic three-dimensional reconstruction.

Images