A lesion real-time three-dimensional imaging system of a minimally invasive surgery robot

Abstract

The present application relates to the technical field of image data processing, in particular to a lesion real-time three-dimensional imaging system of a minimally invasive surgical robot, corresponding steps comprising: acquiring a two-dimensional image in a user's body, screening the two-dimensional image based on artifact feature analysis of the two-dimensional image; determining a generation time interval of the screened two-dimensional image to a three-dimensional image and generating an initial real-time three-dimensional image based on a user's breathing curve and movement information of a surgical robot; registering the initial real-time three-dimensional image with a corresponding preoperative three-dimensional model, constructing a difference distance field between the registered real-time three-dimensional image and the preoperative three-dimensional model; determining an interference region in the registered real-time three-dimensional image based on the difference distance field, replacing the interference region with a region corresponding to the interference region in the preoperative three-dimensional model to obtain a target real-time three-dimensional image. Through the technical scheme of the present application, the real-time performance of the three-dimensional image on the lesion region is more accurate and has more reference value.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and more specifically to a real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot. Background Technology

[0002] Real-time 3D imaging in minimally invasive surgical robots has brought revolutionary benefits to surgery. It acquires data through a stereoscopic endoscope and a high-precision tracker, and then reconstructs the image in real time using a GPU (Graphics Processing Unit) workstation, generating a dynamically updated 3D model of the lesion. This technology provides surgeons with a "see-through" ability that surpasses naked-eye vision, clearly showing the three-dimensional relationship between the lesion and surrounding key blood vessels and nerves, thereby greatly improving the precision and safety of surgery, while effectively optimizing the surgical procedure and shortening the operation time.

[0003] Traditional methods identify interfering regions based solely on grayscale differences and the fragmentation of boundary distribution within the generated 3D image. They fail to consider the masking interference of tissue fluid and other substances that resemble the appearance of lesion tissue in the image, resulting in inaccurate real-time representation of the lesion region in the final 3D imaging. Summary of the Invention

[0004] To address the current technical problem that the real-time representation of lesion areas obtained through minimally invasive surgical robots in 3D imaging of the human body is not accurate enough, the present invention aims to provide a real-time 3D imaging system for lesions using a minimally invasive surgical robot. The specific technical solution adopted is as follows: This invention provides a real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot, the system comprising: The image filtering module is used to acquire two-dimensional images of the user's body through the surgical robot during the operation, and to filter two-dimensional images based on the artifact feature analysis of the two-dimensional images; The 3D modeling module is used to determine the generation time interval from the selected 2D images to 3D images based on the user's breathing curve and the movement information of the surgical robot during the operation, and to generate an initial real-time 3D image. The initial real-time 3D image is then registered with the corresponding preoperative 3D model to obtain a registered real-time 3D image, and a difference distance field is constructed between the registered real-time 3D image and the preoperative 3D model. Based on the difference distance field, interference regions in the registered real-time 3D image are determined, and the interference regions are replaced with the corresponding regions in the preoperative 3D model to obtain the target real-time 3D image.

[0005] Furthermore, the step of filtering two-dimensional images based on artifact feature analysis includes: Determine the target edge features in the target 2D image, and determine the number of feature matches in the target 2D image that match the target edge features; Determine the mean gray value in the target 2D image, and use the number of feature matches and the mean gray value to determine the degree of necessity for filtering out the target 2D image; Based on the degree of necessity for filtering, it is determined whether to retain the target two-dimensional image, and the filtered two-dimensional image is obtained.

[0006] Furthermore, the step of determining the necessity of filtering out the target two-dimensional image using the number of feature matches and the mean gray level includes: Determine the difference between the maximum number of feature matches in the target 2D image and the maximum number of feature matches in all historical 2D images; Identify other two-dimensional images at the same breathing fluctuation position as the target two-dimensional image, and determine the difference in grayscale mean between the target two-dimensional image and other two-dimensional images; The degree of necessity for filtering out the target two-dimensional image is determined by using the difference in quantity and the difference in grayscale mean.

[0007] Furthermore, determining the time interval for generating a three-dimensional image from the filtered two-dimensional image based on the user's breathing curve and the movement information of the surgical robot during the operation includes: The respiratory pressure sensor converts the user's breathing movements into pressure fluctuations, resulting in a breathing curve with corresponding electrical signals. Based on the user's respiratory curve during surgery, the user's respiratory rate data is determined. Using the user's respiratory rate data and the movement rate of the surgical robot during surgery, the time interval for generating three-dimensional images from the selected two-dimensional images is determined.

[0008] Furthermore, the determination of the user's respiratory rate data based on the user's respiratory curve during surgery includes: The respiratory rate at each moment is obtained by using the time interval between adjacent peaks or troughs in the user's respiratory curve during the operation. Based on the respiratory rate at each moment, the fastest respiratory rate, the current respiratory rate at the current moment, and the difference in respiratory rate between the current moment and the previous moment are determined and used as the user's respiratory rate data.

[0009] Further, the step of registering the initial real-time 3D image with the corresponding preoperative 3D model to obtain the registered real-time 3D image includes: Determine the registration point distance between the initial real-time 3D image and the corresponding preoperative 3D model under the target registration orientation; Determine the maximum displacement distance of the registration point in the preoperative 3D model within the same breathing action of the user; The registration rationality of the target registration orientation is determined by using the registration point distance and the maximum displacement distance. Based on the maximum registration rationality, registration is performed to obtain a real-time 3D image after registration.

[0010] Furthermore, the determination of the registration rationality of the target registration orientation using the registration point distance and the maximum displacement distance includes: Determine the number of similar registration points whose distance is less than a preset distance threshold and the total number of registration points; determine the registration point ratio between the number of similar registration points and the total number of registration points. Determine the distance ratio between the registration point distance and the maximum displacement distance, and use the registration point ratio and the distance ratio to determine the registration rationality of the target registration orientation.

[0011] Furthermore, a difference distance field is constructed between the registered real-time 3D image and the preoperative 3D model, including: Determine the nearest point distance between the closest vertices of the real-time 3D image after registration and the preoperative 3D model, and construct a difference distance field using the nearest point distances.

[0012] Furthermore, the step of determining the interference region in the registered real-time 3D image based on the difference distance field includes: Density clustering algorithm is used to cluster the difference distance field to obtain multiple clusters, and the first distance difference between the maximum and minimum distances of the nearest points in the target cluster is determined. Determine the maximum angle between the vectors formed by the closest vertices in the target cluster, and use the first distance difference and the maximum angle to determine the interference region in the real-time 3D image after registration.

[0013] Further, determining the interference region in the registered real-time 3D image using the first distance difference and the maximum included angle includes: Determine the second distance difference between the first distance difference of the target cluster and the maximum first distance difference in the historical clusters; determine the angle ratio between the maximum included angle of the target cluster and the maximum included angle value in the historical clusters. The number of point pairs with the closest vertices in the target cluster is determined, and the interference region in the real-time 3D image after registration is determined using the second distance difference, the included angle ratio, and the number of point pairs.

[0014] The present invention has the following beneficial effects: This invention adjusts the real-time construction frequency of 3D images based on the user's breathing rate and the movement speed of the surgical robot; it judges the image quality based on the presence of artifacts in the acquired 2D images and filters them accordingly, using the filtered 2D images to construct 3D images and obtain the real-time construction quality of the 3D images; it performs position registration by combining the changes of registration points in the preoperative 3D model with breathing and the matching performance of registration points between the 3D images and the preoperative 3D model; and it replaces corresponding interference areas in the real-time 3D images based on the differences between the registered 3D images and the preoperative 3D model, so that interference factors, including tissue fluid, smoke, and instrument obstruction, are eliminated as much as possible in the final real-time 3D imaging provided to doctors for reference, making the real-time representation of the lesion area in the 3D images more accurate and more valuable for reference. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 The flowchart shows the steps of a real-time three-dimensional imaging system for lesions provided by a minimally invasive surgical robot according to an embodiment of the present invention. Figure 2 A detailed flowchart of step S1 in a real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot, provided in an embodiment of the present invention; Figure 3 A detailed flowchart of step S2 in a real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot, provided in an embodiment of the present invention; Figure 4 A detailed flowchart of step S3 in a real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot, provided as an embodiment of the present invention; Figure 5 A detailed flowchart of step S4 in a real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot, provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the hardware operating environment of the real-time three-dimensional imaging device for lesions of the minimally invasive surgical robot involved in the embodiments of the present invention. Figure 7 This is a schematic diagram of the framework structure of the real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot involved in the embodiments of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] It should be noted that, for ease of calculation, all indicator data involved in the calculation in this embodiment of the invention have undergone data preprocessing to eliminate the influence of dimensions. The specific methods for eliminating the influence of dimensions are well known to those skilled in the art and are not limited here.

[0020] The specific solution of the real-time three-dimensional imaging system for lesions of a minimally invasive surgical robot provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Example 1: For a real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot provided by this invention, please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram of the framework structure of the real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot involved in the embodiments of the present invention.

[0022] The real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot (hereinafter referred to as the "real-time three-dimensional imaging system for lesions") includes: The image filtering module A10 is used to acquire two-dimensional images of the user's body through the surgical robot during the operation, and to filter two-dimensional images based on the artifact feature analysis of the two-dimensional images. The 3D modeling module A20 is used to determine the generation time interval from the selected 2D image to the 3D image based on the user's breathing curve and the movement information of the surgical robot during the operation, and to generate an initial real-time 3D image. The initial real-time 3D image is then registered with the corresponding preoperative 3D model to obtain a registered real-time 3D image. A difference distance field is constructed between the registered real-time 3D image and the preoperative 3D model. Based on the difference distance field, interference regions in the registered real-time 3D image are determined, and the interference regions are replaced with the corresponding regions in the preoperative 3D model to obtain the target real-time 3D image.

[0023] Please see Figure 1 , Figure 1The flowchart of the steps corresponding to the real-time three-dimensional imaging system of the lesion of the minimally invasive surgical robot provided in one embodiment of the present invention is shown.

[0024] The various method steps corresponding to the real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot include: Step S1: During the surgery, a two-dimensional image of the user's body is acquired through a surgical robot, and the two-dimensional image is filtered based on the artifact feature analysis of the two-dimensional image. Generally, when acquiring two-dimensional video of a user's body using a minimally invasive surgical robot, the robotic arm is precisely controlled, and the end effector drives the stereoscopic endoscope to perform smooth and continuous scanning movements along a predetermined trajectory. Common methods include linear translation or fan-shaped oscillation. During this process, the system continuously captures a continuous two-dimensional video stream at a high frame rate and synchronously records the spatial position and orientation parameters of the endoscope corresponding to each frame of the two-dimensional image through a precisely calibrated "hand-eye" coordinate system. This data package, which combines image information and spatial pose, provides the most crucial input for subsequent three-dimensional reconstruction.

[0025] Specifically, please refer to Figure 2 Step S1, which involves filtering two-dimensional images based on artifact feature analysis, includes: Step S11: Determine the target edge features in the target two-dimensional image, and determine the number of feature matches in the target two-dimensional image that match the target edge features; Step S12: Determine the mean gray value in the target two-dimensional image, and use the number of feature matches and the mean gray value to determine the degree of necessity for filtering out the target two-dimensional image; More specifically, step S12, which determines the necessity of filtering out the target two-dimensional image using the number of feature matches and the average gray level, includes: Determine the difference between the maximum number of feature matches in the target 2D image and the maximum number of feature matches in all historical 2D images; Identify other two-dimensional images at the same breathing fluctuation position as the target two-dimensional image, and determine the difference in grayscale mean between the target two-dimensional image and other two-dimensional images; The degree of necessity for filtering out the target two-dimensional image is determined by using the difference in quantity and the difference in grayscale mean.

[0026] Step S13: Determine whether to retain the target two-dimensional image based on the degree of necessity for filtering, and obtain the filtered two-dimensional image.

[0027] In this embodiment, it is necessary to judge the image quality based on the presence of artifacts in the two-dimensional image and to filter the two-dimensional images used for three-dimensional image construction.

[0028] Specifically, non-rigid deformations of internal tissues caused by respiration can lead to artifacts when acquiring two-dimensional images using an endoscope. Artifacts (such as specular highlights, motion blur, and electrosurgical smoke) create false features in the two-dimensional image, masking real features. This results in numerous errors in feature matching, causing the generated three-dimensional image to deviate significantly from the actual anatomical structure. Therefore, two-dimensional images with severe artifacts and poor quality should be screened to ensure the accuracy of the final three-dimensional image.

[0029] Edges in a 2D image are detected and obtained using the Canny edge detection method.

[0030] The target edge k is matched in a 2D image j (which can be any 2D image) using a feature-based matching method, and the number of matches for edge k is obtained. That is, the number of feature matches.

[0031] The maximum number of feature matches for each edge in the two-dimensional image j is obtained by comparison. The maximum number of matches among edges is obtained by comparing all historical images (such as the user's own samples). This refers to the maximum number of feature matches, which is a "preset theoretical limit" at a certain sampling time. Greater than At that time, it can be determined that a processing error has occurred, and The value was directly adjusted to be the same as same.

[0032] It also obtains the average grayscale value of each pixel in the two-dimensional image j. .

[0033] when and Quantity difference The smaller the value, the better the mean gray level of each pixel in the two-dimensional image j. The grayscale mean of p obtained at the same respiratory phase (the same respiratory fluctuation position, which is represented by the corresponding respiratory phase in the respiratory curve) and other two-dimensional images (which can be user-owned samples) The difference in grayscale mean between and ( A higher number of two-dimensional images (the total number of images acquired during a single breathing phase) indicates a greater number of artifacts in the two-dimensional image (artifact features are closer to edge features), and more severe gray-level occlusion caused by artifact regions. This results in a significant difference in the average gray-level of two-dimensional image j compared to other images in the same phase. In this case, the lower the image quality of two-dimensional image j, the less suitable it is as a reference for constructing a three-dimensional image.

[0034] Therefore, the necessity of filtering out image j in the two-dimensional image sequence acquired at the current time t can be obtained. Using maximum and minimum value normalization After normalization, we get Its range is [0,1]. When (When a preset screening threshold is set, the specific value can be determined based on historical data analysis.) This screens out 2D image j within its 2D image sequence, ensuring that the 3D image constructed from the screened 2D images better matches the user's anatomical structure, providing a more accurate reference for doctors in minimally invasive surgery. The remaining 2D images with pose information are then transmitted to a high-performance GPU workstation, where stereo matching and 3D reconstruction algorithms fuse the 2D sequences to generate the corresponding 3D image of the lesion.

[0035] This indicates a normalization process used to achieve dimensionless measurement and standardization. In one embodiment of the present invention, the normalization process may specifically be, for example, a maximum and minimum value normalization process.

[0036] Unless otherwise specified, Maximum-minimum normalization is used to normalize the results to the [0, 1] interval or other continuous intervals. The maximum and minimum values ​​used in maximum-minimum normalization can be obtained based on the actual situation. For example, when multiple values ​​can be obtained and the relationship between different values ​​needs to be compared, multiple values ​​can be counted to obtain the maximum and minimum values. However, when only a single value can be obtained or the relationship between different values ​​does not need to be compared, the maximum and minimum values ​​can be obtained based on a large amount of historical experimental data or prior data. When performing maximum-minimum analysis based on a large amount of historical experimental data, the maximum and minimum values ​​are used as cutoff values ​​for the analysis. When the data is greater than the historical maximum or less than the historical minimum, the value is truncated and determined as the corresponding cutoff value for maximum-minimum normalization calculation.

[0037] Step S2: Based on the user's breathing curve and the movement information of the surgical robot during the operation, determine the time interval for generating the three-dimensional image from the screened two-dimensional image and generate the initial real-time three-dimensional image. Specifically, please refer to Figure 3 Step S2, based on the user's breathing curve and the movement information of the surgical robot during the operation, determines the time interval for generating a three-dimensional image from the filtered two-dimensional image, including: Step S21: The user's breathing motion is converted into pressure fluctuations by a breathing pressure sensor to obtain the corresponding electrical signal breathing curve. Step S22: Determine the user's respiratory rate data based on the user's respiratory curve during the operation, and determine the time interval for generating the filtered two-dimensional image into a three-dimensional image using the user's respiratory rate data and the movement rate of the surgical robot during the operation.

[0038] More specifically, step S22, determining the user's respiratory rate data based on the user's respiratory curve during surgery, includes: The respiratory rate at each moment is obtained by using the time interval between adjacent peaks or troughs in the user's respiratory curve during the operation. Based on the respiratory rate at each moment, the fastest respiratory rate, the current respiratory rate at the current moment, and the difference in respiratory rate between the current moment and the previous moment are determined and used as the user's respiratory rate data.

[0039] In this embodiment, the real-time time interval for building the 3D image needs to be adjusted according to the changes in the user's breathing rate and the movement speed of the surgical robot.

[0040] Specifically, the user's breathing rate will change during surgery. If the surgical robot moves at a high speed at the same time, in order to better assist the doctor in performing the surgery, the frequency of acquiring and building three-dimensional images of the user's body should be increased to meet the doctor's field of vision requirements during surgery and avoid causing harm to the user.

[0041] A respiratory pressure sensor is wrapped around an inflatable bandage around the user's chest or abdomen to monitor periodic pressure changes caused by respiratory movements in real time. When the user inhales, the expansion of the chest and abdomen causes the pressure inside the bandage to increase; when exhaling, the pressure decreases accordingly. This continuous pressure fluctuation is converted into a voltage signal by the sensor, forming a smooth respiratory waveform curve. This waveform directly reflects the depth and rhythm of breathing, providing a raw and continuous data basis for subsequent phase and rate calculations.

[0042] Based on the respiratory waveform, the system automatically identifies peaks and troughs using a real-time algorithm, corresponding to the extreme phases at the end of inspiration and expiration, respectively. By calculating the reciprocal of the time interval between consecutive peaks or troughs, the instantaneous respiratory rate can be directly obtained.

[0043] Specifically, obtain the user's current respiratory rate at time t. (The breathing rate is calculated by taking the reciprocal of the time interval between the most recent completed peak or trough and the previous peak or trough; the shorter the time interval, the faster the breathing.)

[0044] The user's fastest respiratory rate was obtained through comparison from historical monitoring. It should be noted that this fastest breathing rate is a "preset theoretical limit," measured at a specific sampling time. Greater than At that time, it can be determined that a processing error has occurred, and The value was directly adjusted to be the same as Same. It should also be noted that, due to... This represents the user's fastest breathing rate, which can be specifically stated as 90 breaths per minute, and the objective value is not 0.

[0045] The real-time movement rate of the surgical robot is obtained by a rate sensor installed at the endoscope of the surgical robot.

[0046] The movement rate of the surgical robot at time t Compared to the maximum movement rate in historical monitoring The difference The smaller the value, the lower the user's respiratory rate at time t. With the fastest breathing rate ratio The larger, and The larger the value, the faster the user's breathing rate at time t, and the greater the variation in breathing rate. At the same time, the robot's movement is faster. In order to avoid accidental damage to the user's lesions, the frequency of acquiring three-dimensional images should be increased and the time interval between acquiring the next three-dimensional image should be reduced to provide doctors with more accurate surgical assistance.

[0047] Therefore, the required time interval for acquiring the 3D image at time t can be obtained. D is a pre-defined reference interval for acquiring 3D images, such as 0.5 seconds. norm is a normalization process using the maximum and minimum values, resulting in a value range of [0,1]. and For the corresponding preset weight values, For example, 0.4. For example, it could be 0.6. This serves as the time interval for generating a 3D image from the filtered 2D image.

[0048] The specific preset values ​​given in the embodiments of the present invention (such as...) and The values ​​(etc.) are empirical values ​​obtained under typical hardware configurations and test scenarios, intended to facilitate understanding of the present invention. In practical applications, those skilled in the art can adjust, calibrate, or optimize these parameters according to specific hardware performance, scenario complexity, and data characteristics, which does not constitute a limitation of the present invention.

[0049] This allows for real-time acquisition of the 3D image acquisition and construction time interval at different times, enabling the 3D image of the user's body obtained based on the adjusted generation time interval (denoted as the initial real-time 3D image) to better assist doctors in surgical operations.

[0050] Step S3: Register the initial real-time 3D image with the corresponding preoperative 3D model to obtain the registered real-time 3D image, and construct the difference distance field between the registered real-time 3D image and the preoperative 3D model. Interference from tissue fluid, smoke, instruments, and specular highlights at the lesion site can cause interference areas in the obtained 3D images, affecting the real-time display of the lesion area and consequently interfering with the surgeon's operation. Therefore, the obtained 3D images are registered with the 3D model of the lesion obtained from preoperative CT (Computed Tomography) images to identify and prepare for the procedure.

[0051] Specifically, please refer to Figure 4 Step S3, which involves registering the initial real-time 3D image with the corresponding preoperative 3D model to obtain a registered real-time 3D image, includes: Step S31: Determine the distance between the initial real-time 3D image and the corresponding preoperative 3D model under the target registration orientation; Step S32: Determine the maximum displacement distance of the registration point in the preoperative 3D model within the same breathing action of the user; Step S33: Determine the registration rationality of the target registration orientation using the registration point distance and the maximum displacement distance, and perform registration based on the maximum registration rationality to obtain a real-time three-dimensional image after registration.

[0052] More specifically, step S33, which determines the registration rationality of the target registration orientation using the registration point distance and the maximum displacement distance, includes: Determine the number of similar registration points whose distance is less than a preset distance threshold and the total number of registration points; determine the registration point ratio between the number of similar registration points and the total number of registration points. Determine the distance ratio between the registration point distance and the maximum displacement distance, and use the registration point ratio and the distance ratio to determine the registration rationality of the target registration orientation.

[0053] In this embodiment, multiple pre-set reference registration points are obtained to register the (initial) real-time 3D image with the preoperative 3D model.

[0054] For the preoperative 3D model, by acquiring CT images of the user at different respiratory phases before surgery, the maximum displacement distance of registration point i during one breath of the user is calculated. .because It represents the maximum displacement distance under one breath, and its objective value is not 0.

[0055] Multiple registration orientations are arranged between real-time 3D images and preoperative 3D models using a computer. The distances between each registration point i in the real-time 3D image and the registration point in the preoperative 3D model are calculated at registration orientation a (target registration orientation, referring to any registration orientation). .

[0056] In the statistical registration orientation a, the distance between the registration points in the real-time 3D image and the preoperative 3D model is... Number of registration points smaller than L (preset distance threshold, set according to actual needs, such as 5 pixels in length) As the number of similar registration points.

[0057] when ( Given the total number of registration points, this scheme mainly analyzes the registration points; therefore... The value is at least 1, in When the value is 0, the registration rationality is directly determined to be 0, and the distance ratio is... The smaller the registration distance (excluding the influence of registration point i's breathing displacement), the greater the number of... and Registration point ratio The larger the value, the more reasonable the registration orientation 'a' is, and the more suitable it is for matching and identifying interference regions. It should be noted that... and Since the dimensions are eliminated through ratio calculation, all data are dimensionless.

[0058] This allows us to determine the degree of registration rationality of registration orientation a. The above analysis was performed on each registration orientation, and the registration form (registration orientation) with the highest degree of registration rationality was selected for registration to obtain a real-time 3D image after registration. This image serves as a reference for subsequent identification of interference areas, making the identification of interference areas more accurate and the display of real-time 3D images of lesions more consistent with the user's anatomical structure.

[0059] Specifically, step S3, constructing the difference distance field between the registered real-time 3D image and the preoperative 3D model, includes: Determine the nearest point distance between the closest vertices of the real-time 3D image after registration and the preoperative 3D model, and construct a difference distance field using the nearest point distances.

[0060] In this embodiment, interfering regions are identified based on the differences between the registered 3D image and the preoperative 3D model of the lesion. A (difference) distance field is generated by calculating the nearest point distance between the surface vertices of the registered real-time 3D image and the preoperative 3D model of the lesion. Interference within individual clusters is identified based on the distribution and length differences of distances across different cluster ranges, and interfering regions are then replaced.

[0061] For the nearest point distance, the corresponding lesion surface mesh and the vertices of each surface are extracted from the preoperative 3D model and the intraoperative real-time 3D image, respectively. For example, for each vertex of the intraoperative surface, the nearest point distance between it and the nearest vertex in the preoperative model surface is calculated, thereby generating a (difference) distance field.

[0062] Step S4: Based on the difference distance field, determine the interference region in the registered real-time 3D image, replace the interference region with the region corresponding to the interference region in the preoperative 3D model, and obtain the target real-time 3D image.

[0063] Specifically, please refer to Figure 5 Step S4, determining the interference region in the registered real-time 3D image based on the difference distance field, includes: Step S41: Use density clustering algorithm to cluster the difference distance field to obtain multiple clusters, and determine the first distance difference between the maximum and minimum distances of the nearest points in the target cluster. Step S42: Determine the maximum angle between the vectors formed by the closest vertices in the target cluster, and use the first distance difference and the maximum angle to determine the interference region in the real-time 3D image after registration.

[0064] More specifically, step S42, which uses the first distance difference and the maximum included angle to determine the interference region in the registered real-time 3D image, includes: Determine the second distance difference between the first distance difference of the target cluster and the maximum first distance difference in the historical clusters; determine the angle ratio between the maximum included angle of the target cluster and the maximum included angle value in the historical clusters. The number of point pairs with the closest vertices in the target cluster is determined, and the interference region in the real-time 3D image after registration is determined using the second distance difference, the included angle ratio, and the number of point pairs.

[0065] In this embodiment, the distance field is clustered using the DBSCAN density clustering method to obtain multiple cluster ranges, i.e., clusters. For a single cluster range v (the target cluster, referring to any cluster), the number of points whose corresponding nearest vertices on the preoperative model surface are greater than 1 is counted. The statistics are performed, that is, the number of pairs of points that are closest to each other is counted, and the number of pairs of points that are excluded is 1.

[0066] Calculate the difference between the maximum and minimum distances among the nearest points in a cluster range v. Let be the first distance difference.

[0067] Calculate the maximum angle in space between the vectors corresponding to the distances between each pair of closest points in the cluster range v. That is, the maximum angle between the displacement vectors of the surface vertices of the real-time 3D image after registration within the cluster and the corresponding nearest vertex of the preoperative 3D model.

[0068] When the number of points The larger the value, the greater the difference. The maximum value of the difference between the maximum and minimum distances to the nearest point within the cluster range of historical records (the first distance difference). The difference (second distance difference) The smaller the value, and the larger the maximum included angle. The maximum value of the angle between the maximum and the maximum range of each cluster in the historical records. The larger the angle ratio, the more disordered the distance field corresponding to the clustering range v becomes, the more mismatched the preoperative model and the intraoperative image surface become, and the more likely there is interference.

[0069] This allows us to obtain the interference conformity of the intraoperative image surface as the interference region corresponding to the cluster range v. The parameter 0.01 is set to avoid the numerator or denominator being 0. It can be understood that the 0.01 in different positions has the same computational dimension as its location. The maximum-minimum normalization method is used to... After normalization, we get Its range is [0,1]. When (When the preset interference threshold is set, it can be adjusted.) The intraoperative image surface corresponding to the cluster range v in the real-time 3D image after registration is identified as the interference area. The corresponding area on the preoperative model surface is replaced with it to obtain the final target real-time 3D image. This ensures the accuracy of the intraoperative reference and reduces interference from tissue fluid, smoke, instrument obstruction, specular highlights, etc.

[0070] The above process yields more accurate three-dimensional images of the lesion at different surgical times.

[0071] In a real-time 3D imaging system, the generated 3D image of the lesion is rapidly retrieved from the GPU memory via a high-speed data bus. High-bandwidth, low-latency digital image signals are transmitted to a high-resolution stereoscopic display on the surgeon's console via a dedicated video output interface (such as DisplayPort 1.4). The system employs dual-channel synchronous rendering technology to project the 3D model onto the display as a stereo pair. Surgeons can perceive the 3D structure of the lesion with realistic depth relationships by wearing active stereoscopic glasses or using a naked-eye 3D display with built-in optical paths. The entire process relies on a high-performance closed loop consisting of a powerful graphics workstation and dedicated display hardware, ensuring real-time synchronization between visualization and surgical procedures.

[0072] This invention adjusts the real-time construction frequency of 3D images based on the user's breathing rate and the movement speed of the surgical robot; it judges the image quality based on the presence of artifacts in the acquired 2D images and filters them accordingly, using the filtered 2D images to construct 3D images and obtain the real-time construction quality of the 3D images; it performs position registration by combining the changes of registration points in the preoperative 3D model with breathing and the matching performance of registration points between the 3D images and the preoperative 3D model; and it replaces corresponding interference areas in the real-time 3D images based on the differences between the registered 3D images and the preoperative 3D model, so that interference factors, including tissue fluid, smoke, and instrument obstruction, are eliminated as much as possible in the final real-time 3D imaging provided to doctors for reference, making the real-time representation of the lesion area in the 3D images more accurate and more valuable for reference.

[0073] Example 2: This invention also proposes a real-time three-dimensional imaging device for lesions using a minimally invasive surgical robot. The device can be a data processing device such as a computer or a server, or a combination of multiple devices.

[0074] like Figure 6 As shown, Figure 6 This is a schematic diagram of the hardware operating environment of the real-time three-dimensional imaging device for lesions of the minimally invasive surgical robot involved in the embodiments of the present invention.

[0075] like Figure 6 As shown, the real-time three-dimensional imaging device for lesions in this minimally invasive surgical robot may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display or an input unit such as a control panel; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a stable, non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001. The memory 1005, as a computer storage medium, may include the real-time three-dimensional imaging program for lesions of the minimally invasive surgical robot, referred to as the "real-time three-dimensional imaging program for lesions."

[0076] Those skilled in the art will understand that Figure 6 The hardware structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0077] Continue to refer to Figure 6 , Figure 6 The memory 1005, which is a computer-readable storage medium, may include an operating system, a user interface module, a network communication module, and a real-time three-dimensional imaging program for lesions of a minimally invasive surgical robot.

[0078] exist Figure 6 In this embodiment, the network communication module is mainly used to connect to the server and can communicate with the server for data; while the processor 1001 can call the real-time three-dimensional imaging program of the lesion of the minimally invasive surgical robot stored in the memory 1005 and execute the steps in the above embodiments.

[0079] Based on the hardware structure of the real-time three-dimensional imaging device for lesions of the minimally invasive surgical robot described above, various embodiments of the real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot of the present invention are implemented.

[0080] Furthermore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a real-time three-dimensional imaging program for lesions using a minimally invasive surgical robot. When executed by a processor, the real-time three-dimensional imaging program for lesions using a minimally invasive surgical robot implements the steps of the method corresponding to the real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot described above.

[0081] The method implemented when the real-time three-dimensional imaging program of the lesion of the minimally invasive surgical robot is executed can be referred to in various embodiments of the real-time three-dimensional imaging system of the lesion of the minimally invasive surgical robot of the present invention, and will not be repeated here.

[0082] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] The above description is only a preferred embodiment of the present invention and does not limit the scope of protection of the present invention. All equivalent structural / method transformations made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the scope of protection of the present invention.

Claims

1. A real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot, characterized in that, The system includes: The image filtering module is used to acquire two-dimensional images of the user's body through the surgical robot during the operation, and to filter two-dimensional images based on the artifact feature analysis of the two-dimensional images; The 3D modeling module is used to determine the generation time interval from the selected 2D images to 3D images based on the user's breathing curve and the movement information of the surgical robot during the operation, and to generate an initial real-time 3D image. The initial real-time 3D image is then registered with the corresponding preoperative 3D model to obtain a registered real-time 3D image, and a difference distance field is constructed between the registered real-time 3D image and the preoperative 3D model. Based on the difference distance field, interference regions in the registered real-time 3D image are determined, and the interference regions are replaced with the corresponding regions in the preoperative 3D model to obtain the target real-time 3D image.

2. The real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot according to claim 1, characterized in that, The method of filtering two-dimensional images based on artifact feature analysis includes: Determine the target edge features in the target 2D image, and determine the number of feature matches in the target 2D image that match the target edge features; Determine the mean gray value in the target 2D image, and use the number of feature matches and the mean gray value to determine the degree of necessity for filtering out the target 2D image; Based on the degree of necessity for filtering, it is determined whether to retain the target two-dimensional image, and the filtered two-dimensional image is obtained.

3. The real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot according to claim 2, characterized in that, The determination of the necessity of filtering out target two-dimensional images using the number of feature matches and the mean gray level includes: Determine the difference between the maximum number of feature matches in the target 2D image and the maximum number of feature matches in all historical 2D images; Identify other two-dimensional images at the same breathing fluctuation position as the target two-dimensional image, and determine the difference in grayscale mean between the target two-dimensional image and other two-dimensional images; The degree of necessity for filtering out the target two-dimensional image is determined by using the difference in quantity and the difference in grayscale mean.

4. The real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot according to claim 1, characterized in that, The process of determining the time interval for generating a three-dimensional image from the filtered two-dimensional image based on the user's breathing curve and the movement information of the surgical robot during the operation includes: The respiratory pressure sensor converts the user's breathing movements into pressure fluctuations, resulting in a breathing curve with corresponding electrical signals. Based on the user's respiratory curve during surgery, the user's respiratory rate data is determined. Using the user's respiratory rate data and the movement rate of the surgical robot during surgery, the time interval for generating three-dimensional images from the selected two-dimensional images is determined.

5. The real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot according to claim 4, characterized in that, The method of determining the user's respiratory rate data based on the user's respiratory curve during surgery includes: The respiratory rate at each moment is obtained by using the time interval between adjacent peaks or troughs in the user's respiratory curve during the operation. Based on the respiratory rate at each moment, the fastest respiratory rate, the current respiratory rate at the current moment, and the difference in respiratory rate between the current moment and the previous moment are determined and used as the user's respiratory rate data.

6. The real-time three-dimensional imaging system for lesions using a minimally invasive surgical robot according to claim 1, characterized in that, The process of registering the initial real-time 3D image with the corresponding preoperative 3D model to obtain the registered real-time 3D image includes: Determine the registration point distance between the initial real-time 3D image and the corresponding preoperative 3D model under the target registration orientation; Determine the maximum displacement distance of the registration point in the preoperative 3D model within the same breathing action of the user; The registration rationality of the target registration orientation is determined by using the registration point distance and the maximum displacement distance. Based on the maximum registration rationality, registration is performed to obtain a real-time 3D image after registration.

7. The real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot according to claim 6, characterized in that, The determination of the registration rationality by using the registration point distance and the maximum displacement distance to determine the target registration orientation includes: Determine the number of similar registration points whose distance is less than a preset distance threshold and the total number of registration points; determine the registration point ratio between the number of similar registration points and the total number of registration points. Determine the distance ratio between the registration point distance and the maximum displacement distance, and use the registration point ratio and the distance ratio to determine the registration rationality of the target registration orientation.

8. The real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot according to claim 1, characterized in that, Constructing the difference distance field between the registered real-time 3D image and the preoperative 3D model, including: Determine the nearest point distance between the closest vertices of the real-time 3D image after registration and the preoperative 3D model, and construct a difference distance field using the nearest point distances.

9. The real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot according to claim 8, characterized in that, The determination of interference regions in the registered real-time 3D image based on the difference distance field includes: Density clustering algorithm is used to cluster the difference distance field to obtain multiple clusters, and the first distance difference between the maximum and minimum distances of the nearest points in the target cluster is determined. Determine the maximum angle between the vectors formed by the closest vertices in the target cluster, and use the first distance difference and the maximum angle to determine the interference region in the real-time 3D image after registration.

10. The real-time three-dimensional imaging system for lesions of the minimally invasive surgical robot according to claim 9, characterized in that, Determining the interference region in the registered real-time 3D image using the first distance difference and the maximum included angle includes: Determine the second distance difference between the first distance difference of the target cluster and the maximum first distance difference in the historical clusters; determine the angle ratio between the maximum included angle of the target cluster and the maximum included angle value in the historical clusters. The number of point pairs with the closest vertices in the target cluster is determined, and the interference region in the real-time 3D image after registration is determined using the second distance difference, the included angle ratio, and the number of point pairs.

Images