A three-dimensional model display method and device based on model registration and a storage medium

Abstract

The embodiment of the specification provides a three-dimensional model display method and device based on model registration and a storage medium, which can be applied to the technical field of surgical positioning. The method comprises the following steps: constructing a preoperative three-dimensional model corresponding to a target part according to a preoperative scanning image; constructing an intraoperative three-dimensional model corresponding to the target part according to an intraoperative scanning image; determining a registration relationship between the preoperative three-dimensional model and the intraoperative three-dimensional model; determining a constraint point pair between the preoperative three-dimensional model and the intraoperative three-dimensional model based on the registration relationship; fusing the preoperative three-dimensional model and the intraoperative three-dimensional model to obtain a target three-dimensional model by combining the registration relationship and the constraint point pair; and displaying the target three-dimensional model. The target three-dimensional model is used to identify the positions of an intraoperative lesion and / or a blood vessel. The above method enables a doctor to intuitively master the positions of the blood vessel and the lesion, guides and assists the doctor in a needle insertion process, optimizes a surgical effect, and improves the treatment experience of a patient.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of surgical positioning technology, and in particular to a method, device and storage medium for displaying three-dimensional models based on model registration. Background Technology

[0002] Currently, when performing procedures such as focal ablation, doctors generally need to insert needles into the lesion area to perform the corresponding surgical operations. For example, the tips of several pairs of electrode needles are sent to the lesion area in the patient's body. In order to determine the needle insertion location, a CT scan is taken of the corresponding area, such as the target organ, before performing the surgical operation. The needle insertion operation is then performed based on the location of the lesion area shown on the CT scan.

[0003] However, since CT scans are typically taken preoperatively, surgeons rely heavily on their experience and technique during surgery to insert the needle. This lack of visual guidance can lead to blood vessel puncture and subsequent massive intraoperative bleeding. Furthermore, even after insertion, it's difficult for surgeons to objectively assess accuracy, requiring a follow-up CT scan to confirm the needle placement. Errors necessitate readjustment and another CT scan, prolonging surgery time and increasing radiation exposure for the patient. Therefore, a method that accurately and effectively assists surgeons in intraoperative needle localization is urgently needed. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a method, device, and storage medium for displaying three-dimensional models based on model registration, so as to solve the problem of how to accurately and effectively assist doctors in intraoperative positioning.

[0005] To address the aforementioned technical problems, embodiments of this specification propose a method for displaying a three-dimensional model based on model registration, comprising: constructing a preoperative three-dimensional model corresponding to a target site based on preoperative scan images; identifying the location of lesions and / or blood vessels in the preoperative three-dimensional model; constructing an intraoperative three-dimensional model corresponding to the target site based on intraoperative scan images; determining a registration relationship between the preoperative three-dimensional model and the intraoperative three-dimensional model; the registration relationship being used to achieve mapping between the models; determining constraint point pairs between the preoperative three-dimensional model and the intraoperative three-dimensional model based on the registration relationship; the constraint point pairs being used to indicate matching points between the preoperative three-dimensional model and the intraoperative three-dimensional model under the registration relationship; fusing the preoperative three-dimensional model and the intraoperative three-dimensional model by combining the registration relationship and the constraint point pairs to obtain a target three-dimensional model; and displaying the target three-dimensional model; the target three-dimensional model being used to identify the location of lesions and / or blood vessels during surgery.

[0006] In some embodiments, constructing a preoperative three-dimensional model corresponding to the target site based on preoperative scan images includes: identifying lesion area images in preoperative scan images, and / or identifying vascular images in preoperative scan images; and constructing a three-dimensional structure of the lesion and / or a three-dimensional structure of the vascular system at the corresponding positions in the preoperative three-dimensional model based on the positions of the lesion area images and / or vascular images.

[0007] Based on the above implementation, identifying the lesion region image in the preoperative scan image includes: determining candidate lesion regions in the preoperative scan image based on changes in brightness; comparing the preoperative scan image with a reference image to filter actual lesion regions from the candidate lesion regions; wherein, it includes: outlining areas of abnormal density in the reference image; determining the degree of overlap between the reference image and the areas of abnormal density in the candidate lesion regions; if the degree of overlap is less than a comparison threshold, determining the candidate lesion region as the actual lesion region; and segmenting the image corresponding to the actual lesion region as the lesion region image.

[0008] Based on the aforementioned embodiments, identifying vascular images in preoperative scan images includes: identifying contrast-enhanced images from preoperative scan images; the contrast-enhanced images are images presented after contrast agents are injected into the blood vessels; constructing a three-dimensional vascular structure at the corresponding position of the preoperative three-dimensional model based on the position of the vascular image includes: performing noise reduction processing on the constructed three-dimensional vascular structure; the noise reduction processing includes surface normal filtering processing.

[0009] In some implementations, constructing a preoperative three-dimensional model corresponding to the target site based on preoperative scan images includes: obtaining an average model trained based on sample image data; the average model is used to reflect the contour of the target site; and fitting the constructed preoperative three-dimensional model using the average model.

[0010] In some implementations, determining the registration relationship between the preoperative 3D model and the intraoperative 3D model includes: performing coarse registration on the preoperative 3D model and the intraoperative 3D model; the coarse registration is used to define the spatial position of the preoperative 3D model and the intraoperative 3D model; and calculating the transformation matrix between the models based on the spatial coordinates of each point on the preoperative 3D model and the intraoperative 3D model after coarse registration.

[0011] Based on the above implementation, the coarse registration of the preoperative 3D model and the intraoperative 3D model includes: establishing minimum bounding boxes adapted to the spatial positions of the preoperative 3D model and the intraoperative 3D model; the minimum bounding boxes corresponding to the preoperative 3D model and the intraoperative 3D model are located in the same spatial position.

[0012] Based on the aforementioned implementation method, the step of calculating the transformation matrix between the models based on the spatial coordinates of each point on the preoperative 3D model and the intraoperative 3D model after coarse registration includes: determining the corresponding registration points based on the spatial relationship between each point on the preoperative 3D model and the intraoperative 3D model after coarse registration; obtaining the transformation matrix based on the spatial coordinates of the registration points; the transformation matrix is ​​used to ensure that the distance difference between the transformed registration points is less than the normalization threshold.

[0013] In some implementations, determining the constraint point pairs between the preoperative 3D model and the intraoperative 3D model based on the registration relationship includes: calculating the residual value of the target transformation function of the point pairs between the preoperative 3D model and the intraoperative 3D model based on the registration relationship; and determining the point pairs with residual values ​​less than a matching threshold as constraint point pairs.

[0014] In some implementations, the step of fusing the preoperative 3D model and the intraoperative 3D model by combining the registration relationship and the constraint point pairs to obtain the target 3D model includes: identifying noise points between the preoperative 3D model and the intraoperative 3D model other than constraint point pairs; and performing normalization processing on the noise points. The normalization processing includes: translating the noise points in the preoperative 3D model so that the distance difference between the translated noise points is less than a normalization threshold.

[0015] In some implementations, the target three-dimensional model is obtained by fusing the preoperative three-dimensional model and the intraoperative three-dimensional model by combining the registration relationship and the constraint point pairs, including: calculating the sum of spatial distances between point pairs between the preoperative three-dimensional model and the intraoperative three-dimensional model based on the registration relationship; finding the minimum value of the sum of spatial distances based on the energy minimization method; and fusing the preoperative three-dimensional model and the intraoperative three-dimensional model based on the point pairing method corresponding to the minimum value of the sum of spatial distances.

[0016] In some implementations, after displaying the target 3D model, the process further includes: receiving a doctor's fine-tuning instruction; adjusting the target 3D model based on the fine-tuning instruction; and displaying the adjusted target 3D model.

[0017] This specification also proposes a three-dimensional model display device based on model registration, comprising: a preoperative three-dimensional model construction module for constructing a preoperative three-dimensional model corresponding to a target site based on preoperative scan images; the location of lesions and / or blood vessels is marked in the preoperative three-dimensional model; an intraoperative three-dimensional model construction module for constructing an intraoperative three-dimensional model corresponding to the target site based on intraoperative scan images; a registration relationship determination module for determining the registration relationship between the preoperative three-dimensional model and the intraoperative three-dimensional model; the registration relationship is used to realize the mapping between models; a constraint point pair determination module for determining constraint point pairs between the preoperative three-dimensional model and the intraoperative three-dimensional model based on the registration relationship; the constraint point pairs are used to indicate the matching points between the preoperative three-dimensional model and the intraoperative three-dimensional model under the registration relationship; a model fusion module for fusing the preoperative three-dimensional model and the intraoperative three-dimensional model by combining the registration relationship and the constraint point pairs to obtain a target three-dimensional model; and a target three-dimensional model display module for displaying the target three-dimensional model; the target three-dimensional model is used to mark the location of lesions and / or blood vessels during surgery.

[0018] This specification also proposes a computer-readable storage medium storing a computer program / instruction thereon, which, when executed, implements the above-described method for displaying 3D models based on model registration.

[0019] As can be seen from the technical solutions provided in the embodiments of this specification above, the method constructs corresponding three-dimensional models of the target site based on preoperative and intraoperative scan images, respectively. Then, based on the spatial positions of the preoperative and intraoperative three-dimensional models, the registration relationship between the models is determined, and the spatial positions of the models are mapped. Based on the registration relationship, constraint point pairs between the preoperative and intraoperative three-dimensional models are further determined. Then, combining the fusion relationship and constraint point pairs, the preoperative and intraoperative three-dimensional models are fused to obtain the target three-dimensional model, which is then displayed, thereby enabling the identification of the location of lesions and blood vessels during surgery. Through this method, not only can a three-dimensional model of the target site be constructed intraoperatively, allowing doctors to intuitively grasp the morphology of the target site, but the location of blood vessels and lesions can also be identified, effectively guiding and assisting the doctor's needle insertion process, ensuring the accuracy of the surgical procedure, avoiding prolonged surgery, optimizing surgical outcomes, and improving the patient's treatment experience. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a 3D model display method based on model registration, as described in this specification.

[0022] Figure 2 This is a schematic diagram illustrating an embodiment of this specification of fitting a preoperative three-dimensional model using an average model;

[0023] Figure 3 This is a schematic diagram of an average model as described in this specification.

[0024] Figure 4 This is a schematic diagram illustrating one embodiment of obtaining a candidate lesion region according to this specification;

[0025] Figure 5 This is a schematic diagram illustrating one method of screening actual lesion areas according to an embodiment of this specification;

[0026] Figure 6 This is a schematic diagram of a blood vessel B-spline surface as an embodiment of this specification;

[0027] Figure 7 This is a schematic diagram of a three-dimensional blood vessel structure according to an embodiment of this specification;

[0028] Figure 8 This is a schematic diagram of a preoperative three-dimensional model as an embodiment of this specification;

[0029] Figure 9 This is a schematic diagram illustrating one embodiment of constructing a minimum bounding box according to this specification;

[0030] Figure 10 This is a schematic diagram illustrating the mapping between a preoperative three-dimensional model and an intraoperative three-dimensional model, as described in an embodiment of this specification.

[0031] Figure 11 This is a schematic diagram illustrating a smoothing process for an intraoperative three-dimensional model, as described in this specification.

[0032] Figure 12 This is a schematic diagram of a target three-dimensional model as an embodiment of this specification;

[0033] Figure 13 This is a module diagram of a three-dimensional model display device based on model registration, as described in this specification. Detailed Implementation

[0034] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0035] To address the aforementioned technical problems, this specification proposes a method for displaying 3D models based on model registration. The execution entity of this method can be a doctor's control terminal, image cart, or other computing device within a minimally invasive surgical environment. Figure 1 As shown, the 3D model display method based on model registration includes the following specific implementation steps.

[0036] S110: Construct a preoperative three-dimensional model corresponding to the target site based on the preoperative scan images; the location of the lesion and / or blood vessels is marked in the preoperative three-dimensional model.

[0037] Preoperative scan images are images obtained after scanning the target area before surgery. Specifically, the preoperative scan images can be, for example, CT images, MRI images, etc., without limitation. The target area can be, for example, a corresponding organ, tissue, or a region containing multiple organs or tissues. The preoperative scan images are used to show and describe the situation within the corresponding layer of the target area.

[0038] Typically, preoperative scan images are two-dimensional images, and a single preoperative scan image can only describe the situation within the corresponding section. Therefore, to obtain a more intuitive display effect, a corresponding preoperative three-dimensional model can be constructed based on the preoperative scan image to display the situation of the target area in a three-dimensional manner.

[0039] Generally, preoperative scan images can show the outline size and shape of the target area within the scan plane. The multiple preoperative scan images acquired during the scanning process usually have a specific interlayer spacing. By combining the external outline and interlayer spacing shown in the preoperative scan images, a rough preoperative 3D model can be constructed.

[0040] In some implementations, in order to ensure the authenticity of the constructed preoperative three-dimensional model, the preoperative three-dimensional model can be modified using a three-dimensional model constructed based on real sample data.

[0041] Specifically, a large number of sample image data corresponding to the target area can be acquired in advance, and an average model can be built and trained using these sample image data. The average model is used to reflect the external contour of the target area under normal circumstances, and can be used to cover the contour of most organs.

[0042] Then, the preoperative 3D model and the average model are fused. The average model is then used to fit the constructed preoperative 3D model, thereby eliminating unreasonable areas due to data errors and other factors during the construction of the preoperative 3D model, ensuring the realism of the preoperative 3D model, and improving the accuracy of subsequent processing. Figure 2 The diagram shown illustrates the fitting of the preoperative 3D model using an average model. The specific fitting process can be tailored to the specific needs of the application and will not be elaborated upon here.

[0043] The following is combined with Figure 3 The construction process of the average model will be further described. Given the spherical harmonic function... It is obtained by decomposing spherical functions defined on the unit sphere, and it possesses symmetry and orthogonality. Its matrix form is as follows:

[0044] .

[0045] Define organ model If we place its center point at the origin of the spherical coordinate system, then its spherical coordinate system function can be fitted by a finite number of discrete spherical harmonic functions:

[0046] ……………………………①.

[0047] Expressing equation ① in matrix form: ;( );in, It can be obtained using the least squares method, defined by n vertices uniformly sampled from the model surface: By utilizing the orthogonality and symmetry of spherical harmonic functions, equation ① can be transformed to obtain... Then the model is obtained. Given M training samples of the organ, each sample model is translated and scaled so that its center point coincides with the origin of the spherical coordinate system, resulting in a series of sample models. Then the average model of this organ is: .

[0048] To ensure the effectiveness of the model, the location of the lesion and / or blood vessels can be marked in the preoperative 3D model to facilitate the doctor's observation of the model and understanding of the surgical situation at the target site. The location of the lesion and / or blood vessels can also be represented in the model in a 3D format.

[0049] Specifically, the lesion area image and / or the blood vessel image can be identified in the preoperative scan images. Based on the identified lesion area image and / or blood vessel image, the above-mentioned method for constructing a preoperative three-dimensional model is used, along with the position of the lesion area and / or blood vessels in the model, to construct the three-dimensional structure of the lesion and / or the three-dimensional structure of the blood vessels.

[0050] In some implementations, candidate lesion areas can be identified in preoperative scan images based on changes in brightness. For human organs, if there is a lesion such as a tumor, the lesion area may have increased or decreased density. The difference in density will appear as brightness and darkness different from the surrounding normal tissue in scan images such as CT images.

[0051] Based on this principle, and with the help of gradient tools ( Within the contours already identified in the preoperative scan images, partial derivatives are calculated for each pixel. For normal tissue, the partial derivatives in both the x and y directions approach 0, while for lesion areas, their partial derivatives... or There will be a pixel that reaches its maximum value; this pixel is a point on the contour, and the direction perpendicular to its maximum partial derivative (i.e., the direction of the maximum rate of change of brightness) is the direction of the envelope of the lesion area contour at that point. Through this gradient calculation, all contour points and their envelope directions can be identified, thereby pre-drawing the contour of the lesion area in the global coordinate system.

[0052] like Figure 4 As shown, these are the candidate lesion regions identified in the preoperative scan image. These two candidate lesion regions can be labeled as candidate lesion region a and candidate lesion region b, respectively.

[0053] However, because natural cavities and other tissues can exhibit abnormal brightness and darkness in scanned images due to their density differing from normal tissues or organs, they can be misidentified as lesions by gradient tools, resulting in the identification of candidate lesion areas that may actually be normal areas. Therefore, it is possible to compare candidate lesion areas with reference images corresponding to normal target organs, and retain the true lesion areas based on the comparison results.

[0054] Specifically, an image database of normal organs can be pre-established. During comparison, image sequences of reference images of normal organs are retrieved from the database based on the target area, and these reference images are pre-delineated, i.e., areas of abnormal density are also delineated in the reference images. If areas of abnormal density are pre-delineated, the normal images and preoperative scan image sequences are sorted to form image pairs. For each image pair, coarse registration and translation are performed based on the organ contour using simple rotation / stretching, and the overlap rate of the envelope regions of the registered candidate lesions is calculated. If the overlap rate is greater than a certain threshold (e.g., 80%), it indicates that the candidate lesion region is only abnormal in brightness due to the density of normal tissue and does not belong to the distribution area of ​​lesions. This candidate lesion region can be excluded from the preoperative scan images. If the overlap rate is less than a certain threshold, the candidate lesion region does indeed show a difference from normal tissue due to brightness, and this lesion region can be retained as the actual lesion region. For example, if... Figure 5 As shown, after coarse registration and translation, the candidate lesion area in the preoperative scan image is compared with the density abnormal area in the reference image. It can be calculated that the overlap rate between lesion b and the density abnormal area in the normal image is >80%, while the overlap rate of lesion a is 0. Therefore, lesion a is retained and lesion b is excluded.

[0055] Correspondingly, the image corresponding to the actual lesion area can be segmented from it as the lesion area image, so as to complete the construction of the three-dimensional structure of the lesion in subsequent steps.

[0056] Similarly, to avoid the adverse situation of ruptured blood vessels during surgery when the constructed model is used to guide the operation, which would lead to a prolongation of operation time or even surgical risks, the location of blood vessels needs to be accurately displayed in the constructed preoperative 3D model.

[0057] To image blood vessels in a scanned image, CTA technology is required. This involves injecting a contrast agent into the vein before a preoperative CT scan, allowing the blood vessels to become visible. Correspondingly, the visualized image can be identified from the preoperative scan; this visualized image is the image presented after the injection of the contrast agent into the blood vessel. This method allows for the marking of blood vessel locations in CT and other scanned images, thereby effectively enabling image segmentation and the construction of blood vessel models.

[0058] Preferably, to make the blood vessel segmentation process faster and more accurate, segmentation training results based on neural networks can be used. Based on CT image sequences, blood vessel segmentation can be combined with B-spline basis functions to linearly combine the control vertex mesh, constructing a blood vessel surface in a global coordinate system. Neural network-based segmentation training can be performed by creating labels and then conducting UNET training. Figure 6The diagram shown illustrates the segmentation of blood vessel B-spline surfaces. The specific training process and image processing using B-spline basis functions can be tailored to the specific application requirements and will not be elaborated upon here.

[0059] Furthermore, to optimize the rendering effect of the 3D blood vessel structure, denoising processing can be applied. Specifically, the constructed B-spline surface of the blood vessel can be denoised using normal filtering, making the contour smoother and thus completing the construction of the 3D blood vessel structure in the global coordinate system. For example... Figure 7 The diagram illustrates the construction of blood vessel contours using normal filtering. The specific processing steps can be configured based on the needs of the actual application and will not be elaborated upon here.

[0060] After obtaining the overall preoperative three-dimensional model of the target site, as well as the three-dimensional structure of the lesion and / or blood vessels, these three-dimensional models can be combined to obtain the final preoperative three-dimensional model, thereby enabling the location of the lesion and blood vessels to be determined in the model.

[0061] Specifically, the model can be combined and generated based on the relative positions of the lesion's three-dimensional structure and the blood vessel's three-dimensional structure within the target site. For example... Figure 8 The image shown is a schematic diagram of the final preoperative 3D model obtained through synthesis. The transparency of the model can be adjusted to achieve a unified display of lesions, blood vessels, etc.

[0062] It should be noted that although the preoperative scan images are taken before the surgery is performed on the patient, the construction of the preoperative 3D model based on the preoperative scan images can be done before the surgery or during the surgery, without any restriction.

[0063] S120: Construct an intraoperative 3D model corresponding to the target site based on intraoperative scan images.

[0064] Since preoperative scans are acquired during the preoperative preparation phase, they cannot effectively reflect the condition of the target site or the insertion of surgical instruments during the operation. Therefore, in order to determine the real-time effect of the operation, intraoperative scans can be acquired during the operation when there is a need to determine the situation.

[0065] Similarly, an intraoperative 3D model corresponding to the target site can be constructed based on the intraoperative scan images. The process of constructing the intraoperative 3D model based on the intraoperative scan images can be referred to the description in step S110, and will not be repeated here.

[0066] Because it is impossible to visualize blood vessels during surgery, or there is insufficient time for computational analysis to sequentially analyze and determine the lesion area in intraoperative scans, the intraoperative 3D model can only reflect the approximate external outline of the target site. Therefore, it is necessary to fuse the preoperative and intraoperative 3D models in subsequent steps to represent the spatial location of the lesion and / or blood vessels in the intraoperative 3D model, facilitating surgical execution and determining the current surgical outcome.

[0067] S130: Determine the registration relationship between the preoperative 3D model and the intraoperative 3D model; the registration relationship is used to realize the mapping between the models.

[0068] Registration relationships are used to quantitatively reflect the transformation between the preoperative 3D model and the intraoperative 3D model. Since the preoperative and intraoperative 3D models are mainly used to reflect the corresponding spatial morphology, and their coordinate systems may also differ, registration relationships can be determined first. These registration relationships can then quantitatively reflect the mapping relationship between the models, thereby effectively achieving model fusion in subsequent steps.

[0069] In some implementations, when determining the registration relationship, coarse registration can be performed on the preoperative 3D model and the intraoperative 3D model first. The coarse registration is used to define the spatial position of the preoperative 3D model and the intraoperative 3D model.

[0070] Specifically, a minimum bounding box with orientation can be created for both the preoperative and intraoperative 3D models. This minimum bounding box can be a minimal cubic structure capable of containing the 3D model. Then, for the minimum bounding boxes corresponding to the preoperative and intraoperative 3D models, the angles between each side of the bounding box and the coordinate axes, as well as the centroid coordinates of the bounding box, are obtained. Coordinate transformation is then applied to ensure that the two bounding boxes are in essentially the same spatial position. That is, the spatial positions of the preoperative and intraoperative 3D models are defined by the same minimum bounding box, thereby reducing the computational load in subsequent steps and improving accuracy. Specifically, the OBB registration method can be used to complete the above coarse registration process. Figure 9 The diagram shown is a schematic of the established minimum bounding box, in which the preoperative 3D model and the intraoperative 3D model are confined to the same region in space to facilitate the registration between models in subsequent steps.

[0071] After coarse registration, fine registration can be performed between the models to obtain the transformation matrix between the preoperative and intraoperative 3D models. The transformation matrix describes the transformation relationship between the preoperative and intraoperative 3D models, enabling the conversion of spatial point coordinates between them using a fixed transformation function or formula. Specifically, the transformation matrix between the models can be calculated based on the spatial coordinates of each point on the preoperative and intraoperative 3D models after coarse registration.

[0072] Specifically, based on the spatial relationship between points on the preoperative 3D model and the intraoperative 3D model after coarse registration, corresponding registration points are determined; the transformation matrix is ​​obtained according to the spatial coordinates of the registration points; the transformation matrix is ​​used to ensure that the distance difference between the transformed registration points is less than the normalization threshold.

[0073] To illustrate the above process using a concrete example, after completing the coarse registration, let {Q} be the target point set corresponding to the intraoperative 3D model, and {P} be the source point set corresponding to the preoperative 3D model. For each point in {P}... You can find a little bit of it in {Q}. ,make and The shortest distance is achieved by using rotation matrix R and translation matrix T such that each point in {P}, after the transformation by R / T, approximately coincides with the corresponding point in {Q}. Let there be an objective function. Iteratively calculate the average distance between corresponding points in {P'} and {Q} after the RT transformation, making it less than a certain threshold. The RT matrix that satisfies the objective threshold is the transformation matrix. For example... Figure 10 The diagram shown illustrates the transformation matrix between the preoperative 3D model and the intraoperative 3D model.

[0074] S140: Determine the constraint point pairs between the preoperative 3D model and the intraoperative 3D model based on the registration relationship; the constraint point pairs are used to indicate the matching points between the preoperative 3D model and the intraoperative 3D model under the registration relationship.

[0075] In practical applications, although the transformation matrix is ​​obtained after the calculation process in step S130, it cannot be guaranteed that all points between models have a good correspondence. That is, only some points may perfectly conform to the transformation relationship of the transformation matrix, while other points may have deviations when transformed based on the transformation matrix. Therefore, the above registration relationship can be combined to determine the constraint point pairs, and the transformation relationship between models can be optimized by combining the constraint point pairs.

[0076] To achieve the above effect, constraint points and noise points between the preoperative 3D model and the intraoperative 3D model can be identified. Noise points are those points that cannot achieve the desired transformation effect based on the transformation matrix. Since the main difference between constraint points and noise points lies in whether the registration relationship can be satisfied to a certain extent, constraint points and noise points can be distinguished based on the accuracy of the registration.

[0077] Specifically, based on the registration relationship, such as the corresponding transformation matrix, the residual value of the target transformation function between the point pairs of the preoperative 3D model and the intraoperative 3D model can be calculated. The specific method for calculating the residual value can be set according to the actual application requirements, which will not be elaborated here.

[0078] After obtaining the residual values ​​of the point pairs, the residual values ​​can be compared with the matching threshold. Point pairs with residual values ​​less than the matching threshold are identified as constraint point pairs, while point pairs with residual values ​​not less than the matching threshold are identified as noise points.

[0079] The matching threshold is a pre-set value used to limit the degree of matching between point pairs. The specific value of the matching threshold can be adjusted based on the actual calculation process.

[0080] To illustrate this, let's use a concrete example, such as... Figure 11 The image shows the result after mapping the preoperative and intraoperative 3D models using a transformation matrix. Points that completely overlap after transformation are selected as constraint points (e.g., in the left image). , , , Points). For pairs of points that cannot completely overlap (as shown in the figure). , ), which is the corresponding noise.

[0081] S150: Combine the registration relationship and the constraint point pair to fuse the preoperative 3D model and the intraoperative 3D model to obtain the target 3D model.

[0082] After distinguishing between constraint point pairs and noise points in the above steps, normalization processing can be performed on the noise points. Specifically, normalization processing may include translating the noise points in the preoperative 3D model so that the distance difference between the translated noise points is less than the normalization threshold.

[0083] For example, such as Figure 11 As shown, point F on the source model can be translated radially to point [point name missing]. Make the points match (as shown in the figure). , The distances between the vertex constraints and the model are all less than a certain threshold. This threshold can be set based on the needs of the actual application, thus obtaining a model that has been smoothed using Laplacian with vertex constraints.

[0084] After smoothing, the maximum distance difference between the front-end and back-end model point pairs is less than the normalization threshold, and from the image perspective, noise (peaks, spikes, etc.) has been eliminated, thus avoiding noise fusion, improving fusion accuracy, and ensuring the effectiveness of model utilization in subsequent processes.

[0085] In practical applications, other methods can also be used to improve the correspondence between noise points, not limited to the examples above, which will not be elaborated here.

[0086] Based on the above process, the conversion and registration relationship between the preoperative 3D model and the intraoperative 3D model was effectively determined, the mapping relationship between the preoperative 3D model and the intraoperative 3D model was quantitatively described, and the mapping effect between the models was improved, which is beneficial to the subsequent fusion and display of the models.

[0087] After obtaining the registration relationship, based on the transformation between models reflected by the registration relationship, the preoperative 3D model and the intraoperative 3D model can be fused to obtain the final target 3D model.

[0088] In some implementations, the fusion process may first calculate the sum of spatial distances between points in the preoperative 3D model and the intraoperative 3D model based on the registration relationship, then find the minimum value of the sum of spatial distances based on the energy minimization method, and finally fuse the preoperative 3D model and the intraoperative 3D model based on the point pairing method corresponding to the minimum value of the sum of spatial distances.

[0089] To illustrate this with a concrete example, when fusing models, radial basis functions can be used to minimize the energy of the two models. First, for each point pair in the preoperative 3D model and the intraoperative 3D model... and The distance can be expressed as The form; then set an objective function, for To perform summation, that is: Then, using the energy minimization method, we obtain... minimum value Ultimately, we obtain the point-to-point pairing method under the energy minimization condition, which is the final elastic fusion model.

[0090] like Figure 12 The image shown is a schematic diagram of the fused target 3D model. This target 3D model effectively displays the spatial location of the lesion area and blood vessels, providing guidance for the surgical procedure and ensuring the effectiveness of the surgeon's operation.

[0091] S160: Display the target 3D model; the target 3D model is used to identify the location of the lesion and / or blood vessels during the operation.

[0092] Once the target 3D model is obtained, it can be displayed to identify the location of lesions and / or blood vessels during surgery, assisting doctors in performing corresponding surgical procedures.

[0093] Preferably, after generating the target 3D model, it can first be shown to the doctor for confirmation. If the doctor confirms that there are no problems with the target 3D model, then the target 3D model can be used in actual surgical applications.

[0094] Furthermore, in some implementations, doctors can fine-tune the target 3D model based on their experience or surgical needs. Specifically, they can input corresponding fine-tuning commands to adjust specific parts of the model. After receiving the fine-tuning command, the doctor can execute the corresponding adjustment operation to adjust the target 3D model and then re-display the adjusted target 3D model to better meet the needs of the actual application.

[0095] To achieve better application results, doctors can also rotate, scale, and view cross-sections of the target 3D model, thereby providing better guidance for the actual surgical procedure.

[0096] Based on the above embodiments and scenario examples, it can be seen that the method constructs corresponding 3D models of the target site for preoperative and intraoperative scan images, respectively. Then, based on the spatial positions of the preoperative and intraoperative 3D models, the registration relationship between the models is determined, and the spatial positions of the models are mapped. Based on the registration relationship, constraint point pairs between the preoperative and intraoperative 3D models are further determined. Finally, the preoperative and intraoperative 3D models are fused together using the fusion relationship and constraint point pairs to obtain the target 3D model, which is then displayed. This allows for the identification of the location of lesions and blood vessels during surgery. Through this method, not only can a 3D model of the target site be constructed intraoperatively, enabling doctors to intuitively grasp the morphology of the target site, but the location of blood vessels and lesions can also be identified, effectively guiding and assisting the doctor's needle insertion process, ensuring the accuracy of the surgical procedure, avoiding prolonged surgery, optimizing surgical outcomes, and improving the patient's treatment experience.

[0097] based on Figure 1 The corresponding 3D model display method based on model registration is also described in the embodiments of this specification, along with a 3D model display device based on model registration. Figure 13 As shown, the 3D model display device based on model registration may include the following modules.

[0098] The preoperative three-dimensional model construction module 1310 is used to construct a preoperative three-dimensional model corresponding to the target site based on the preoperative scan image; the location of the lesion and / or blood vessel is marked in the preoperative three-dimensional model.

[0099] The intraoperative 3D model construction module 1320 is used to construct an intraoperative 3D model corresponding to the target site based on intraoperative scan images.

[0100] The registration relationship determination module 1330 is used to determine the registration relationship between the preoperative three-dimensional model and the intraoperative three-dimensional model; the registration relationship is used to realize the mapping between the models.

[0101] The constraint point pair determination module 1340 is used to determine the constraint point pairs between the preoperative 3D model and the intraoperative 3D model based on the registration relationship; the constraint point pairs are used to indicate the matching points between the preoperative 3D model and the intraoperative 3D model under the registration relationship.

[0102] The model fusion module 1350 is used to combine the registration relationship and the constraint point pair to fuse the preoperative three-dimensional model and the intraoperative three-dimensional model to obtain the target three-dimensional model.

[0103] The target 3D model display module 1360 is used to display the target 3D model; the target 3D model is used to identify the location of lesions and / or blood vessels during surgery.

[0104] based on Figure 1 The corresponding 3D model display method based on model registration is provided in this specification through embodiments of a computer-readable storage medium storing computer programs / instructions. The computer-readable storage medium can be read by a processor via the device's internal bus, and the processor can then implement the program instructions in the computer-readable storage medium.

[0105] In this embodiment, the computer-readable storage medium can be implemented in any suitable manner. The computer-readable storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), memory card, etc. The computer storage medium stores computer program instructions. When the computer program instructions are executed, this specification is implemented. Figure 1 The program instructions or modules corresponding to the embodiments.

[0106] It should be noted that the above-mentioned three-dimensional model display method, device and storage medium based on model registration can be applied to the field of surgical positioning technology, and can also be applied to other technical fields without limitation.

[0107] Although the process described above includes multiple operations that occur in a specific order, it should be clearly understood that these processes may include more or fewer operations that can be executed sequentially or in parallel (e.g., using parallel processors or a multithreaded environment).

[0108] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0113] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0114] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0116] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments in this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0117] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for displaying 3D models based on model registration, characterized in that, include: A preoperative 3D model corresponding to the target area is constructed based on the preoperative scan images; The location of the lesion and / or blood vessels is marked in the preoperative three-dimensional model; Construct an intraoperative 3D model corresponding to the target site based on intraoperative scan images; Determine the registration relationship between the preoperative 3D model and the intraoperative 3D model; the registration relationship is used to achieve mapping between the models. Based on the registration relationship, the constraint point pairs between the preoperative 3D model and the intraoperative 3D model are determined; The constraint point pairs are used to indicate the matching points between the preoperative 3D model and the intraoperative 3D model under the registration relationship; The target 3D model is obtained by fusing the preoperative 3D model and the intraoperative 3D model by combining the registration relationship and the constraint point pair. The target 3D model is displayed; the target 3D model is used to identify the location of the lesion and / or blood vessels during the operation; The step of constructing a preoperative three-dimensional model corresponding to the target site based on preoperative scan images includes: Candidate lesion areas are identified in preoperative scan images based on changes in brightness; The preoperative scan image is compared with a reference image to screen actual lesion areas from candidate lesion areas; wherein the process includes: outlining areas of abnormal density from the reference image; determining the degree of overlap between the reference image and the areas of abnormal density in the candidate lesion areas; and determining the candidate lesion areas as actual lesion areas if the degree of overlap is less than a comparison threshold. Segmenting the image corresponding to the actual lesion region as the lesion region image; and / or, Identify contrast images from preoperative scan images; the contrast images are those that appear after the contrast agent is injected into the blood vessel.

2. The 3D model display method based on model registration as described in claim 1, characterized in that, The step of constructing a preoperative three-dimensional model corresponding to the target site based on preoperative scan images includes: Identify vascular images in preoperative scan images; Based on the location of the lesion area image and / or blood vessel image, construct the 3D structure of the lesion and / or blood vessel at the corresponding location in the preoperative 3D model.

3. The 3D model display method based on model registration as described in claim 2, characterized in that, The construction of the 3D vascular structure at the corresponding location in the preoperative 3D model based on the location of the vascular image includes: The constructed three-dimensional vascular structure is subjected to noise reduction processing; the noise reduction processing includes surface normal filtering processing.

4. The 3D model display method based on model registration as described in claim 1, characterized in that, The step of constructing a preoperative three-dimensional model corresponding to the target site based on preoperative scan images includes: Obtain an average model trained based on sample image data; the average model is used to reflect the contour of the target region; The constructed preoperative three-dimensional model was fitted using the average model.

5. The 3D model display method based on model registration as described in claim 1, characterized in that, Determining the registration relationship between the preoperative 3D model and the intraoperative 3D model includes: Coarse registration is performed on the preoperative 3D model and the intraoperative 3D model; the coarse registration is used to define the spatial position of the preoperative 3D model and the intraoperative 3D model; wherein, it includes: establishing a minimum bounding box adapted to the spatial position of the preoperative 3D model and the intraoperative 3D model; the minimum bounding boxes corresponding to the preoperative 3D model and the intraoperative 3D model are located in the same spatial position; Based on the spatial coordinates of each point on the preoperative 3D model and the intraoperative 3D model after coarse registration, the transformation matrix between the models is calculated. This includes: determining the corresponding registration points based on the spatial relationship between each point on the preoperative 3D model and the intraoperative 3D model after coarse registration; and obtaining the transformation matrix based on the spatial coordinates of the registration points. The transformation matrix is ​​used to ensure that the distance difference between the transformed registration points is less than a normalization threshold.

6. The 3D model display method based on model registration as described in claim 1, characterized in that, The step of determining the constraint point pairs between the preoperative 3D model and the intraoperative 3D model based on the registration relationship includes: The residual value of the target transformation function between the point pairs of the preoperative 3D model and the intraoperative 3D model is calculated based on the registration relationship; Point pairs with residual values ​​less than the matching threshold are identified as constraint point pairs.

7. The 3D model display method based on model registration as described in claim 1, characterized in that, The process of fusing the preoperative 3D model and the intraoperative 3D model by combining the registration relationship and the constraint point pairs to obtain the target 3D model includes: Identify noise points between the preoperative 3D model and the intraoperative 3D model, excluding constraint point pairs; The noise points are normalized; the normalization process includes: translating the noise points in the preoperative 3D model so that the distance difference between the translated noise points is less than the normalization threshold.

8. The 3D model display method based on model registration as described in claim 1, characterized in that, The process of fusing the preoperative 3D model and the intraoperative 3D model by combining the registration relationship and the constraint point pairs to obtain the target 3D model includes: The sum of spatial distances between point pairs between the preoperative 3D model and the intraoperative 3D model is calculated based on the registration relationship; The minimum value of the sum of spatial distances is obtained based on the energy minimization method; The preoperative 3D model and the intraoperative 3D model are fused based on the point pairing method corresponding to the minimum sum of spatial distances.

9. A three-dimensional model display device based on model registration, characterized in that, include: The preoperative 3D model building module is used to build a preoperative 3D model corresponding to the target site based on the preoperative scan images; The location of the lesion and / or blood vessels is marked in the preoperative three-dimensional model; The intraoperative 3D model construction module is used to construct an intraoperative 3D model corresponding to the target site based on intraoperative scan images; The registration relationship determination module is used to determine the registration relationship between the preoperative 3D model and the intraoperative 3D model; The registration relationship is used to achieve mapping between models; The constraint point pair determination module is used to determine the constraint point pairs between the preoperative 3D model and the intraoperative 3D model based on the registration relationship; The constraint point pairs are used to indicate the matching points between the preoperative 3D model and the intraoperative 3D model under the registration relationship; The model fusion module is used to combine the registration relationship and the constraint point pair to fuse the preoperative 3D model and the intraoperative 3D model to obtain the target 3D model. The target 3D model display module is used to display the target 3D model; the target 3D model is used to identify the location of lesions and / or blood vessels during surgery; The step of constructing a preoperative three-dimensional model corresponding to the target site based on preoperative scan images includes: Candidate lesion areas are identified in preoperative scan images based on changes in brightness; The preoperative scan image is compared with a reference image to screen actual lesion areas from candidate lesion areas; wherein the process includes: outlining areas of abnormal density from the reference image; determining the degree of overlap between the reference image and the areas of abnormal density in the candidate lesion areas; and determining the candidate lesion areas as actual lesion areas if the degree of overlap is less than a comparison threshold. Segmenting the image corresponding to the actual lesion region as the lesion region image; and / or, Identify contrast images from preoperative scan images; the contrast images are those that appear after the contrast agent is injected into the blood vessel.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed, it implements the three-dimensional model display method based on model registration as described in any one of claims 1-8.

Images