Approximate visualization method and system for matching degree of three-dimensional model and two-dimensional image

Abstract

The invention relates to the technical field of image matching degree visualization, and provides a three-dimensional model and two-dimensional image matching degree approximate visualization method, which comprises the following steps of: S1, generating a relocation point for each spatial point of a reconstructed three-dimensional model by adopting a mode of combining transverse expansion with parallel projection or a mode of combining longitudinal compression with perspective projection; s2, matching the relocation point with a projection point on the two-dimensional image to generate a 3D visual effect picture of the matching degree of the three-dimensional model and the two-dimensional image; and S3, calculating a projection contour edge of the three-dimensional model, and projecting the projection contour edge to generate a contour line so as to generate a 2D visual effect picture of the matching degree of the three-dimensional model and the two-dimensional image. Three technical effects are realized: 1, 3D space visualization of the matching degree of the 3D model and the physical fan beam scanning full-length image is realized; 2, the approximate 3D space visualization of the small cone beam scanning full-length image matching degree is realized, and the hardware compatibility is good; and thirdly, approximate 2D space visualization of the matching degree is realized, so that the calculation time complexity is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of visualization of image matching degree, and in particular to a method and system for approximately visualizing the matching degree between a three-dimensional model and a two-dimensional image. Background Art

[0002] In the field of clinical DR, due to the limited size of the flat panel detector, in order to obtain full-length images of a patient's lower extremities or spine, a method of taking multiple shots and then performing stitching post-processing is usually adopted: the light source and the flat panel are sequentially shot at 2-3 different heights, and an image post-processing algorithm stitches and fuses multiple images into a single full-length image. Since the projection method is point light source projection, oblique rays far from the center of the flat panel will introduce geometric distortion when projecting objects with a certain thickness that are not closely attached to the flat panel. This geometric distortion will cause difficulties in stitching the upper and lower images, and structures at different depths cannot be fully aligned, resulting in double-image problems in the stitched image. Therefore, when using this method for shooting, radiographers will require the patient to stand closely against the flat panel. In addition, since the thickness of the human body in the front-back direction is narrower than that in the left-right direction, it is relatively easier to process the stitching of anteroposterior images than that of lateral images.

[0003] To solve the above problems, X-ray slot scanning came into being. Its main principle is to suppress the longitudinal cone angle during physical projection, so that the physical projection changes from a conical beam to a fan beam projection that is closer to a fan beam. In this way, the longitudinal oblique rays will be reduced, the geometric distortion will also be reduced, the stitching will be relatively easier, and the requirements for the patient's body thickness and standing position will be more relaxed.

[0004] According to the actual longitudinal cone angle in physics, slot scanning can be further divided into two categories: The first category is ideal fan beam projection, represented by the French company EOS. It uses a self-developed linear detector, and the single-shot imaging height of this detector is one pixel, and the longitudinal cone angle has been suppressed to the theoretical minimum. The second category is to use small cone beam projection engineering approximation fan beam projection, mainly represented by the Japanese company Shimadzu. Although the linear detector of EOS has advantages in obtaining full-length images, it is not as convenient and fast as a common flat panel detector for single-shot exposure when obtaining local images. This leads to a single shooting mode for EOS's imaging products. Therefore, the common flat panel detector solution adopted by Shimadzu can meet the requirements for taking plain films during daily clinical use. When it is necessary to shoot a full-length slot scanning image, by using a collimator combined with the flat panel detector central strip ROI technology, a strip image with a height of a few millimeters is obtained for stitching at a single time. In this way, the device has a richer shooting mode and a wider clinical application scenario.

[0005] After obtaining full-length images with a single-sided or double-sided perspective, a 3D reconstruction algorithm can reconstruct the structures of interest from one or two images. Compared with CT scans, this algorithm can recover a large range of 3D structures with a very low radiation dose, which has important application values in preoperative diagnosis, planning, and postoperative rehabilitation evaluation. In order to visualize the matching degree between the reconstructed 3D structure and the 2D full-length image, and show the mismatched areas for users to interactively adjust accordingly, it is necessary to develop a visualization and interaction algorithm for the matching degree between the 3D structure and the 2D full-length image.

[0006] The currently relatively mature visualization solution is the sterEOS 3D station of EOS France. This post-processing workstation software can import the full-length anteroposterior and lateral X-ray images of EOS. EOS's line-scan imaging system uses a 45-cm-wide linear radiation source and detector. Each exposure obtains a pair of anteroposterior and lateral linear images with a height of one pixel. This physical projection mode is very similar to the fan-beam scanning in CT plain scans. Through the vertical movement of the gantry, the system obtains linear images at different heights and simply performs longitudinal stacking in the post-processing stitching algorithm to finally obtain a pair of anteroposterior and lateral matching images with the same number of pixel rows in the longitudinal direction.

[0007] After importing the anteroposterior and lateral images into the sterEOS 3D workstation and making several annotations, the software will reconstruct the 3D model of interest to the doctor. In order to visualize the effect and matching degree of the reconstructed model, EOS has adopted a 2D visualization method, that is, instead of showing the model itself, only the contour lines of the model are projected onto the 2D full-length image and drawn into the image in the form of colored lines, as Figure 1 shown.

[0008] EOS adopts this 2D visualization mainly for the convenience of subsequent manual adjustment by users. As can be seen from Figure 1 , in the anteroposterior view, there is a large mismatch between the left transverse process of the model and the image; similarly, in the lateral view, there is an obvious mismatch in the pedicle of the model, indicating that there are still large errors in the reconstruction of the model reconstruction algorithm in these two places. There are control handles distributed on the projection contour line of the software, which is convenient for users to manually drag the parts with large mismatches to the correct positions in the corresponding view, so as to further adjust the model. It is an interactive-visualization method that interacts with 2D but modifies the calculation in the background for the 3D model.

[0009] Another advantage of this 2D visualization is its simplicity in implementation: Since the physical scan is a fan-beam projection horizontally and a parallel movement vertically, and EOS's linear detector ensures a high degree of consistency in height between the actual physical projection and the ideal mathematical model projection. And the mathematical formula of this projection is known and determined, so as long as the projection of the contour line is calculated and drawn in the 2D image, this visualization is completed.

[0010] As shown Figure 2 in the figure, SterEOS also provides a 3D window interface for modeling, but only the model is placed in this interface, and no image texture is placed. Therefore, it can only be regarded as a 3D visualization of the 3D model at most, and there is no 3D visualization of the fitting degree between the model and the image.

[0011] There are mainly two problems with this visualization technology of EOS: (1) The visualization technology of EOS cannot be directly applied to the full-length image stitched from small cone-beam strip images: The mathematical projection model used behind the visualization is closely related to the physical projection model used during actual imaging. The linear detector used by EOS is a special gas detector, and the principle of this detector was proposed by Georges Charpak in 1922, for which he won the Nobel Prize. This detector has only one pixel in the height direction, making the actual scanning process very close to the ideal fan-beam + translation projection method. This detector is exclusive to EOS and is protected by patents. Therefore, if other manufacturers want to achieve a similar fan-beam scanning effect, they can only find another way. For example, Shimadzu Corporation in Japan obtains the image of the central slit narrow strip by using a collimator combined with flat ROI photosensitivity. However, each strip is not one pixel high and still has hundreds of pixels. Therefore, this implementation method essentially uses small cone-angle images to approximately replace the ideal fan-beam images. So, due to the difference in the actual physical projection method, the fan-beam projection mathematical model of EOS cannot be directly applied to the full-length image stitched from small cone-beam strip images. In addition, companies like Shimadzu in Japan do not provide the function of 3D reconstruction, and naturally do not study the visualization of the matching degree between the 3D model and the 2D small cone-beam image. Therefore, how to achieve the visualization of the matching between the model and the image based on the full-length image stitched from physical small cone-beam strip images is a brand-new engineering problem.

[0012] (2) EOS only provides 2D visualization. Although 2D visualization has its advantages, in essence, it still flattens the three-dimensional shape onto a 2D image. This visualization process discards a lot of structural information of the model and only retains the contour for matching with the 2D image. Therefore, due to the dimensionality reduction operation, 2D visualization abandons the natural advantages of the 3D scene and sacrifices the fact that the reconstructed data is 3D. Since the projection process occurs in 3D space, there should be a corresponding 3D visualization to display the matching relationship in 3D space. Summary of the Invention

[0013] In view of the above problems, the object of the present invention is to provide a method and system for approximately visualizing the matching degree between a three-dimensional model and a two-dimensional image, and the method and system achieve three technical effects: one is the 3D spatial visualization of the matching degree between the 3D model and the full-length image of physical fan-beam scanning; the second is the approximate 3D spatial visualization of the matching degree with the full-length image of small cone-beam scanning, which has good hardware compatibility; the third is the approximate 2D spatial visualization of the matching degree, which greatly reduces the computational time complexity.

[0014] The above object of the present invention is achieved by the following technical solutions: A method for approximately visualizing the matching degree between a three-dimensional model and a two-dimensional image includes the following steps: S1: For the reconstructed three-dimensional model, generate repositioning points for each spatial point in the three-dimensional model by using the method of combining lateral dilation with parallel projection or the method of combining longitudinal compression with perspective projection; S2: Use the generated repositioning points to match the projection points on the two-dimensional image corresponding to the three-dimensional model, and generate a 3D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image; S3: Calculate the projection contour edges of the three-dimensional model, and project the projection contour edges to generate contour lines, so as to generate a 2D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image.

[0015] Further, in step S1, when generating repositioning points for each spatial point in the three-dimensional model by using the method of combining lateral dilation with parallel projection, specifically: In the x direction, based on the cross-section of any height of the three-dimensional model, perform the above-mentioned lateral dilation on each spatial point on the current cross-section to obtain the repositioning point corresponding to the current spatial point. The repositioning point has the same depth as the original spatial point, but under the action of parallel beam projection, it can obtain the same projection point on the two-dimensional image as the original spatial point under the action of fan-beam rays; In the y direction, use parallel beam projection to ensure the matching in the y direction between the two-dimensional image and the three-dimensional model.

[0016] Further, regarding the method of combining lateral dilation with parallel projection, it also includes: When the pose or shape of the three-dimensional model in space changes, regenerate the repositioning points for the spatial points in the three-dimensional model; Calculate the repositioning points based on the two-dimensional images corresponding to the anterior and lateral views of the three-dimensional model respectively; When the position of the foot of the perpendicular from the light source to the flat panel detector changes, regenerate the repositioning points for several spatial points in the three-dimensional model.

[0017] Further, in step S1, a relocalization point is generated for each spatial point in the reconstructed three-dimensional model by using longitudinal compression in combination with perspective projection, specifically as follows: In the x direction, perspective projection is used to ensure that the x direction of the two-dimensional image matches the three-dimensional model; In the y direction, based on any longitudinal section of the width of the three-dimensional model, each spatial point on the current longitudinal section is respectively longitudinally compressed to obtain the relocalization point corresponding to the current spatial point, and the relocalization point is in the same position as the original spatial point under the action of perspective projection in the fan beam / parallel beam projection.

[0018] Further, in step S3, the projection contour edge of the three-dimensional model is calculated, and the projection contour edge is projected to generate a contour line to generate a 2D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image, specifically as follows: An operation of combining lateral expansion with parallel projection is applied to the three-dimensional model, and the projection contour edge of the original three-dimensional model is obtained on the expanded three-dimensional model; For each projection contour edge, the three-dimensional model is projected using a fan beam to generate a contour line.

[0019] Further, an operation of combining lateral expansion with parallel projection is applied to the three-dimensional model, and the projection contour edge of the original three-dimensional model is obtained on the expanded three-dimensional model, specifically as follows: The projection contour edge of the three-dimensional model is calculated, and an operation of combining lateral expansion with parallel projection is applied to the three-dimensional model. The condition for the edge of the original three-dimensional model to be a contour edge is that the orientations of the two adjacent faces of the corresponding edge of the expanded three-dimensional model are opposite to the direction of the parallel beam projection ray.

[0020] Further, for each projection contour edge, the three-dimensional model is projected using a fan beam to generate a contour line, specifically as follows: For the two endpoints of the projection contour edge, two projection points are respectively calculated, and the line connecting the two projection points is the contour line; Among them, when calculating the x coordinate of the projection point, the magnification is considered using similar triangles, and when calculating the y coordinate of the projection point, after parallel projection, the pixel size is used to convert it into pixel coordinates.

[0021] A three-dimensional model and two-dimensional image matching degree approximate visualization system for performing the three-dimensional model and two-dimensional image matching degree approximate visualization method as described above, including: A spatial point relocation module, used to generate a relocation point for each spatial point in the reconstructed three-dimensional model by adopting a method of lateral expansion combined with parallel projection or a method of longitudinal compression combined with perspective projection; A 3D visualization module, used to match the generated relocation points with projection points on the two-dimensional image corresponding to the three-dimensional model, and generate a 3D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image; The 2D visualization module is used to calculate the projection contour edge of the three-dimensional model and project the projection contour edge to generate a contour line to generate a 2D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image.

[0022] A computer device comprises a memory and one or more processors, wherein the memory stores computer codes, and when the computer codes are executed by the one or more processors, the one or more processors execute the above method.

[0023] A computer-readable storage medium stores computer codes. When the computer codes are executed, the above method is executed.

[0024] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) Spatial visualization of the matching degree between the 3D model and the full-length image using physical fan beam scanning: This technical effect can display the matching effect between the 3D model and the EOS full-length image in 3D space. This technical effect makes up for the defect that the EOS built-in post-processing software only has 2D visualization but no 3D visualization.

[0025] (2) Approximate spatial visualization 3D method for the matching degree between 3D models and full-length images scanned using small cone beams: This effect can display the matching effect between 3D models and full-length images scanned using small cone beams in 3D space. Compared with 1, this technology has a higher tolerance for physical scanning and is more compatible with hardware solutions from various manufacturers.

[0026] (3) Approximate spatial visualization 2D method for the matching degree between 3D models and full-length images scanned using small cone beams: This effect uses a fan beam projection mathematical model engineering to approximate the simulation of multiple physical small cone beam projection processes. Compared with the most primitive cone beam projection calculation method of regression slices, it can reduce the calculation time complexity of 2D contour lines by 90-100 times. Its essence is to replace multiple cone beam projections with a single fan beam projection. Its time complexity is as low as EOS, but its approximation effect is very good. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1Schematic diagram of the 2D visualization method for the EOS contour line projection; Figure 2 Schematic diagram of the EOS 3D model visualization; Figure 3 Overall flowchart of the method for approximately visualizing the matching degree between the 3D model and the 2D image of the present invention; Figure 4 Schematic diagram of the lateral expansion principle of the present invention; Figure 5 Schematic diagram of the lateral expansion principle of the present invention considering the actual space calibration information; Figure 6 Schematic diagram of the longitudinal compression principle of the present invention; Figure 7 Schematic diagram of the present invention using the physical fan-beam full-length images in the anterior-posterior view; Figure 8 Schematic diagram of the present invention using the small cone-beam full-length images in the anterior-posterior view; Figure 9 Schematic diagram of the L3-L5 lumbar vertebrae and 4 key points segmented by CT of the present invention; Figure 10 Schematic diagram of the annotation of key points on the physical fan-beam full-length image of the present invention; Figure 11 Annotation of key points on the small cone-beam full-length image of the present invention; Figure 12 Frontal view spatial matching effect diagram of the 3D model and the physical fan-beam full-length image of the present invention; Figure 13 Lateral view spatial matching effect diagram of the 3D model and the physical fan-beam full-length image of the present invention; Figure 14 Frontal view spatial matching effect diagram of the 3D model and the small cone-beam full-length image of the present invention; Figure 15 Lateral view spatial matching effect diagram of the 3D model and the small cone-beam full-length image of the present invention; Figure 16 2D visualization frontal view effect diagram of the fast contour line projection algorithm of the present invention on the small cone-beam full-length image; Figure 17 2D visualization lateral view effect diagram of the fast contour line projection algorithm of the present invention on the small cone-beam full-length image; Figure 18 Overall structure diagram of the system for approximately visualizing the matching degree between the 3D model and the 2D image of the present invention. Specific implementation manner

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Apparently, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the protection scope of this application.

[0029] Those skilled in the art of this technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups.

[0030] First Embodiment As Figure 3 shown, this embodiment provides a method for approximately visualizing the matching degree between a three-dimensional model and a two-dimensional image, including the following steps: S1: For the reconstructed three-dimensional model, generate repositioning points for each spatial point in the three-dimensional model by using a method of combining horizontal dilation with parallel projection or a method of combining vertical compression with perspective projection.

[0031] As mentioned above, the full-length image of EOS can be described by a physical (mathematical) process of longitudinal translation and horizontal fan-beam scanning. The horizontal fan-beam projection and the perspective projection are the same in the x-direction processing. The longitudinal translation results in a 1:1 projection in the longitudinal direction, which is equivalent to the processing method of parallel projection (orthogonal projection). Therefore, this process is essentially a hybrid projection method with different projection methods in the x and y directions. However, the common projection methods provided by general visualization libraries are perspective projection and parallel projection. The former is used to visualize the real world, showing objects larger when closer and smaller when farther away, and the latter is commonly used in industrial 3D software, where objects are projected onto the screen and shown to the user with the same size regardless of distance. Therefore, directly applying these two projections cannot correctly visualize the matching between the 3D model and the full-length image.

[0032] In this embodiment, a method of generating repositioning points for each spatial point in the reconstructed three-dimensional model by combining lateral expansion with parallel projection is proposed. Specifically: in the x direction, based on the cross-section of any height of the three-dimensional model, each spatial point on the current cross-section is laterally expanded to obtain the repositioning point corresponding to the current spatial point. The repositioning point has the same depth as the original spatial point, but under the action of parallel beam projection, it can obtain the same projection point on the two-dimensional image as the original spatial point under the action of fan beam rays; in the y direction, parallel beam projection is used to ensure the matching of the y direction of the two-dimensional image with the three-dimensional model.

[0033] In this embodiment, the following is a further explanation of the method of combining lateral expansion with parallel projection: This embodiment proposes a technique of combining lateral expansion of a model with parallel projection to simulate the effect of fan beam projection. Using parallel projection can ensure the matching of the y direction (longitudinal direction) of the image and the model. Since the calculation of parallel projection is very simple, it will not be elaborated here. To make the x direction (lateral direction) also match, the model is laterally expanded so that after lateral expansion, under the action of parallel beam projection, the projection result is equivalent to the position of the original model under fan beam projection. The specific process can be seen in Figure 4 Schematic diagram of the principle of lateral expansion.

[0034] Fan beam scanning (projection) can be imagined as cutting through the cross-section of the human body with slices of pizza of different heights. Therefore, Figure 4 The correct way to understand it is to regard it as a top view, reflecting the position relationship between the light source, the flat panel, and the projection point in the fan beam plane. Since the fan beam projection is the same at any height and has nothing to do with the height, therefore Figure 4 it can be understood as the situation within any height. Since according to the habit in the field of images, the abscissa of the image is x and the ordinate is y, this schematic diagram clearly and intuitively reveals the calculation process in the x direction by means of illustration.

[0035] Figure 4 In [reference], the black solid line represents the flat panel detector. Considering a spatial point (black solid circle), under the action of the red ray of the fan beam projection, it is projected onto the black dashed hollow dot on the flat panel detector. The gray solid circle on the left represents the new position after repositioning (lateral stretching) of the original point (black solid circle). This position has the same depth as the original point, but under the action of the ray of the parallel beam projection, it can also be projected onto the same point (black dashed hollow circle).

[0036] Substitute the above analysis of the spatial points into all vertices of the model, and calculate the new coordinates of the points after repositioning for all vertices respectively. Since after repositioning, the points remain unchanged in the depth direction, but are farther from the perpendicular line from the ray source to the flat plate in the lateral direction. From the overall effect of the model, it seems that a lateral stretching (expansion) has occurred.

[0037] Regarding using expansion + parallel projection to equivalent fan-beam projection effect, the following 3 points are additionally supplemented: (1) When the pose or shape of the three-dimensional model in space changes, regenerate the repositioned points for the spatial points in the three-dimensional model: In the above schematic diagram, once the spatial points change, their corresponding correct repositioned points will also change and need to be recalculated. Therefore, when the pose or shape of the 3D model in space changes, its lateral expansion also needs to be recalculated.

[0038] (2) Calculate the repositioned points based on the two-dimensional images corresponding to the frontal and lateral views of the three-dimensional model respectively: When visualizing the frontal and lateral views, the directions of model expansion are different and need to be calculated separately.

[0039] (3) When the position of the foot of the perpendicular from the light source to the flat panel detector changes, regenerate the repositioned points for the spatial points in the three-dimensional model: In the above, drawing the foot of the perpendicular from the ray source to the flat plate at the center of the flat plate is purely for the convenience of explanation. In fact, due to mechanical assembly reasons, the foot of the perpendicular from the light source to the flat plate is not necessarily exactly at the center, and its specific position is generally provided by a spatial geometry calibration algorithm. The above calculation process of repositioning also needs to be considered accordingly. For example, for the case where the foot of the perpendicular where the light source hits the flat plate is eccentric to the left as in Figure 5 For the same spatial point (black solid circle), the expansion distance of its repositioned expansion point will become smaller.

[0040] In addition, using the principle of duality and complementarity in mathematics, we additionally supplement that the reconstructed three-dimensional model generates repositioned points for each spatial point in the three-dimensional model by using longitudinal compression combined with perspective projection. Specifically: in the x direction, use perspective projection to ensure that the x direction of the two-dimensional image matches the three-dimensional model; in the y direction, based on any longitudinal section of the width of the three-dimensional model, perform the longitudinal compression on each spatial point on the current longitudinal section respectively to obtain the repositioned point corresponding to the current spatial point, and the repositioned point is in the same position as the original spatial point under the action of perspective projection in the fan-beam (parallel beam) projection.

[0041] In this embodiment, the following is a further explanation of the method of using longitudinal compression combined with perspective projection: Using vertical compression + perspective projection can also achieve a similar equivalent effect. When using perspective projection, it can be matched in the x direction. To also match in the y direction, the model is vertically compressed so that after vertical compression, under the action of perspective projection, the projection result is equivalent to the position of the original model under fan-beam (parallel-beam) projection. Figure 6 A schematic diagram of this principle is supplemented. To emphasize the need for compression in the y direction (vertical direction), Figure 6 actually Figure 4 is rotated 90° based on Figure 6 It can be understood as the situation within a longitudinal section.

[0042] S2: Use the generated repositioning points to match with the projection points on the two-dimensional image corresponding to the three-dimensional model, and generate a 3D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image.

[0043] S3: Calculate the projection contour edges of the three-dimensional model, and project the projection contour edges to generate contour lines to generate a 2D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image. Specifically: A: Apply an operation of combining lateral expansion and parallel projection to the three-dimensional model, and obtain the projection contour edges of the original three-dimensional model on the expanded three-dimensional model.

[0044] When calculating the projection contour edges of the three-dimensional model and applying an operation of combining lateral expansion and parallel projection to the three-dimensional model, the condition for the edge of the original three-dimensional model to be a contour edge is that the orientations of the two adjacent faces of the corresponding edge of the expanded three-dimensional model are opposite to the direction of the parallel-beam projection ray.

[0045] B: For each of the projection contour edges, project the three-dimensional model using a fan beam to generate contour lines.

[0046] For the two endpoints of the projection contour edge, calculate two projection points respectively, and the line connecting the two projection points is the contour line; among them, when calculating the x coordinate of the projection point, consider the magnification using similar triangles, and when calculating the y coordinate of the projection point, use parallel projection and then convert it into pixel coordinates using the pixel size. Second Embodiment This embodiment provides a specific example of the specific effect achieved based on the method in the first embodiment. All technical effects of the present invention have been implemented using the Matlab language. It should be noted that achieving the above effects has nothing to do with the specific programming language used. In fact, for any language, as long as the above algorithm principle is mastered, similar effects can be achieved.

[0047] To verify the actual effect of the above visualization method and ensure the correctness of the relevant calculations in the visualization, we applied the visualization algorithm to the 2D-3D registration scenario where the 3D model segmented from CT was registered into the biplane space. If the visualization algorithm is implemented correctly, since the model is imported from an external CT, its shape correctness is guaranteed by the CT. Then, as long as the model is registered in the correct spatial pose, after setting it to be transparent, it is expected to cover the underlying X-ray map well. Similarly, in the 2D visualization, its projected contour is expected to fit well with the edges of the bony structures in the 2D full-length image.

[0048] It should be noted that the example of 2D-3D registration we gave is entirely for the consideration of the control variable method to facilitate the verification of the accuracy of the visualization algorithm. Because when the visualization algorithm has not been verified to be accurate, directly displaying the model reconstructed from the biplane, due to the errors in the biplane reconstruction algorithm, the shape and pose of the reconstructed model may not be accurate. In this case, when the model cannot match the spatial map or its contour cannot match the edges of the bony structures in the 2D image, it is impossible to determine whether it is a problem of visualization. As can be seen from the following display figures, the visualization algorithm is accurate. As long as the shape and pose of the model are correct, the visualization algorithm can show a perfect match. Then, when applying this algorithm to import the model reconstructed from the biplane, if there is a mismatch, it can be determined that it must be a problem of the reconstruction algorithm.

[0049] In addition, this example shows the matching situation of lumbar vertebrae L3-L5. However, in the process of implementing this algorithm, it does not depend on any anatomical information. Any structure is treated as a general 3D surface model. Therefore, this visualization method can be applied to other bones, prostheses, organs, and even any object structure beyond the medical field, and it is a general visualization method.

[0050] The specific steps and results of the experiment we conducted are as follows: The human body phantom was positioned according to the patient's standing pose required by the protocol during actual shooting, and full-length X-ray slit images were taken using two devices, namely EOS physical fan beam and small cone beam, respectively, to obtain the spliced images. As Figure 7 is a schematic diagram of using the physical fan beam anteroposterior full-length images, as Figure 8 is a schematic diagram of using the small cone beam anteroposterior full-length images.

[0051] As Figure 9 shown, the CT of the phantom was used to segment lumbar vertebrae L3-L5 using commercial segmentation software, and 4 key points were marked.

[0052] As Figure 10 shown, the projection points of these key points were marked on the EOS physical fan beam anteroposterior full-length images. As Figure 11As shown, project the projection points of these key points on the full-length frontal and lateral view diagrams of the small cone beam.

[0053] The following mainly shows the effects of the embodiments: As Figure 12 shown is the frontal view spatial matching effect diagram of the 3D model and the physical fan beam full-length diagram. As Figure 13 shown is the lateral view spatial matching effect diagram of the 3D model and the physical fan beam full-length diagram. As Figure 14 shown is the frontal view spatial matching effect diagram of the 3D model and the small cone beam full-length diagram. As Figure 15 shown is the lateral view spatial matching effect diagram of the 3D model and the small cone beam full-length diagram.

[0054] Figures 12 - 15 The 3D visualization effect is shown, in which Figures 12 - 13 the effect applied to the EOS physical fan beam full-length image is shown, Figures 14 - 15 and the effect applied to the small cone beam full-length diagram is shown. It can be seen that for both the full-length mosaic diagram projected by the EOS physical fan beam and the full-length mosaic diagram projected by the physical small cone beam, the visualization algorithm can accurately visualize the correct matching effect.

[0055] As Figure 16 shown is the 2D visualization frontal view effect diagram of the fast contour line projection algorithm on the small cone beam full-length diagram. As Figure 17 shown is the 2D visualization lateral view effect diagram of the fast contour line projection algorithm on the small cone beam full-length diagram.

[0056] Figures 16 - 17 The 2D visualization effect of using the fast contour line projection algorithm applied to the small cone beam full-length diagram is shown. It can be seen that this approximate calculation effect is very good, and the contour fits very well with the edges of the bony structures in the full-length diagram. This visualization algorithm is very fast, and the time for calculating the two perspectives of the frontal and lateral views is only 1.03 s.

[0057] The third embodiment As Figure 18 shown, this embodiment provides a 3D model and 2D image matching degree approximate visualization system for performing the 3D model and 2D image matching degree approximate visualization method as in the first embodiment, including: The spatial point repositioning module 1 is used to generate repositioned points for each spatial point in the reconstructed 3D model by using the method of lateral expansion combined with parallel projection or the method of longitudinal compression combined with perspective projection; The 3D visualization module 2 is used to match the generated repositioned points with the projection points on the 2D image corresponding to the 3D model to generate a 3D visualization effect diagram of the matching degree between the 3D model and the 2D image; The 2D visualization module 3 is used to calculate the projected contour edges of the three-dimensional model, project the projected contour edges to generate contour lines, and generate a 2D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image.

[0058] A computer-readable storage medium stores computer code, and when the computer code is executed, the above method is executed. Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

[0059] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, several improvements and refinements can be made without departing from the principle of the present invention, and these improvements and refinements should also be regarded as the protection scope of the present invention.

[0060] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0061] It should be noted that the above embodiments can be freely combined according to needs. The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, several improvements and refinements can be made without departing from the principle of the present invention, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for approximate visualization of matching degree between a three-dimensional model and a two-dimensional image, characterized in that: The following steps are involved: S1: generating a relocation point for each spatial point in the reconstructed three-dimensional model by using a method of lateral expansion combined with parallel projection or a method of longitudinal compression combined with perspective projection; S2: using the generated relocation points to match the projection points on the two-dimensional image corresponding to the three-dimensional model, to generate a 3D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image; S3: Calculate the projection contour edge of the three-dimensional model, and project the projection contour edge to generate a contour line to generate a 2D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image.

2. The method for visualizing the approximate matching degree between a three-dimensional model and a two-dimensional image according to claim 1, characterized in that: In step S1, a relocation point is generated for each spatial point in the reconstructed three-dimensional model by using a method of lateral expansion combined with parallel projection, specifically: In the x direction, based on a cross section at any height of the three-dimensional model, each of the spatial points on the current cross section is laterally expanded to obtain the repositioning point corresponding to the current spatial point, wherein the repositioning point has the same depth as the original spatial point, but can obtain the same projection point on the two-dimensional image as the original spatial point under the action of the fan beam ray under the action of the parallel beam projection; In the y direction, parallel beam projection is used to ensure that the 2D image matches the 3D model in the y direction.

3. The method for visualizing the approximate matching degree between a three-dimensional model and a two-dimensional image according to claim 1, characterized in that: Regarding the method of combining the lateral expansion with the parallel projection, it also includes: When the position or shape of the three-dimensional model in space changes, regenerating the relocation point for the spatial point in the three-dimensional model; Calculate the relocation points based on the two-dimensional images corresponding to the frontal and lateral positions of the three-dimensional model respectively; When the position of the foot of the perpendicular from the light source to the flat panel detector changes, the relocation points are regenerated for a plurality of spatial points in the three-dimensional model.

4. The method for visualizing the approximate matching degree between a three-dimensional model and a two-dimensional image according to claim 1, characterized in that: In step S1, a relocation point is generated for each spatial point in the reconstructed three-dimensional model by using a longitudinal compression combined with a perspective projection method, specifically: In the x direction, perspective projection is used to ensure that the two-dimensional image matches the three-dimensional model in the x direction; In the y direction, based on a longitudinal section of any width of the three-dimensional model, each of the spatial points on the current longitudinal section is longitudinally compressed to obtain the repositioning point corresponding to the current spatial point, and the repositioning point under the effect of perspective projection has the same position as the original spatial point under fan beam / parallel beam projection.

5. The method for visualizing the approximate matching degree between a three-dimensional model and a two-dimensional image according to claim 1, characterized in that: In step S3, the projection contour edge of the three-dimensional model is calculated, and the projection contour edge is projected to generate a contour line to generate a 2D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image, specifically: Applying a lateral expansion combined with a parallel projection operation to the three-dimensional model, and obtaining the projection contour edge of the original three-dimensional model on the expanded three-dimensional model; For each of the projected contour edges, a contour line is generated by projecting the three-dimensional model using a fan beam.

6. The method for visualizing the approximate matching degree between a three-dimensional model and a two-dimensional image according to claim 5, characterized in that: Applying lateral expansion combined with parallel projection to the three-dimensional model to obtain the projection contour edge of the original three-dimensional model on the expanded three-dimensional model is specifically: The projected contour edge of the three-dimensional model is calculated, and a lateral expansion combined with parallel projection operation is performed on the three-dimensional model. The condition for the original edge of the three-dimensional model to be a contour edge is that the directions of the two adjacent faces of the corresponding edge of the expanded three-dimensional model are in opposite directions relative to the direction of the parallel beam projection ray.

7. The method for visualizing the approximate matching degree between a three-dimensional model and a two-dimensional image according to claim 5, characterized in that: For each of the projected contour edges, a fan beam is used to project the three-dimensional model to generate a contour line, specifically: For the two endpoints of the projected contour edge, two projection points are calculated respectively, and the line connecting the two projection points is the contour line; Among them, when calculating the x-coordinate of the projection point, similar triangles are used to consider the magnification, and when calculating the y-coordinate of the projection point, parallel projection is used and then the pixel size is used to convert it into pixel coordinates.

8. A three-dimensional model and two-dimensional image matching degree approximate visualization system for executing the three-dimensional model and two-dimensional image matching degree approximate visualization method according to any one of claims 1 to 7, characterized in that: include: A spatial point relocation module, used to generate a relocation point for each spatial point in the reconstructed three-dimensional model by adopting a method of lateral expansion combined with parallel projection or a method of longitudinal compression combined with perspective projection; A 3D visualization module, used to match the generated relocation points with projection points on the two-dimensional image corresponding to the three-dimensional model, and generate a 3D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image; The 2D visualization module is used to calculate the projection contour edge of the three-dimensional model and project the projection contour edge to generate a contour line to generate a 2D visualization effect diagram of the matching degree between the three-dimensional model and the two-dimensional image.

9. A computer device comprising a memory and one or more processors, wherein the memory stores computer codes, and when the computer codes are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7. 10 . A computer-readable storage medium storing a computer code. When the computer code is executed, the method according to claim 1 is executed.

Images