Three-dimensional model modeling and two-dimensional image matching degree visualization method and system

Abstract

The invention relates to the technical field of image matching degree visualization, and provides a three-dimensional model modeling and two-dimensional image matching degree visualization method, which comprises the following steps of: S1, calculating a space mapping angular point of a slice image which is generated by adopting cone beam projection scanning and is used for splicing a full-length image in a double-plane coordinate system; s2, calculating the coordinates of the bone key points in the biplane coordinate system on the basis of the coordinates of the bone key points projected on the normal position and the lateral position of the mark points on the film image, so as to complete the initial position registration of the three-dimensional model in the biplane coordinate system; s3, retaining a film image related to the interested three-dimensional model in the current visualization, and realizing 3D visual display of the three-dimensional model and the film image; and S4, performing interaction of front and side view angle change, currently displayed front and side film images and three-dimensional model pose adjustment based on the 3D visual window. Therefore, a visualization algorithm of the matching degree of the 3D model and the two-dimensional image based on the full-length image of the physical small cone beam projection is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of visualization of image matching degree, and particularly to a visualization method and system for the matching degree between three-dimensional model modeling and two-dimensional images. Background Art

[0002] X-ray slit scanning is an advanced X-ray scanning technology. Compared with the large cone angle and discontinuous X-ray exposures at multiple positions of traditional DR, under this technology, the X-ray tube and the detector continuously perform translational movements; by restricting the cone angle of the X-ray beam in the longitudinal direction through a collimator and cooperating with the photosensitivity of the central ROI strip of the detector, a series of central narrow slit images only a few centimeters high can be obtained. This scanning method can reduce the geometric distortion caused by oblique rays, making the finally obtained image have a larger field of view, smaller distortion, and more accurate measurement.

[0003] Based on the full-length slit-scanned image and spatial geometric calibration information, a 2D-3D registration algorithm can import an external surface model and perform registration in space, and a three-dimensional reconstruction algorithm can recover the surface model and spatial pose of the organ of interest from the 2D image. These three-dimensional information is more intuitive and conforms to the 3D perception of humans compared with the anatomical parameters measured on traditional 2D images. At the same time, the 2D measurement essentially measures the projection of the 3D parameter on the two-dimensional plane, and this projection is affected by the patient's posture; while measuring on the reconstructed or registered three-dimensional model has no such problem. This method of calculating the three-dimensional model or / and pose based on the full-length X-ray image can quickly assist doctors in diagnosis and screening with the advantages of low dose and weight-bearing position compared with the commonly used lying CT in clinics, and is particularly irreplaceable in the orthopedic field.

[0004] After registration or three-dimensional reconstruction, it is necessary to visualize the matching degree between the registered or reconstructed model and the 2D image. This visualization needs to intuitively show where the three-dimensional model matches (fits) and where it does not match the 2D image. This visual cue can provide feedback to the doctor, so as to correct the pose of the model during registration and adjust the shape of the model after reconstruction. Figure 1It is a three-dimensional scene display diagram for reconstructing the main bony structures of the whole body using the full-length image of the slit. This perspective is the entrance perspective, that is, the 2D full-length image is displayed in the 3D space through texture mapping, and after putting it into the corresponding 3D model, the camera is roamed to the perspective when entering the corresponding physical equipment compartment and the screenshot is taken. Through this perspective, it can be seen that the shapes of the main bony structures conform to human anatomy, the relative pose relationships between the bony structures are natural, and the distribution of the bony structures and the corresponding parts in the image in the longitudinal direction is consistent. However, this visualization method in this perspective can only give us a general perception of evaluating the matching degree between the model and the image, and the details of the specific matching degree are not visualized. Therefore, it is urgent to develop a matching degree visualization algorithm suitable for the image product.

[0005] The commonly used visualization library in the medical field is the open-source library VTK (Visualization toolkit) of Kitware Company. This library is dedicated to scientific visualization and provides ready-made solutions for common visualization scenarios in all walks of life, such as the common coronal plane, sagittal plane, cross-sectional plane, and 3D space four-pane visualization layouts in the medical field. However, the visualization solution for the matching degree between the 3D model and the 2D image concerned by the present invention is to solve the visualization problem of the matching degree between 3D data and 2D data. VTK does not provide a ready-made solution for this cross-dimensional visualization scenario for calling, so it needs to be studied and developed independently.

[0006] The currently relatively mature visualization solution is the sterEOS 3D station of EOS Company in France. This post-processing workstation software can import the line-scanned full-length anteroposterior and lateral X-ray images of EOS Company. The line-scanned imaging system of EOS Company uses a 45-cm-wide linear radiation source and detector, and each exposure obtains a pair of anteroposterior and lateral linear images with a height of one pixel. This physical projection mode is very close to the fan-beam scanning in CT plain scan. Through the vertical movement of the gantry, the system obtains linear images at different heights and simply performs longitudinal stacking in the post-processing stitching algorithm, and finally obtains a pair of anteroposterior and lateral matching images with the same number of pixel rows in the longitudinal direction.

[0007] After importing the anteroposterior and lateral images into the sterEOS 3D workstation and making several annotations, the software will reconstruct the 3D model of interest to the doctor. In order to visualize the effect and matching degree of the reconstructed model, EOS Company adopts a 2D visualization method, that is, instead of displaying the model itself, only the contour line of the model is projected onto the 2D full-length image and drawn into the image in the form of a colored line, as Figure 2 shown.

[0008] EOS uses this 2D visualization mainly for the convenience of subsequent manual adjustment by users. As can be seen from Figure 2 , on the anteroposterior radiograph, there is a large mismatch between the left transverse process of the model and the image; similarly, on the lateral radiograph, there is an obvious mismatch in the pedicle of the model, indicating that there are still relatively large errors in the reconstruction of the model reconstruction algorithm at these two places. There are control handles distributed on the projection contour line of the software, which facilitates the user to manually drag the parts with large mismatches to the correct positions from the corresponding perspectives, thereby further adjusting the model. This is an interactive-visualization method that interacts in 2D but modifies and calculates in the background for the 3D model.

[0009] Another advantage of this 2D visualization is its simplicity in implementation: Since the physical scan has a fan-beam projection horizontally and a parallel movement longitudinally, and EOS's linear detector ensures a high degree of consistency between the actual physical projection and the ideal mathematical model projection. And the mathematical formula of this projection is known and determined, so as long as the contour line projection is calculated and drawn in the 2D image, the visualization is completed.

[0010] As Figure 3 shown, SterEOS also provides a 3D window interface for modeling, but only the model is placed in this interface, and no image texture is placed, so it can at most be regarded as a 3D visualization of the 3D model, and there is no 3D visualization of the model and image fitting degree.

[0011] There are mainly two problems with this visualization technology of EOS: (1) The mathematical projection model used behind the visualization is closely related to the physical projection model used in actual imaging. The linear detector used by EOS is a special gas detector, and the principle of this detector was proposed by Georges Charpak in 1922 and he won the Nobel Prize for it. This detector has only one pixel in the height direction, making the actual scanning process very close to the ideal fan-beam + translation projection method. This detector is exclusive to EOS and is protected by patents. Therefore, if other manufacturers want to achieve a similar fan-beam scanning effect, they can only find another way. For example, Shimadzu Corporation in Japan realizes it by using a collimator combined with flat ROI photosensing to obtain images of the central slit narrow strip. However, each strip is not one pixel high and still has hundreds of pixels. Therefore, this implementation method essentially uses small cone-angle images to approximately replace the ideal fan-beam images. So due to the difference in the actual physical projection method, the fan-beam projection mathematical model of EOS cannot be directly applied to the small cone-beam images. In addition, companies like Shimadzu in Japan do not provide 3D reconstruction functions, and naturally do not study the visualization of the matching degree between 3D models and two-dimensional small cone-beam images. Therefore, how to realize the visualization of the matching between the model and the image based on physical small cone-beam images is a brand-new engineering problem.

[0012] (2) EOS only provides 2D visualization. Although 2D visualization has its advantages, in essence, it flattens a three-dimensional shape onto a 2D image. This visualization process discards a lot of structural information of the model and only retains the contour for matching with the 2D image. Therefore, due to the dimensionality reduction operation, 2D visualization abandons the natural advantages of the 3D scene and sacrifices the fact that the reconstructed data is 3D. Since the projection process occurs in 3D space, there should be a corresponding 3D visualization to display the matching relationship in 3D space. Summary of the Invention

[0013] In view of the above problems, the purpose of the present invention is to provide a visualization method and system for the matching degree between a three-dimensional model and a two-dimensional image, so as to implement a visualization algorithm for the matching degree between a 3D model and a two-dimensional image based on a full-length image of physical small cone-beam projection. This algorithm has relatively loose hardware requirements and does not require the use of special linear detectors like EOS. Instead, a general flat panel detector can be used by setting the central ROI. It has wide adaptability to hardware and low requirements.

[0014] The above object of the present invention is achieved by the following technical solutions: A visualization method for the matching degree between a three-dimensional model and a two-dimensional image includes the following steps: S1: Calculate the spatial mapping corner points of the slice images used for stitching the full-length image generated by cone-beam projection scanning in a double-plane coordinate system, and calculate the coordinates of each point on the slice image based on the spatial mapping corner points; S2: Select the bony key points used for marking on the three-dimensional model, and calculate the coordinates of the bony key points in the double-plane coordinate system based on the coordinates of the projection of the marked points of the bony key points in the frontal and lateral views on the slice image, so as to complete the initial position registration of the three-dimensional model in the double-plane coordinate system; S3: Retain the slice images related to the three-dimensional model of interest in the current visualization, and preferentially display the slice image with the ID number in the median among the related slice images, to achieve the 3D visualization display of the three-dimensional model and the slice image; S4: The user performs interactions including changing the frontal and lateral views, switching the currently displayed frontal and lateral slice images, and adjusting the pose of the three-dimensional model based on the 3D visualization window.

[0015] Further, before step S1, it includes calculating the positions of the light source and the flat panel corner points during actual exposure, specifically: According to the travel of the gantry, calibrate the position information of the light source and the flat panel corner points at several heights at equal intervals, and record the corresponding PLC height for each calibrated height; Obtain the exposure PLC height during the actual exposure of the film image, write it into the data element tag, and use the exposure PLC height as the independent variable. Based on the calibrated light source and flat plate corner point position information at several heights, use interpolation or regression algorithms to calculate the position information of the light source and flat plate corner points during actual exposure.

[0016] Furthermore, in step S1, calculate the spatial mapping corner points of the film image for stitching the full-length image generated by cone-beam projection scanning in the double-plane coordinate system, and calculate the coordinates of each point on the film image based on the spatial mapping corner points. Specifically: The slit scanning used in the cone-beam projection scanning is realized by setting a central photosensitive ROI on the flat plate, and the physical space area corresponding to the actually obtained film image is the area of the central photosensitive ROI; Read the ROI position information of the photosensitive ROI in the flat plate in the data element tag, and deduce the position information of the ROI corner points based on the position information of the flat plate corner points during actual exposure. The position information of the ROI corner points is also the spatial mapping corner points of the actually scanned film image in the double-plane coordinate system; Deduce the coordinates of each point on the film image based on the spatial mapping corner points.

[0017] Furthermore, in step S2, select the bony key points for marking on the three-dimensional model, and calculate the coordinates of the bony key points in the double-plane coordinate system based on the coordinates of the marked point projections of the bony key points in the frontal and lateral views on the film image, so as to complete the initial position registration of the three-dimensional model in the double-plane coordinate system. Specifically: After identifying the marked point projections of the bony key points in the frontal and lateral views on the stitched full-length image respectively, the stitching algorithm maps the coordinates on the stitched full-length image back to the film image corresponding to the corresponding ID number and the coordinates on the film image; Based on the coordinates of the marked point projections of the bony key points in the frontal and lateral views on the film image and the coordinates of the light sources in the frontal and lateral views, use the triangulation method to calculate the coordinates of the bony key points in the double-plane coordinate system; After marking the bony key points on the three-dimensional model, transfer the three-dimensional model from its own local coordinate system to the double-plane coordinate system to realize the initial positioning of the three-dimensional model and the two-dimensional film image.

[0018] Furthermore, in step S3, retain the film images related to the three-dimensional model of interest in the current visualization, and preferentially display the film image with the ID number in the median among the related film images. Specifically: For the 3D model of interest in the current visualization, only a number of the slice images at the corresponding height record relevant projection information during scanning. The slice images of the corresponding structure of the 3D model not captured at other heights are irrelevant to the current visualization session. After importing the 3D model and determining the initial positioning pose, for the frontal and lateral views, all the slice images are traversed respectively. Based on the spatially calibrated environment, a spatial pyramid geometry is constructed for the spatial mapping corner points of each slice image and the light sources placed at the corresponding positions, and an intersection is calculated in 3D space with the 3D model. Those with intersections are recorded as relevant, and those without intersections are recorded as irrelevant. After determining the relevance, a continuous ID range of slice images for both the frontal and lateral views is retained as relevant in this visualization session. For the frontal and lateral views respectively, the slice image with the ID number at the median among the relevant slice images is preferentially displayed, and other slice images are hidden. The open display right of other slice images can be switched by the user during interaction.

[0019] Further, in step S4, the user performs interactions including changing the frontal and lateral views, switching the currently displayed slice images for the frontal and lateral views, and adjusting the pose of the 3D model based on the 3D visualization window. Specifically: Changing the frontal and lateral views: Switch the viewing point to the light source corresponding to the currently displayed slice image for the frontal or lateral view, and switch the line-of-sight direction to look towards the slice image. Switching the currently displayed slice images for the frontal and lateral views: Independently adjust the ID of the slice image displayed for the frontal view and the ID of the slice image displayed for the lateral view respectively. If there are m relevant slice images for the frontal view and n relevant slice images for the lateral view, then there are m * n combinations of the IDs of the slice images displayed for the frontal and lateral views simultaneously in space. When any one slice image is displayed for each view, other slice images in that view are hidden to ensure a clean interface. Interactions including adjusting the pose of the 3D model: Based on the frontal and lateral views, adjust the pose of the 3D model in line with the intuitive interaction habits, including up / down / left / right displacement adjustments relative to the frontal and lateral views, and rotations in 3 directions along the horizontal axis, vertical axis of the screen, and the axis pointing out of the screen towards the user.

[0020] Further, the visualization method for the matching degree between 3D model modeling and 2D images is characterized in that it also includes realizing the 2D visualization display of the 3D model and the slice images. Specifically: Based on each of the slice images, determine the contour edges of the three-dimensional model when the light source corresponding to the slice image is used as the viewing point. Among the edges on the three-dimensional model existing in the form of a surface model, the condition for classifying them as the contour edges is that they contain two adjacent patches, and the normal vectors of the two adjacent patches are in opposite directions with respect to the vector direction from the viewing point to the midpoint of the edge; Project the contour edges onto the relevant slice image to obtain the starting and ending coordinates of the contour line, and discard the projections of the contour edges that exceed the range of the slice image; Use the stitching relationship between the slice image and the stitched full-length image to map the coordinates of the contour line in the slice image to the stitched full-length image, and draw it on the stitched full-length image in the form of a colored line to achieve the 2D visualization display of the three-dimensional model and the slice image.

[0021] The present invention also provides a visualization system for three-dimensional model modeling and two-dimensional image matching degree for performing the visualization method as described above, including: A slice image coordinate determination module, configured to calculate the spatial mapping corner points of the slice images for stitching the full-length image generated by cone-beam projection scanning in a double-plane coordinate system, and calculate the coordinates of each point position on the slice image based on the spatial mapping corner points; An initial position registration module, configured to select the bony key points for marking on the three-dimensional model, and calculate the coordinates of the bony key points in the double-plane coordinate system based on the coordinates of the projection of the marked points of the bony key points in the frontal and lateral views on the slice image, so as to complete the initial position registration of the three-dimensional model in the double-plane coordinate system; A 3D visualization display module, configured to retain the slice images related to the three-dimensional model of interest in the current visualization, and preferentially display the slice image with the ID number in the median among the related slice images, to achieve the 3D visualization display of the three-dimensional model and the slice image; A 3D visualization interaction module, configured to provide the user with interactions including changing the frontal and lateral view angles, switching the currently displayed frontal and lateral slice images, and adjusting the pose of the three-dimensional model based on the 3D visualization window; A 2D visualization interaction module, configured to achieve the 2D visualization display of the matching degree between the three-dimensional model and the slice image.

[0022] A computer device includes a memory and one or more processors. When the computer code stored in the memory is executed by the one or more processors, the one or more processors execute the method as described above.

[0023] A computer-readable storage medium stores computer code, which, when executed, causes the above-described method to be executed.

[0024] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) A visualization algorithm for the matching degree between a 3D model and an image based on a full-length physical small cone-beam projection is realized. This algorithm has relatively relaxed hardware requirements. Most visualization solutions on the market rely on special hardware devices. For example, EOS requires the use of special linear detectors, which not only have high costs but also are difficult to purchase and maintain, limiting their wide application. However, this algorithm only needs to use a general flat-panel detector and can be achieved by setting the central ROI, greatly reducing the hardware cost and acquisition difficulty. Whether it is a large medical institution, a grass-roots medical unit with relatively limited resources, or even a small scientific research laboratory, they can easily meet the hardware requirements and realize the visualization analysis of the matching degree between the 3D model and the image, greatly improving the accessibility and universality of this technology and broadening its application scenarios.

[0025] (2) Compared with the single 2D visualization of EOS, this technical solution has achieved a major breakthrough in visualization presentation based on physical cone-beam projection. On the one hand, 2D visualization similar to the contour line projection effect of EOS is realized. By determining the contour edges of the 3D model in the corresponding light source of the slice image as the viewing point and projecting them onto the relevant slice image and mapping them onto the stitched full-length image, and presenting them in the form of color lines, doctors and researchers can clearly observe the matching situation between the model and the image on the 2D plane, and accurately analyze the contour and details of the model, providing intuitive 2D image basis for disease diagnosis, surgical planning, etc. On the other hand, this solution has also realized 3D visualization of the matching degree between the model and the image texture in 3D space. From calculating the positions of the light source and the corner points of the flat panel during actual exposure, to completing the initial position registration of the 3D model in the double-plane coordinate system, to retaining and displaying the relevant slice images, and to the multi-dimensional interaction of the user based on the 3D visualization window, it comprehensively demonstrates the matching relationship between the model and the image in 3D space. The user can observe from different angles and different slice combinations, and more comprehensively and deeply understand the spatial position and morphological differences between the model and the image, providing a more powerful and three-dimensional visualization tool for the analysis and research of complex medical cases, significantly improving the depth and breadth of visualization, and meeting the diverse needs in different scenarios. Description of the Drawings

[0026] Figure 1 It is a three-dimensional scene display diagram for reconstructing the main bony structures of the whole body using a slit full-length image; Figure 2 It is a schematic diagram of 2D visualization adopted by EOS Company; Figure 3Schematic diagram of the modeled 3D window interface provided for SterEOS; Figure 4 Overall flowchart of the visualization method for the matching degree between the 3D model modeling and 2D images of the present invention; Figure 5 Schematic diagram of the present invention for taking full-length X-ray slit images and obtaining stitched images; Figure 6 Schematic diagram of the present invention for segmenting the lumbar vertebrae L3-L5 from the CT of the body film using commercial segmentation software and marking 4 key points; Figure 7 Schematic diagram of the present invention for marking the projection points of these key points on the full-length anteroposterior and lateral position diagrams; Figure 8 Schematic diagram after the initial registration of the algorithm execution of the present invention; Figure 9 Schematic diagram of the present invention for the user to freely select the frontal view image and the lateral view image; Figure 10 Schematic diagram of the present invention for adjusting the model pose; Figure 11 Schematic diagram of the present invention showing the full-length image and the full-length image with the contour line projection superimposed on the left and right sides; Figure 12 Overall structure diagram of the visualization system for the matching degree between the 3D model modeling and 2D images of the present invention. Detailed implementation mode

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0028] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups.

[0029] First embodiment As Figure 4 shown, this embodiment provides a visualization method for the matching degree between 3D model modeling and 2D images, including the following steps: S1: Calculate the spatial mapping corner points of the slice images for stitching the full-length image generated by cone-beam projection scanning in the double-plane coordinate system, and calculate the coordinates of each point on the slice image based on the spatial mapping corner points.

[0030] In this embodiment, before step S1, it is first necessary to calculate the positions of the light source and the flat-panel corner points during actual exposure. Specifically: According to the travel of the gantry, calibrate the position information of the light source and the flat-panel corner points at several heights at equal intervals, and record the corresponding PLC height for each calibrated height. However, the position of the flat-panel corner points and the light source during actual exposure depends on the timing when the user presses the hand switch, and the shooting height is inconsistent with the acquisition height during calibration. Therefore, it is necessary to obtain the exposure PLC height during actual exposure of the slice image being taken and write it into the data element tag dicom tag. Using the exposure PLC height as the independent variable and based on the position information of the light source and the flat-panel corner points at several calibrated heights, use interpolation or regression algorithms to calculate the position information of the light source and the flat-panel corner points during actual exposure.

[0031] Among them, an example of using interpolation or regression algorithms to calculate the position information of the light source and the flat-panel corner points during actual exposure is as follows: Example steps of the interpolation algorithm (taking linear interpolation as an example): (1) Preliminary preparation: During the calibration stage, we selected several height points at different positions in the gantry travel, recorded the PLC height corresponding to each height, and the specific positions of the light source and the flat-panel corner points at these heights. For example, we selected three height points. The PLC height corresponding to the first height is a certain value, and at this time, the position of the light source in space and the positions of the four flat-panel corner points are all recorded; the other two height points are also recorded in the same way. During actual exposure, we will obtain an exposure PLC height.

[0032] (2) Determine the range: Check which two calibrated PLC height values the actual exposure PLC height is between in the previously calibrated PLC height data. For example, the actual PLC height is between the PLC values of the first and second calibrated heights.

[0033] (3) Calculate the ratio: Based on the difference between the actual PLC height and the PLC values of these two calibrated heights, calculate a ratio value. Simply put, it is to see what proportion the difference between the actual value and the first calibrated value accounts for in the total difference between the two calibrated values.

[0034] (4) Calculate the light source position: For the light source position, use this ratio value to adjust the light source position data at the two calibrated heights. For example, mix and calculate the x coordinate of the light source at the first calibrated height and the x coordinate of the light source at the second calibrated height according to the ratio calculated just now to obtain the x coordinate of the light source during actual exposure. Using the same method, calculate the y coordinate and z coordinate of the light source.

[0035] (5) Calculate the corner positions of the flat panel: Similar to calculating the light source position, mix the x - coordinates of the flat - panel corners at the first calibration height and the x - coordinates of the flat - panel corners at the second calibration height in proportion to obtain the x - coordinate of the flat - panel corners during actual exposure. Then, use the same method to calculate the y - coordinate and z - coordinate of the flat - panel corners.

[0036] Example of the regression algorithm steps (taking simple linear regression as an example, assuming the relationship between the x - coordinate of the light source and the height of the PLC) (1) Collect data: Collect the heights of the PLC at different heights during calibration and the x - coordinates of the light source at the corresponding heights.

[0037] (2) Analyze the data to find the pattern: First, calculate the average value of these PLC heights, and then calculate the average value of the corresponding x - coordinates of the light source. Then, look at the differences between each PLC height and the average value, and the differences between the corresponding x - coordinates of the light source and the average value. Analyze the relationship between them through these differences to find a pattern that represents this relationship, that is, determine two key values. These two values can help us establish an equation that can represent the connection between the PLC height and the x - coordinate of the light source.

[0038] (3) Predict the x - coordinate of the light source: Substitute the height of the PLC during actual exposure into the equation just found to calculate the x - coordinate of the light source during actual exposure.

[0039] (4) Calculate other coordinates: For the y - coordinate and z - coordinate of the light source, as well as the x, y, and z coordinates of the flat - panel corners, establish equations in the same way respectively, and then substitute the height of the PLC during actual exposure to calculate their respective actual position information.

[0040] The position information of the light source and the flat - panel corners during actual exposure can also adopt the following scheme: The position information of the light source and the flat - panel corners under unknown PLC height readings through actual interpolation or regression calculation can also adopt the following scheme: (1) System modeling Model a single X - ray exposure system as a classic pinhole camera model. In the pinhole camera model, light follows the principle of straight - line propagation, passes through an extremely small pinhole (which can be regarded as an ideal pinhole), and forms an inverted image on the imaging plane. The X - ray exposure emits rays from a point light source and hits the flat - panel detector. From the perspective of the optical path propagation path, the two models are equivalent. Therefore, with the help of this model, we can model the geometric propagation part of the optical path of the X - ray exposure system into a classic model. (2) Parameter determination Identify 5 key parameters of this pinhole camera model (single - plane X - ray system): f (Focal length, SID): It refers to the distance from the small hole to the imaging plane. In single-plane X-ray, it corresponds to the distance from the light source to the flat panel (SID).

[0041] a (aspect ratio): It measures the ratio of the physical pixel size of the detector unit in the x-direction to that in the y-direction. The ideal value should be close to 1, indicating that the sampling intervals in the x and y directions of the detector are equidistant.

[0042] s (skew): The skew factor, which measures the angle between the sampling directions in the x and y directions. The ideal value should be 0, indicating that the x and y directions are strictly perpendicular.

[0043] x0 and y0: They represent the x and y coordinates in the image coordinate system of the point where the light source hits the imaging plane vertically. In the ideal case, they are half of the length and width of the image, and at this time, the light source is strictly centered with the flat panel.

[0044] (3) Equation regression Interpolate or regress the five camera model parameters independently for the plc height values respectively to obtain the five parameter values at the uncollected heights, construct the camera model at the unknown heights, and convert it back to the representation of the light source and flat panel coordinates.

[0045] (4) Calculation and storage optimization Compared with directly interpolating or regressing the flat panel and light source coordinates, the advantage of first converting to the camera model and then regressing or interpolating the model parameters is that it saves the calculation and storage costs. Directly regressing or interpolating the coordinates requires establishing interpolation or regression analysis for 15 scalar values, which are 5 points with 3 coordinate components for each point, while regressing or interpolating the camera model parameters only requires performing analysis on 5 scalars.

[0046] The slit scan used in the cone beam projection scan is realized by setting the central sensitive ROI on the flat panel. Therefore, the physical space area corresponding to the actually acquired slice image is not equal to the physical area of the entire flat panel. The physical space area corresponding to the actually acquired slice image is the area of the central sensitive ROI; read the ROI position information of the sensitive ROI in the flat panel in the data element tag, and calculate the ROI corner position information based on the position information of the flat panel corner points during actual exposure. The ROI corner position information is also the spatial mapping corner points of the actually scanned slice image in the double-plane coordinate system; calculate the coordinates of each point on the slice image based on the spatial mapping corner points.

[0047] S2: Select the bony key points for marking on the three-dimensional model, calculate the coordinates of the bony key points in the bi-plane coordinate system based on the coordinates of the marked point projections of the bony key points in the frontal and lateral views on the slice image, so as to complete the initial position registration of the three-dimensional model in the bi-plane coordinate system.

[0048] In this embodiment, step S2 is specifically as follows: After respectively identifying the marked point projections of the bony key points in the frontal and lateral views on the stitched full-length image, the stitching algorithm maps the coordinates on the stitched full-length image back to the coordinates on the corresponding slice image with the ID number and the slice image; Based on the coordinates of the marked point projections of the bony key points in the frontal and lateral views on the slice image, and the coordinates of the frontal and lateral light sources, use the triangulation method to calculate the coordinates of the bony key points in the bi-plane coordinate system; After marking the bony key points on the three-dimensional model, transfer the three-dimensional model from its own local coordinate system to the bi-plane coordinate system to achieve the initial positioning of the three-dimensional model and the two-dimensional slice image.

[0049] Among them, the calculation example of the triangulation method in this embodiment is as follows: (1) Data preparation On the stitched full-length images in the frontal and lateral views, the marked point projections of the bony key points have been accurately identified. Through the stitching algorithm, the coordinates of these projections on the full-length image are corresponding to the slice images with specific ID numbers. For example, on the frontal slice image, the coordinates of the projection of the bony key point are (X1, Y1, Z1), and on the lateral slice image, the projection coordinates are (X2, Y2, Z2).

[0050] It is clearly known that the coordinates of the frontal light source in space are (Sx1, Sy1, Sz1), and the coordinates of the lateral light source in space are (Sx2, Sy2, Sz2).

[0051] (2) Construct rays Starting from the frontal light source (Sx1, Sy1, Sz1), construct a ray towards the projection point of the bony key point with coordinates (X1, Y1) on the frontal slice image. The direction vector of this ray in space is (X1 - Sx1, Y1 - Sy1, Z1 - Sz1).

[0052] Similarly, construct a ray from the lateral light source (Sx2, Sy2, Sz2) towards the projection point of the bony key point with coordinates (X2, Y2) on the lateral slice image, and its direction vector is (X2 - Sx2, Y2 - Sy2, Z2 - Sz2).

[0053] (3) Triangulation calculation Since these two rays starting from the frontal and lateral light sources respectively both pass through the same bony key point, they must intersect at this point in space.

[0054] Let the coordinates of the bony key point in the bi - plane coordinate system be (X, Y, Z). From the frontal view, there exists a proportionality coefficient k1 such that the relationship between the coordinates of the bony key point and the coordinates of the frontal light source and the frontal projection point is satisfied: X = Sx1 + k1 * (X1 - Sx1) Y = Sy1 + k1 * (Y1 - Sy1) Z = Sz1 + k1 * (Z1 - Sz1) From the lateral view, there exists a proportionality coefficient k2 such that: X = Sx2 + k2 * (X2 - Sx2) Y = Sy2 + k2 * (Y2 - Sy2) Z = Sz2 + k2 * (Z1 - Sz2) Rearrange the above equations into the form of a system of binary equations about k1 and k2. There are two unknowns, k1 and k2, and three equations: Use the least - squares method to solve this over - determined system of equations to determine the proportionality coefficients k1 and k2. The goal of the least - squares method is to find the values of k1 and k2 that minimize the sum of the squares of all equation errors. By taking the partial derivatives of the error function and setting them to zero, a system of linear equations can be obtained, and solving this system of linear equations can give the optimal solutions of k1 and k2. Substitute the obtained values of k1 and k2 into the above equations about the X, Y, Z coordinates to calculate the accurate coordinates (X, Y, Z) of the bony key point in the bi - plane coordinate system.

[0055] (4)Model coordinate system conversion After marking this bony key point on the 3D model, assume that in the local coordinate system of the 3D model, the coordinates of this bony key point are (X0, Y0, Z0).

[0056] Then select two other bony key points that are not collinear with this point. According to the above triangulation calculation method, obtain their coordinates in the bi - plane coordinate system and their coordinates in the local coordinate system of the 3D model respectively.

[0057] Using the coordinate relationships of these three non - collinear bony key points in two coordinate systems, a transformation matrix containing translation and rotation information is calculated by the least - squares method. This matrix contains the translation amounts of the model in the X, Y, and Z directions and the rotation angle information about the X, Y, and Z axes.

[0058] For any point (Xm, Ym, Zm) on the three - dimensional model, its coordinates in the local coordinate system are represented as a column vector [Xm, Ym, Zm, 1] (adding one dimension for convenient matrix operations). By multiplying with the transformation matrix through matrix multiplication, the coordinates of this point in the double - plane coordinate system are obtained, thus realizing the transfer of the three - dimensional model from its own local coordinate system to the double - plane coordinate system and completing the initial positioning of the three - dimensional model and the two - dimensional slice image.

[0059] S3: Retain the slice images related to the three - dimensional model of interest in the current visualization, and preferentially display the slice image with the ID number at the median among the related slice images, to achieve the 3D visualization display of the three - dimensional model and the slice images.

[0060] In this embodiment, step S3 is specifically as follows: For the three - dimensional model of interest in the current visualization, during scanning, only a number of slice images at the corresponding height record the relevant projection information, and the slice images at other heights that do not capture the corresponding structure of the three - dimensional model are irrelevant to the current visualization session; After importing the three - dimensional model and determining the initial positioning pose, for the frontal and lateral views, respectively traverse all the slice images, construct a spatial pyramid geometry based on the spatial calibration environment for the spatial mapping corner points of each slice image and the light sources placed at the corresponding positions, perform a 3D space intersection with the three - dimensional model, record those with intersections as relevant and those without intersections as irrelevant. After determining the relevance, the frontal and lateral views each retain a continuous segment of slice images with relevant IDs in this visualization session; If all relevant slice images are displayed, because the actual spatial regions corresponding to these slice images will overlap, the interface will be very messy. Therefore, the frontal and lateral views respectively preferentially display the slice image with the ID number at the median among the relevant slice images and hide the other slice images, and the open display rights of the other slice images are switched by the user during interaction.

[0061] The detailed process of constructing the spatial pyramid geometry is exemplified as follows: (1) Determine the basic elements Spatial texture corner points: For each tile image, use the spatial texture corner points in the double-plane coordinate system calculated previously. Assume that the tile image has four spatial texture corner points, denoted as Point 1, Point 2, Point 3, and Point 4. These corner points define the position and range of the tile image in space.

[0062] Corresponding light source position: Determine the coordinates of the light source placed at the corresponding position of each tile image in the double-plane coordinate system, denoted as the light source point. The light source position is the vertex for constructing the spatial pyramid geometry.

[0063] (2)Construct the bottom polygon Using the four spatial texture corner points Point 1, Point 2, Point 3, and Point 4 of the tile image as vertices, construct a quadrilateral bottom surface in space. This quadrilateral represents the planar position of the tile image in space. The shape and boundary of the bottom surface can be determined by connecting these four points to form four sides: Side 12, Side 23, Side 34, and Side 41.

[0064] (3)Connect the vertex to the bottom surface Connect the light source point to the four vertices Point 1, Point 2, Point 3, and Point 4 of the bottom surface quadrilateral respectively. In this way, four line segments are formed: Line segment light source point 1, Line segment light source point 2, Line segment light source point 3, and Line segment light source point 4, starting from the light source point and reaching the respective vertices of the bottom surface. These line segments and the bottom surface quadrilateral together form a spatial pyramid geometry with the light source as the vertex and the quadrilateral determined by the spatial texture corner points of the tile image as the bottom surface.

[0065] (4)Complete the construction of the geometry After the above steps, a spatial pyramid geometry based on the spatial texture corner points of the tile image and the corresponding light source position is constructed. This geometry represents the spatial range of the light rays emitted from the light source passing through the planar region where the tile image is located. When performing a 3D spatial intersection operation with a 3D model later, it can be used to determine whether there is an intersection between this light ray region and the 3D model, thereby determining the relevance between this tile image and the current 3D model.

[0066] By constructing the spatial pyramid geometry for each tile image according to the above process and performing a 3D spatial intersection with the 3D model, the determination of the relevance between all tile images and the 3D model can be completed.

[0067] S4: The user performs interactions based on the 3D visualization window, including changing the front and side views, switching the tile images of the current front and side views displayed, and adjusting the pose of the 3D model.

[0068] In this embodiment, step S4 is specifically as follows: Changing the front and side views: Switch the viewpoint to the light source corresponding to the frontal or lateral view of the current displayed slice image, and switch the line of sight direction to look towards the slice image; Switching between the frontal and lateral views of the currently displayed slice images: Independently adjust the IDs of the slice images displayed in the frontal view and the IDs of the slice images displayed in the lateral view respectively. If there are m slice images related to the frontal view and n slice images related to the lateral view, then there are m*n combinations of the IDs of the slice images displayed in the frontal and lateral views simultaneously in space. When any one of the slice images is displayed in each viewing angle, the other slice images in that viewing angle are hidden to ensure the cleanliness of the interface; Interactions including the pose adjustment of the 3D model: Based on the frontal and lateral views, adjust the pose of the 3D model that conforms to the intuitive interaction habits, including the up, down, left, and right displacement adjustments relative to the frontal and lateral views, and the rotations in 3 directions along the horizontal axis, vertical axis, and the axis pointing out of the screen towards the user of the screen. Here, up, down, left, and right are all relative to the frontal and lateral views. For example, in the frontal view and the lateral view, a left translation means the object moves left in the window, but the actual underlying object translation is different. Similarly, the rotations along the up and down axes, left and right axes, and out-of-screen axis are defined relative to different views.

[0069] The above are the process steps for realizing the 3D visualization display of the 3D model and the slice image. In addition, based on the settings in steps S1 - S3 of this embodiment, it also includes realizing the 2D visualization display of the 3D model and the slice image, specifically as follows: Based on each slice image, determine the contour edges of the 3D model when the light source corresponding to the slice image is used as the viewpoint. Among the edges on the 3D model existing in the form of a surface model, the condition for classifying them as the contour edges is that they contain two adjacent patches, and the normal vectors of the two adjacent patches are in opposite directions relative to the vector direction from the viewpoint to the midpoint of the edge; Project the contour edges onto the relevant slice images to obtain the starting and ending point coordinates of the contour lines, and discard the projections of the contour edges that exceed the range of the slice images; Use the stitching relationship between the slice image - the stitched full-length image to map the coordinates of the contour lines under the slice image onto the stitched full-length image, and draw them on the stitched full-length image in the form of color lines to realize the 2D visualization display of the 3D model and the slice image.

[0070] Second Embodiment This embodiment provides a specific example of the visualization display result based on the specific process steps of the first embodiment. The specific example is as follows: All the technical effects of the present invention have been realized through the Matlab language. It should be noted that achieving the above effects has nothing to do with the specific programming language used. In fact, for any language, as long as the above method principles are mastered, similar effects can be achieved.

[0071] To verify the actual effect of the above visualization method and ensure the correctness of the relevant calculations in the visualization, we applied the visualization algorithm to the 2D-3D registration scenario where the 3D model segmented from CT is registered into the biplane space. If the visualization algorithm is implemented correctly, since the model is imported from an external CT, the correctness of its shape is guaranteed by the CT. Then, as long as the model is registered in the correct spatial pose, after setting it to be transparent, it is expected to cover the underlying X-ray map well. Similarly, in the 2D visualization, its projected contour is expected to fit well with the edges of the bony structures in the 2D full-length image.

[0072] It should be noted that when we gave an example of the 2D-3D registration embodiment, it was entirely out of consideration of the control variable method to facilitate the verification of the accuracy of the visualization algorithm. Because when the visualization algorithm has not been verified to be accurate, directly displaying the model reconstructed from the biplane, due to the errors in the biplane reconstruction algorithm, the shape and pose of the reconstructed model may not be accurate. In this case, when the model cannot match the spatial map or its contour cannot match the edges of the bony structures in the 2D image, it is impossible to determine whether it is a problem of visualization. As can be seen from the following display figures, the visualization algorithm is accurate. As long as the shape and pose of the model are correct, the visualization algorithm can show a perfect match. Then, when applying this algorithm to import the model reconstructed from the biplane, if a mismatch occurs, it can be determined that it must be a problem of the reconstruction algorithm.

[0073] In addition, this embodiment demonstrates the matching situation of the lumbar vertebrae L3-L5. However, in the process of implementing this algorithm, it does not depend on any anatomical information. Any structure is treated as a general 3D surface model. Therefore, this visualization method can be applied to other bones, prostheses, organs, or even any object structure beyond the medical field, and it is a general visualization method.

[0074] The specific steps and results of the experiments we conducted are as follows: Position the human body phantom in the standing pose required by the protocol during actual shooting, as Figure 5 shown, and take the full-length X-ray slit image and obtain the spliced image.

[0075] As Figure 6 shown, use commercial segmentation software to segment the lumbar vertebrae L3-L5 from the CT of the phantom and mark 4 key points.

[0076] As Figure 7 shown, mark the projection points of these key points on the full-length anteroposterior and lateral view images.

[0077] As Figure 8 shown, the algorithm will perform initial registration and pop up an interface for the user to view the matching situation from each perspective. It can be seen that the irrelevant slice images have been hidden, and the median slice image (the slice image corresponding to the middle L4) is displayed by default.

[0078] As Figure 9 shown, the user can freely select the frontal view slice image and the lateral view slice image through shortcut keys. In order to show the independence of the slice images shown in the two perspectives, we specifically select the slice image corresponding to L3 for the frontal view and the slice image corresponding to L5 for the lateral view. It can be seen that the frontal view slice image shown is higher in position in space than the lateral view slice image, which is consistent with the fact that the position of L3 is higher than that of L5.

[0079] If the situation of mismatch between the transparent model and the image occurs, the user can also use the shortcut keys to perform translation and rotation operations on the pose of the model as in step S4. As Figure 10 shown, for example, the lower right corner of L4 does not match well with the lower right corner part in the slice image (as shown by the lower arrow), then the user hopes to rotate the model counterclockwise in the window plane (as shown by the upper arrow, equivalent to positive rotation around the axis out of the screen), or translate the model as a whole to the right side of the window. The user can select an appropriate adjustment step size according to the degree of mismatch. As Figure 10 shown, the rotation step size is set to adjust 0.1° each time, and the translation step size is set to 0.1 mm. After adjustment, it is found that the matching degree has improved.

[0080] After the adjustment is completed, the user can also call out the 2D visualization interface to fully display the contour line in the full-length image. Figure 11 The full-length image and the full-length image with the projection of the contour line superimposed are respectively shown on the left and right sides.

[0081] Third Embodiment As Figure 12 shown, this embodiment provides a visualization system for three-dimensional model modeling and two-dimensional image matching degree for performing the visualization method of three-dimensional model modeling and two-dimensional image matching degree in the first embodiment, including: Slice image coordinate determination module 1, which is used to calculate the spatial mapping corner points of the slice image for splicing the full-length image generated by cone-beam projection scanning in the double-plane coordinate system, so as to calculate the coordinates of each point on the slice image according to the spatial mapping corner points; The initial position registration module 2 is configured to select bony key points for marking on the three-dimensional model, calculate the coordinates of the bony key points in the biplane coordinate system based on the coordinates of the marked point projections of the bony key points in the frontal and lateral views on the slice image, so as to complete the initial position registration of the three-dimensional model in the biplane coordinate system; The 3D visualization display module 3 is configured to retain the slice images related to the three-dimensional model of interest in the current visualization, and preferentially display the slice image with the median ID number among the relevant slice images, so as to realize the 3D visualization display of the three-dimensional model and the slice images; The 3D visualization interaction module 4 is configured to provide the user with interactions including changing the frontal and lateral views, switching the currently displayed frontal and lateral slice images, and adjusting the pose of the three-dimensional model based on the 3D visualization window; The 2D visualization interaction module 5 is configured to realize the 2D visualization display of the matching degree between the three-dimensional model and the slice images.

[0082] A computer-readable storage medium stores computer code, and when the computer code is executed, the above method is executed. Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0083] The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

[0084] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0085] It should be noted that the above embodiments can be freely combined as needed. The above are only the preferred embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for visualizing the matching degree between three-dimensional model building and two-dimensional image, characterized in that: The following steps are involved: S1: calculating the spatial mapping corner points of a slice image for stitching a full-length image generated by cone beam projection scanning in a dual-plane coordinate system, so as to calculate the coordinates of each point on the slice image according to the spatial mapping corner points; S2: Selecting bony key points for marking on the three-dimensional model, and calculating the coordinates of the bony key points in the biplane coordinate system based on the coordinates of the marked points of the bony key points in the frontal and lateral positions projected on the slice image, so as to complete the initial position registration of the three-dimensional model in the biplane coordinate system; S3: retaining the slice images related to the three-dimensional model of interest in the current visualization, and preferentially displaying the slice images with the median ID number among the related slice images, so as to realize the 3D visualization display of the three-dimensional model and the slice images; S4: The user performs interactions based on the 3D visualization window, including changing the frontal and lateral viewing angles, switching the currently displayed frontal and lateral image slices, and adjusting the posture of the three-dimensional model.

2. The method for visualizing the matching degree between three-dimensional model building and two-dimensional image according to claim 1, characterized in that: Before step S1, the positions of the light source and the plate corners during actual exposure are calculated, specifically: According to the travel of the rack, calibrate the position information of the light source and the corner points of the plate at several heights at equal intervals, and record the corresponding PLC height for each calibrated height; The exposure PLC height when the sheet image is actually exposed is obtained and written into the data element tag. The exposure PLC height is used as an independent variable, and the position information of the light source and the flat panel corner points at several calibrated heights is used as a basis, and an interpolation or regression algorithm is used to calculate the position information of the light source and the flat panel corner points during actual exposure.

3. The method for visualizing the matching degree between three-dimensional model building and two-dimensional image according to claim 2, characterized in that: In step S1, the spatial mapping corner points of the slice image for stitching the full-length image generated by cone beam projection scanning in the biplane coordinate system are calculated, so as to calculate the coordinates of each point on the slice image according to the spatial mapping corner points, specifically: The slit scanning adopted by the cone beam projection scanning is realized by setting a central photosensitive ROI on the flat plate, and the physical space area corresponding to the slice image actually obtained is the area of ​​the central photosensitive ROI; Reading the ROI position information of the photosensitive ROI in the flat panel in the data element tag, and calculating the position information of the ROI corner point according to the position information of the flat panel corner point during the actual exposure, wherein the position information of the ROI corner point is also the spatial mapping corner point of the sheet image actually scanned and generated in the dual-plane coordinate system; The coordinates of each point on the slice image are calculated according to the spatial mapping corner points.

4. The method for visualizing the matching degree between three-dimensional model building and two-dimensional image according to claim 1, characterized in that: In step S2, a bony key point for marking on the three-dimensional model is selected, and based on the coordinates of the bony key point projected on the sheet image in the anteroposterior and lateral positions, the coordinates of the bony key point in the biplane coordinate system are calculated to complete the initial position registration of the three-dimensional model in the biplane coordinate system, specifically: After identifying the projections of the marker points of the bony key points in the anteroposterior and lateral positions on the spliced ​​full-length images in the anteroposterior and lateral positions respectively, the splicing algorithm maps the coordinates on the spliced ​​full-length image back to the slice image corresponding to the ID number and the coordinates on the slice image; Based on the coordinates of the bony key points projected on the image of the film at the anteroposterior and lateral marking points, and the coordinates of the anteroposterior and lateral light sources, the coordinates of the bony key points in the biplane coordinate system are calculated using a triangulation method; After marking the bony key points on the three-dimensional model, the three-dimensional model is transferred from its own local coordinate system to the dual-plane coordinate system to achieve initial positioning of the three-dimensional model and the two-dimensional slice image.

5. The method for visualizing the matching degree between three-dimensional model building and two-dimensional image according to claim 3, characterized in that: In step S3, the slice images related to the three-dimensional model of interest in the current visualization are retained, and the slice images with ID numbers in the median are preferentially displayed among the related slice images, specifically: For the three-dimensional model of interest in the current visualization, only some of the slice images at the corresponding heights record relevant projection information during scanning, and the slice images at other heights that do not capture the corresponding structure of the three-dimensional model are irrelevant to the current visualization session; After the three-dimensional model is imported and the initial positioning posture is determined, for the frontal and lateral view angles, all the slice images are traversed respectively, and a spatial pyramid geometry is constructed for the spatial mapping corner points of each slice image and the light source placed at the corresponding position based on the spatial calibration environment, and a 3D spatial intersection is performed with the three-dimensional model. Records with intersections are related, and records without intersections are irrelevant. After determining the relevance, the slice images that retain a continuous ID in the frontal and lateral views are related in this visualization session; The frontal and lateral views respectively give priority to displaying the slice images with the median ID number in the relevant slice images, and hide the other slice images, and the open display rights of the other slice images are switched by the user during interaction.

6. The method for visualizing the matching degree between three-dimensional model building and two-dimensional image according to claim 1, characterized in that: In step S4, the user performs interactions based on the 3D visualization window, including changing the frontal and lateral view angles, switching the currently displayed frontal and lateral image slices, and adjusting the posture of the three-dimensional model, specifically: Changes in frontal and lateral perspectives: Switching the viewpoint to the light source corresponding to the image of the sheet currently displayed in the front or side position, and switching the sight direction to look toward the image of the sheet; Switching between the currently displayed frontal and lateral images: The IDs of the image displayed in the front view and the image displayed in the side view are adjusted independently. If there are m images of the image related to the front view and n images of the image related to the side view, there are m*n combinations of IDs of the image of the front view and the image of the side view displayed simultaneously in the space. When any one image of the image is displayed in each viewing angle, the other images of the image in the viewing angle are hidden to ensure the neatness of the interface. Interactions including the 3D model pose adjustment: Based on the frontal and side perspectives, the posture of the three-dimensional model that conforms to the interactive intuitive habits is adjusted, including up, down, left, and right displacement adjustments relative to the frontal and side perspectives, and rotation in three directions along the horizontal axis, vertical axis, and axis out of the screen pointing to the user.

7. The method for visualizing the matching degree between three-dimensional model building and two-dimensional image according to claim 1, characterized in that: The method further includes realizing a 2D visual display of the three-dimensional model and the slice image, specifically: Based on each of the slice images, determine the contour edge of the three-dimensional model when the light source corresponding to the slice image is used as a viewpoint, wherein the edge on the three-dimensional model in the form of a surface model is classified as the contour edge if it contains two adjacent face patches, and the normal vectors of the two adjacent face patches are in opposite directions relative to the vector from the viewpoint to the midpoint of the edge; Projecting the contour edge onto the relevant slice image to obtain the start and end point coordinates of the contour line, and discarding the contour edge projection that exceeds the slice image range; Using the splicing relationship between the slice image and the spliced ​​full-length image, the coordinates of the contour line under the slice image are mapped to the spliced ​​full-length image, and are drawn on the spliced ​​full-length image in the form of colored lines to achieve 2D visualization of the three-dimensional model and the slice image.

8. A system for visualizing the matching degree of a three-dimensional model and a two-dimensional image for executing the method for visualizing the matching degree of a three-dimensional model and a two-dimensional image as claimed in any one of claims 1 to 7, characterized in that: include: A slice image coordinate determination module is used to calculate the spatial mapping corner points of the slice image for stitching the full-length image generated by cone beam projection scanning in a dual-plane coordinate system, so as to calculate the coordinates of each point on the slice image according to the spatial mapping corner points; An initial position registration module is used to select bony key points for marking on the three-dimensional model, and calculate the coordinates of the bony key points in the biplane coordinate system based on the coordinates of the marked points of the bony key points in the frontal and lateral positions projected on the slice image, so as to complete the initial position registration of the three-dimensional model in the biplane coordinate system; A 3D visualization display module is used to retain the slice images related to the three-dimensional model of interest in the current visualization, and preferentially display the slice images with the median ID number among the related slice images, so as to realize the 3D visualization display of the three-dimensional model and the slice images; A 3D visualization interaction module is used to provide the user with interactions based on the 3D visualization window, including changes in the frontal and lateral view angles, switching between the currently displayed frontal and lateral image slices, and adjusting the posture of the three-dimensional model; The 2D visualization interaction module is used to realize the 2D visualization display of the matching degree between the three-dimensional model and the slice image.

9. A computer device comprising a memory and one or more processors, wherein the memory stores computer codes, and when the computer codes are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7. 10 . A computer-readable storage medium storing a computer code. When the computer code is executed, the method according to claim 1 is executed.

Images