Reference marking system, three-dimensional reconstruction method and electronic equipment

Abstract

The invention provides a reference marking system, a three-dimensional reconstruction method and electronic equipment, and relates to the technical field of three-dimensional reconstruction. The marking assembly comprises at least four non-coplanar markers with different target features, and the target features of the markers have projection stability; the marking assembly has full-view projection specificity, and two-dimensional projection images of the marking assembly are in one-to-one correspondence with three-dimensional space postures of the marking assembly. The reference marking system is suitable for X-ray imaging and natural light imaging, and can realize accurate identification of projection positions of all markers, so that accurate spatial position and attitude information can be provided for three-dimensional reconstruction, high-precision three-dimensional reconstruction can be performed from a single or sparse angle, and the spatial positioning precision and the reconstruction efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a reference marking system, a three-dimensional reconstruction method and electronic equipment. Background Art

[0002] In recent years, deep learning-based 3D reconstruction technology for X-ray images has made significant progress. Existing deep learning-based 3D reconstruction methods for X-ray images include methods that use convolutional neural networks to reconstruct 3D models from a single X-ray image and deep learning algorithms that use mammal skull X-ray images to reconstruct 3D CT images. These methods mainly rely on prior knowledge of human anatomical structures. In the absence of clear reference points, the accuracy and robustness of reconstruction are still challenging.

[0003] In the field of machine vision, fiducial marker systems such as ArUco (Augmented Reality Uniform Coordinate Object) markers are widely used in camera pose estimation and augmented reality. Among them, fiducial marker systems are mainly used for positioning and calibration. They are usually markers placed in fixed positions to help sensors or cameras determine the positional relationship relative to these markers. ArUco markers are a type of fiducial marker system consisting of a series of black and white squares. However, these systems are mainly designed for natural light imaging and are not suitable for penetrating imaging modalities such as X-rays. Recently, Cai et al. proposed a method called X-Gaussian, which significantly improved the accuracy and speed of X-ray 3D reconstruction by introducing 3D Gaussian point clouds and radiation intensity modeling mechanisms. However, like previous methods, the X-Gaussian method still assumes that the camera pose information is known and accurate. This assumption limits these methods to devices that can provide accurate pose information, such as CT scanners. For ordinary X-ray machines, these methods are difficult to apply directly due to the lack of accurate pose information. Therefore, it is still challenging to achieve accurate 3D reconstruction on a wider range of X-ray imaging devices. Summary of the invention

[0004] The purpose of the present invention is to provide a reference marking system, a three-dimensional reconstruction method and an electronic device to provide accurate spatial position and posture information and improve spatial positioning accuracy and reconstruction efficiency.

[0005] In a first aspect, the present invention provides a reference mark system, the reference mark system comprising a mark assembly;

[0006] The marking component includes at least four non-coplanar markers with different target features, and the target features of the markers have projection stability; the marking component has full-viewing projection specificity, and the two-dimensional projection image of the marking component corresponds one-to-one with its three-dimensional spatial posture.

[0007] Furthermore, the target features include one or more of relative size, color, grayscale and surface texture.

[0008] Furthermore, the marking component has a rotation center for providing a projection determination point, the rotation center is used as the coordinate origin of a normalized projection coordinate system, and the real size of the marker is used to provide normalization parameters for the normalized projection coordinate system.

[0009] Further, the rotation center includes a sphere or an intersecting line segment.

[0010] Furthermore, the reference marking system is applied to X-ray imaging, and the marking assembly is a non-coplanar rigid structure composed of at least four spheres of different sizes, and the spheres are made of a material that is X-ray developable but has a certain degree of transmittance.

[0011] Furthermore, the reference marking system is applied to natural light imaging, and the marking component is a non-coplanar rigid structure composed of at least four spheres of different colors. Each sphere is fixed by a preset material, and the transparency of the preset material reaches a preset transparency threshold and the refractive distortion rate is less than a preset distortion rate threshold.

[0012] Furthermore, there are multiple marking components.

[0013] In a second aspect, the present invention further provides a three-dimensional reconstruction method based on the reference marking system of the first aspect, comprising:

[0014] Acquire a target projection image captured by a target imaging device for a scene to be tested where a reference marker system is placed;

[0015] Estimate the position and posture of the target imaging device according to the target projection image to obtain the position and posture information of the target imaging device;

[0016] The target projection image is used to perform three-dimensional reconstruction of the object to be imaged in the test scene according to the position and posture information of the target imaging device to obtain a three-dimensional reconstruction result.

[0017] Furthermore, the marking component has a rotation center for providing a projection determination point; the pose information of the target imaging device is obtained by estimating the pose of the target imaging device according to the target projection image, including:

[0018] Extract features from the target projection image to obtain original projection position information corresponding to each marker and the rotation center, as well as projection size information of a reference marker; wherein the reference marker is one of the markers;

[0019] According to the original projection position information of the rotation center and the projection size information and true size information of the reference marker, the original projection position information of each marker is normalized in coordinate system and size, so as to obtain the target projection position information of each marker in the normalized projection coordinate system; wherein the coordinate origin of the normalized projection coordinate system is the rotation center;

[0020] According to the target projection position information of each marker, the pose information of the target imaging device is calculated by a preset pose estimation algorithm; wherein the pose estimation algorithm includes a deep learning algorithm and / or a geometric algorithm.

[0021] In a third aspect, the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the three-dimensional reconstruction method of the second aspect is implemented.

[0022] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the three-dimensional reconstruction method of the second aspect is executed.

[0023] In the reference marker system, three-dimensional reconstruction method and electronic device provided by the embodiments of the present invention, the reference marker system includes a marker assembly; the marker assembly includes at least four non-coplanar markers with different target features, and the target features of the markers have projection stability; the marker assembly has full-viewing projection specificity, and the two-dimensional projection image of the marker assembly corresponds one-to-one with its three-dimensional spatial posture. The reference marker system is suitable for X-ray imaging and natural light imaging. Since the target features of the markers in the marker assembly of the reference marker system have projection stability, accurate identification of the projection position of each marker can be achieved, thereby providing accurate spatial position and posture information for three-dimensional reconstruction; and since the marker assembly has full-viewing projection specificity, the two-dimensional projection image of the marker assembly corresponds one-to-one with its three-dimensional spatial posture, so high-precision three-dimensional reconstruction can be performed from a single or sparse angle, improving spatial positioning accuracy and reconstruction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A structural design diagram of a reference marking system suitable for natural light imaging provided by an embodiment of the present invention;

[0026] Figure 2 for Figure 1 Projection diagram of the reference marker system shown in some poses;

[0027] Figure 3 for Figure 1 Deep learning results of the spatial pose corresponding to the reference marker system shown;

[0028] Figure 4 for Figure 1 Prediction accuracy plot corresponding to the reference marking system shown;

[0029] Figure 5 A structural design diagram of a reference marking system suitable for X-ray imaging provided by an embodiment of the present invention;

[0030] Figure 6 for Figure 5 Projection diagram of the reference marker system shown in some poses;

[0031] Figure 7 A structural design diagram of another reference marking system suitable for X-ray imaging provided by an embodiment of the present invention;

[0032] Figure 8 for Figure 7 Projection diagram of the reference marker system shown in some poses;

[0033] Fig. 9 A deep learning result of a spatial posture corresponding to a reference marker system suitable for X-ray imaging provided in an embodiment of the present invention;

[0034] Fig.10 A prediction accuracy map corresponding to a reference marker system suitable for X-ray imaging provided by an embodiment of the present invention;

[0035] Fig.11 A schematic diagram of a flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention;

[0036] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] At present, the fields of X-ray imaging and natural light imaging lack a unified spatial reference marking system applicable to all viewing angles. Existing polyhedral structure markings based on surface QR codes have limitations in full-view applications and X-ray imaging. In particular, in X-ray imaging, there is a lack of standardized reference objects that can provide accurate spatial position and posture information, which limits the ability to accurately reconstruct three-dimensional images from sparse angle X-ray images. Based on this, a reference marking system, a three-dimensional reconstruction method, and an electronic device provided by an embodiment of the present invention can be applied to multimodal imaging of X-ray imaging and natural light imaging. When a reference marking system based on a combination of non-coplanar feature points with full-view projection specificity is applied to three-dimensional reconstruction, high-precision three-dimensional spatial information extraction and reconstruction from a single or sparse angle can be achieved.

[0039] To facilitate understanding of this embodiment, a reference mark system disclosed in an embodiment of the present invention is first introduced in detail.

[0040] An embodiment of the present invention provides a reference marking system, which includes a marking assembly; the marking assembly includes at least four non-coplanar markers with different target features, and the target features of the markers have projection stability; the marking assembly has full-viewing angle projection specificity, and the two-dimensional projection image of the marking assembly corresponds one-to-one to its three-dimensional spatial posture.

[0041] The above-mentioned reference marker system can be applied to X-ray imaging, natural light imaging, and multimodal applications combining natural light and X-ray, as long as the target features of each marker under the corresponding imaging conditions are different and have projection stability.

[0042] Since the target features of the markers in the marker components of the reference marker system have projection stability, the projection positions of each marker can be accurately identified, thereby providing accurate spatial position and posture information for three-dimensional reconstruction; and since the marker components have full-view projection specificity, the two-dimensional projection image of the marker components corresponds one-to-one with its three-dimensional spatial posture, thus ensuring the uniqueness of the PnP (especially EPnP) solution, so that high-precision three-dimensional reconstruction can be performed from a single or sparse angle, improving spatial positioning accuracy and reconstruction efficiency. Among them, PnP refers to Perspective-n-Point, which is a method for solving the correspondence between 3D and 2D points; EPnP refers to Efficient Perspective-n-Point, which is an efficient algorithm for solving the PnP problem.

[0043] Optionally, the above-mentioned marking assembly may be one or more. When the reference marking system includes multiple marking assemblies, the placement angles of the marking assemblies may be different from each other, which can reduce the unpredictability caused by shielding between markers in the marking assembly; multiple marking assemblies with different placement angles can refer to each other for correction, thereby improving the robustness and accuracy of the reference marking system.

[0044] Optionally, the target features may include one or more of relative size, color, grayscale, and surface texture. Among them, color, grayscale, and surface texture can all be used in the reference mark system of natural light imaging, and relative size and grayscale (i.e., X-ray developability) can all be used in the reference mark system of X-ray imaging. It can be seen that relative size can be applicable to both X-ray imaging and natural light imaging. Specifically, the target features can be spheres of different sizes, marks of different colors, areas with different grayscale values, or special textures / patterns, etc., or any combination of these features.

[0045] It should be noted that the embodiment of the present invention does not limit the geometric shapes of the target features and the markers (ie, the markers are not limited to spheres). In other embodiments, the target features may also adopt other features, and the markers may also adopt other geometric shapes.

[0046] When there are multiple marking components, each marking component can be the same, or different from each other, or some marking components can be the same. In different marking components, the relative position and / or target characteristics of each marker are different. The different target characteristics include different types of target characteristics and / or different characteristic values. For example, two marking components respectively select relative size and color as target characteristics; for another example, two marking components both select relative size as target characteristics, but the sizes of their markers are different.

[0047] Optionally, the above-mentioned marking component has a rotation center for providing a projection determination point, the rotation center is used as the coordinate origin of the normalized projection coordinate system, and the real size of the marker is used to provide a normalization parameter for the normalized projection coordinate system. By normalizing the two-dimensional projection image of the marking component to the normalized projection coordinate system, anti-scaling and anti-translation properties can be achieved, so that the reference marking system has anti-scaling and anti-translation properties.

[0048] Optionally, the above-mentioned rotation center may include a sphere or intersecting line segments. The sphere may be a marker in the marking assembly or may not be a marker. The intersecting line segments may be composed of at least two perpendicularly intersecting line segments or at least two non-perpendicularly intersecting line segments. It should be noted that the embodiment of the present invention does not limit the object type of the rotation center, that is, the rotation center is not limited to a sphere or an intersecting line segment, as long as the rotation center can be easily identified from its two-dimensional projection image.

[0049] The rotation center of the marking component can be provided with a structure of a projection determination point using a sphere or intersecting line segments, etc., which is used as the origin of the projection coordinates. Combined with the projection size information of the marker (such as the diameter of the projection circle corresponding to the spherical marker) and the actual size information, the projection coordinates can be normalized to make the projection coordinates resistant to scaling and translation.

[0050] When there are multiple marker components, each marker component normalizes its own projection and performs its own pose estimation.

[0051] Optionally, when the reference marking system is applied to X-ray imaging, the marking assembly may be a non-coplanar rigid structure composed of at least four spheres of different sizes, and the spheres are made of a material that is X-ray-developable but has a certain degree of transmittance. Certain transmittance means that the transmittance is within a preset transmittance range; wherein the transmittance range can be set according to actual needs and is not limited here, for example, the transmittance range is 20% to 80%.

[0052] In one possible implementation, the reference mark system consists of at least five spheres of different sizes, one of which serves as a rotation center to provide a reference for normalization of projection coordinates. The spheres are connected by connecting rods, which are made of X-ray low-developing materials. The spheres are made of X-ray-developing but highly transparent materials to maintain recognition in the case of overlapping shielding.

[0053] In another possible implementation, the reference mark system includes at least four spheres of different sizes, each of which is fixed by a connecting rod; the intersection of each connecting rod serves as the rotation center of the reference mark system, and is set as the coordinate origin, which, together with the diameter of the specified sphere, provides a reference for normalizing the projection coordinates; the material of the connecting rods is selected as X-ray development material, and the spheres use X-ray development but certain transmittance material to facilitate computer vision recognition.

[0054] Optionally, when the reference marking system is applied to natural light imaging, the marking component may be a non-coplanar rigid structure composed of at least four spheres of different colors, each sphere being fixed by a preset material, the transparency of the preset material reaching a preset transparency threshold and the refractive distortion rate being less than a preset distortion rate threshold. The transparency threshold and the distortion rate threshold may be set according to actual needs and are not limited here. For example, the transparency threshold is 95% and the distortion rate threshold is 20%.

[0055] Optionally, the preset material may be polycarbonate, which has the characteristics of high transparency and low refractive distortion, and can achieve a rigid structure while maintaining full-angle visibility.

[0056] In one possible implementation, the reference mark system consists of four non-coplanar spheres of different colors forming a rigid structure. These spheres are non-coplanarly arranged in three-dimensional space to ensure projection specificity under all viewing angles. A sphere of a different color is placed at the center of rotation as an additional mark, and its two-dimensional projection center serves as the coordinate origin and the projection diameter together provide a reference for normalization of the projection coordinates.

[0057] When using the above-mentioned reference marker system for three-dimensional reconstruction, the following steps may be included: a) placing the reference marker system near the object to be imaged; b) acquiring one or more two-dimensional projection images containing the reference marker system and the object to be imaged; c) identifying and extracting features of the reference marker system from the two-dimensional projection images; d) based on the extracted features, using a pre-trained deep learning algorithm or a geometric algorithm such as PnP to determine the three-dimensional posture of the reference marker system; e) using the determined three-dimensional posture information of the reference marker system as a spatial reference, deriving the posture information of an imaging device such as a camera, and calculating the translation information in combination with the posture information. The translation information and the posture information together provide an external parameter matrix of the camera in the three-dimensional reconstruction to reconstruct the three-dimensional structure of the object to be imaged.

[0058] A key feature of the embodiment of the present invention is to use the rotational posture of the reference marker system around its own rotational center to predict the rotational posture of the reference marker system around its rotational center. Specifically, we first perform pre-training and geometric calculations on the rotation of the reference marker system. Then, through a transformation matrix, we can convert the rotational posture of the reference marker system into the rotational posture of the imaging device. This method not only simplifies the training process, but also improves the flexibility and accuracy of the system in practical applications.

[0059] For ease of understanding, the relationship between the rotation of the reference mark system and the rotation of the imaging device around the reference mark system is introduced below.

[0060] There are two rotation situations: a. The rotation of the reference marker system around its own rotation center (used for pre-training and geometric calculation); b. The rotation of the imaging device around the rotation center of the reference marker system (actual application scenario).

[0061] Definition: Rm is the rotation matrix of the reference marker system relative to its initial position; Rc is the rotation matrix of the imaging device relative to its initial position; T is the transformation matrix from the reference marker system coordinate system to the imaging device coordinate system.

[0062] Then, the conversion of the reference mark system rotation to the imaging device rotation can be expressed as:

[0063] Rc=T×Rm×T -1 ;

[0064] Among them, T -1 is the inverse matrix of T.

[0065] For continuous rotation, at two moments t1 and t2, the rotation change of the marker is ΔRm, and the corresponding camera rotation change ΔRc can be expressed as:

[0066] ΔRc=Rc(t2)×Rc(t1) -1 =(T×Rm(t2)×T^ -1 )×(T×Rm(t1)×T -1 ) -1 .

[0067] The equivalence is explained as follows:

[0068] The equivalence of the reference mark system rotation and the imaging device rotation is based on several key points:

[0069] a) Relative motion: In a visual system, the visual effects produced by the rotation of the reference mark system and the rotation of the imaging device are relative.

[0070] b) Coordinate transformation: Through appropriate coordinate transformation (such as T and T in the above formula -1 ), it is possible to transform between the reference marker system coordinate system and the imaging device coordinate system. This allows us to convert a rotational representation in one system to an equivalent representation in the other.

[0071] c) Invariance: The fundamental properties of the rotation (such as angle and axis) remain unchanged across this transformation. This means that we can use the rotation of the reference marker system to accurately infer the rotation of the imaging device.

[0072] d) Computational simplification: In general, a fixed imaging device observing a rotating object is easier to simulate and calculate than a moving imaging device observing a fixed object. Therefore, we can use the data of the reference marker system rotation to train the model and then apply it to the actual imaging device motion scenario.

[0073] e) Generalization: This equivalence allows the system to generalize better. The system can be equally applicable to situations where the imaging device is moving, even though the data of the reference marker system rotation is used during training, which increases the flexibility and applicability of the system.

[0074] By exploiting this equivalence, the system can be trained and developed in a simplified environment while still being able to be effectively applied to realistic imaging device motion scenarios, which is an important advantage of the approach.

[0075] This rotational posture conversion method has the following advantages: ① It simplifies the pre-training process; ② It improves the adaptability and versatility of the system; ③ It reduces the computational complexity in practical applications. Potential applications: augmented reality and robot vision, etc.

[0076] Embodiments of the present invention relate to a novel reference marker system for multimodal imaging and its application method. The system includes at least four non-coplanar regional points with different characteristics, forming a spatial structure with full-view projection specificity. Through its two-dimensional projection in imaging, combined with computer vision and deep learning algorithms, the position, size and posture information of the marker in three-dimensional space can be accurately inferred. Embodiments of the present invention provide high-precision spatial references for X-ray films and natural light images, significantly improving the accuracy of reconstructing three-dimensional structures from single or sparse angle images. The method can be applied to multiple fields such as medical imaging, industrial non-destructive testing and augmented reality, and helps to improve spatial positioning accuracy and reconstruction efficiency.

[0077] For ease of understanding, the above-mentioned reference marking system and its application are introduced in detail below.

[0078] 1. Natural light imaging reference marker system

[0079] Design description: a) The marker assembly consists of four non-coplanar spheres of different colors to form a rigid structure. A sphere of different color is placed at the center of rotation as an additional marker, and its two-dimensional projection center is used as the coordinate origin and the projection diameter together to provide a normalized reference for the two-dimensional projection coordinates (or other designs such as a combination of vertical segments are used to provide the rotation center positioning). b) These spheres are arranged non-coplanar in three-dimensional space to ensure projection specificity at all viewing angles. c) In order to achieve a rigid structure while maintaining full-angle visibility, polycarbonate material can be used to fix the spheres. Polycarbonate has the characteristics of high transparency and low refractive distortion, which is very suitable for this application. d) To improve the robustness of the system, two or more marker assemblies placed at preset angles can be used to avoid simultaneous occlusion. e) In the projection image, the center of the central sphere is used as the coordinate origin, and the diameter of the sphere is used as a ruler to provide a normalized projection coordinate system. This design is resistant to scaling and translation, and can be used for deep learning or geometric algorithms to calculate rotation posture.

[0080] Figure 1 shows a structural design diagram of a reference marking system suitable for natural light imaging, Figure 2 for Figure 1 Projection diagram of the partial pose of the reference marker system shown, Figure 3 for Figure 1 Deep learning results of the spatial pose corresponding to the reference marker system shown, Figure 4 for Figure 1Prediction accuracy plot corresponding to the reference marking system shown.

[0081] Figure 3 The model loss curves corresponding to the training loss and validation loss under the natural light imaging reference marking system are shown as follows: Figure 3 As shown in the figure, both the training loss and the validation loss decrease rapidly and stabilize after about 20 training rounds. This shows that the model learns well without obvious overfitting or underfitting.

[0082] Figure 4 The comparison diagram between the predicted value and the true value under the natural light imaging reference marking system is shown in FIG. Figure 4 As shown in the figure, the actual prediction results of the model are highly consistent with the ideal prediction results (that is, completely accurate predictions), and most of the points are concentrated near the diagonal. This shows that the actual predictions of the model are very close to the ideal predictions and the prediction accuracy is very high.

[0083] In summary, Figure 3 and Figure 4 The training process and prediction results of a deep learning model for projection and attitude are shown. The loss curve shows that the model converges well, and the comparison chart of the predicted value and the true value intuitively shows the high accuracy of the model prediction. The high overlap between the actual prediction results and the ideal prediction results shows that the model performs well in projection and attitude prediction tasks, and can provide accurate and reliable support for the geometric calibration of natural light imaging systems.

[0084] A possible implementation step of the above-mentioned natural light imaging reference marker system is as follows: a) Make a marker component of the reference marker system: use five spheres of different colors and fix them into a rigid structure with polycarbonate material. b) Place the reference marker system: place one or more marker components in the scene to be measured (including the object to be imaged). c) Image acquisition: use a camera or other natural light imaging device to capture the scene image containing the reference marker system. d) Feature extraction: identify and extract the color and position information of the sphere from the image, where the color is used to distinguish each sphere, and the position information can be the plane coordinate information of the projection of the center of the sphere. e) Pose estimation: use a pre-trained deep learning model or a geometric algorithm such as EPnP to calculate the pose information (such as the rotation matrix) of the camera relative to the marker based on the extracted features, and can further calculate the translation information such as the translation vector based on the pose information. f) Application: The pose information such as the calculated pose information and translation information is used for applications such as augmented reality and robot navigation.

[0085] 2. X-ray imaging reference marker system

[0086] Taking into account the particularity of X-ray imaging, especially the fact that the C-arm X-ray machine mainly rotates around the cranial / caudal axis in clinical applications, two X-ray imaging reference marker systems were designed.

[0087] Design 1: a) The reference mark system consists of at least five spheres of different sizes, one of which serves as a rotation center to provide a reference for normalizing the coordinate system. b) The connecting rod is made of a material with low X-ray imaging, and the sphere is made of a material with X-ray imaging but a certain degree of transparency to maintain identification in the case of overlapping shielding. Figure 5 The structural design diagram of the reference marking system under this design scheme is shown. Figure 6 for Figure 5 Projection diagram of some poses of the reference marker system shown.

[0088] Design Solution 2: a) Use X-ray developed connecting rods to form the main structure of the reference mark system. b) The reference mark system contains at least four spheres of different sizes to provide size scales and additional spatial reference points. c) Use computer vision algorithms to identify the intersection of the connecting rods in the projection, set it as the coordinate origin, and combine it with the diameter of the specified sphere to provide a reference for normalizing the projection coordinates. d) The connecting rod material is selected from X-ray developed materials. Figure 7 The structural design diagram of the reference marking system under this design scheme is shown. Figure 8 for Figure 7 Projection diagram of some poses of the reference marker system shown.

[0089] Fig. 9 The model loss curves corresponding to the training loss and validation loss under the X-ray imaging reference marker system are shown. Fig.10 A comparison chart of the predicted value and the actual value under the X-ray imaging reference marker system is shown. Fig. 9 and Fig.10 The training process and prediction results of a deep learning model for X-ray projection and posture are shown. The loss curve shows that the model converges well, while the comparison chart between the predicted value and the true value intuitively demonstrates the high accuracy of the model prediction.

[0090] like Fig.10 As shown in the figure, the actual prediction results of the model are highly consistent with the ideal prediction results, and most points are concentrated near the diagonal. This shows that the actual prediction of the model is very close to the ideal prediction and the prediction accuracy is very high; the model performs well in projection and posture prediction tasks and can provide accurate and reliable support for the geometric calibration of X-ray imaging systems.

[0091] A possible implementation step of the above X-ray imaging reference marking system is as follows: a) making a marking system:

[0092] For option 1: Use carbon fiber reinforced polymer (CFRP) to make the connecting rod, and precisely fix the sphere on the carbon fiber structure (i.e., the connecting rod) to form a designed non-coplanar rigid structure. CFRP has the characteristics of good X-ray transmittance, high strength and light weight, and is almost invisible on X-ray images to avoid interfering with sphere identification. Use titanium alloy (such as Ti-6Al-4V) or tungsten alloy to make spheres of different sizes. These materials have moderate development under X-rays and will not completely block X-rays. They also have good biocompatibility and corrosion resistance.

[0093] For the second solution: the connecting rod uses X-ray developing material to facilitate computer vision recognition. Titanium alloy (such as Ti-6Al-4V) or tungsten alloy is also used to make spheres of different sizes.

[0094] b) Placement of reference marker system: Taking orthopedic surgery (such as long bone fracture reduction and internal fixation or external fixation surgery) as an example, before or during surgery, the reference marker system is fixed at an appropriate position near the object to be imaged, or fixed to the bone cortex through bone screws, etc., to ensure that the long axis is consistent with the head and tail axis and the relative position with the anatomical structure is fixed. The lightweight design of the reference marker system will not bring additional burden to the object to be imaged. c) X-ray imaging: Use, for example, a C-arm X-ray machine to obtain an X-ray image containing the reference marker system. d) Feature extraction: Identify and extract the position information of spheres of different sizes from the X-ray image (that is, extract the geometric center in the projection, which can be achieved by using geometric calculation methods such as Epnp to improve efficiency, or by calculating or deep learning for each pixel in the entire projection image). Even in the case of partial overlap, the sphere remains identifiable due to the characteristics of the material. The rotation center point combined with the scale information provided by the diameter of the sphere is used as a normalized reference, so that the coordinate information obtained from the projection image has anti-scaling and anti-translation properties. e) Pose estimation: Use pre-trained deep learning algorithms and / or geometric algorithms such as Epnp to calculate the pose information (i.e., posture information and translation information) of the X-ray camera based on the extracted features. f) Application: The calculated pose information is used in medical applications such as surgical navigation and radiotherapy positioning to improve surgical accuracy and patient safety. It can also be applied to industrial non-destructive testing, CT geometric calibration, etc.

[0095] The above design fully considers the characteristics of X-ray imaging and provides a high-precision and robust spatial reference system for medical and other industrial and scientific research applications. Through carefully selected materials and structural design, the embodiments of the present invention significantly improve the recognition accuracy and reliability under various X-ray imaging conditions, providing an important tool for accurate medical diagnosis and industrial applications.

[0096] The embodiment of the present invention also provides a three-dimensional reconstruction method based on the above reference mark system, which can be executed by an electronic device with data processing capability. Fig.11 The flowchart of a three-dimensional reconstruction method shown in FIG. 1 mainly includes the following steps S1110 to S1130:

[0097] Step S1110 , obtaining a target projection image captured by a target imaging device for a scene to be tested where a reference marker system is placed.

[0098] The above-mentioned scene to be measured includes an object to be imaged, and the object to be imaged may be a human body or other objects with three-dimensional requirements. The target projection image may be a two-dimensional projection image under natural light or a two-dimensional projection image under X-rays.

[0099] Step S1120, estimating the position and posture of the target imaging device according to the target projection image to obtain the position and posture information of the target imaging device.

[0100] Step S1130, performing three-dimensional reconstruction of the scene to be measured on the target projection image according to the position and posture information of the target imaging device to obtain a three-dimensional reconstruction result.

[0101] Optionally, the marking component has a rotation center for providing a projection determination point; based on this, the step S1120 may include the following sub-steps 1121 to 1123:

[0102] Sub-step 1121, extracting features from the target projection image, obtaining original projection position information corresponding to each marker and the rotation center, and projection size information of a reference marker; wherein the reference marker is one of the markers.

[0103] Sub-step 1122, based on the original projection position information of the rotation center and the projection size information and true size information of the reference marker, coordinate system normalization and size normalization are performed on the original projection position information of each marker to obtain the target projection position information of each marker in the normalized projection coordinate system; wherein the coordinate origin of the normalized projection coordinate system is the rotation center.

[0104] Sub-step 1123, based on the target projection position information of each marker, the pose information of the target imaging device is calculated by a preset pose estimation algorithm; wherein the pose estimation algorithm includes a deep learning algorithm and / or a geometric algorithm.

[0105] In 3D reconstruction, the position and posture estimation of the imaging device includes the calculation of position change (i.e., translation vector) and rotation posture (the rotation posture can be obtained and annotated for each monocular image). Specifically, the rotation posture is first calculated using a deep learning algorithm and / or a geometric algorithm (the coordinate information after coordinate system normalization and size normalization is required), and then the translation vector is calculated based on the obtained rotation posture (only the coordinate information after size normalization is used).

[0106] The basic principle of pose estimation is to use the position information of the marker in the two-dimensional projection image to infer the position and pose in the three-dimensional space. The embodiments of the present invention provide a variety of methods to achieve this goal:

[0107] 1. Deep learning method: a) Extract features from the 2D projection images through a convolutional neural network (CNN) to accurately locate the 2D coordinates of the markers. b) Use a pre-trained multi-layer perceptron (MLP) network to map these 2D coordinates to position and pose parameters in 3D space. c) Optimize the network by minimizing the error between the predicted 3D parameters and the true parameters.

[0108] 2. Geometric calculation methods (such as EPnP algorithm): a) Extract the 2D coordinates of the marker from the projection image. b) Establish the correspondence between the 2D projection image coordinates and the 3D world coordinates. c) Use the EPnP algorithm to solve the perspective-n-point (PnP) problem and calculate the position and attitude of the imaging device. d) Improve the estimation accuracy through iterative optimization.

[0109] 3. Hybrid approach of deep learning and geometric computation: a) Use a deep learning network to extract and refine the 2D coordinates of the markers. b) Input these high-precision 2D coordinates into a geometric algorithm (such as EPnP) for pose estimation. c) Use a deep learning network to fine-tune and optimize the pose estimation results of the geometric algorithm. d) Through end-to-end training, the deep learning network is able to learn geometric constraints and improve overall accuracy. This hybrid approach combines the adaptability of deep learning with the precision of geometric algorithms, and can maintain high accuracy under various imaging conditions while providing interpretability and robustness.

[0110] For the calculation of rotational posture under the deep learning model: first use the target features of the marker to identify the marker, mark it as A, B, C, D after identification, and record the two-dimensional projection coordinates of the center position of the marker. These two-dimensional projection coordinates are normalized by the coordinate system and the size and then input into the deep learning model. The output of the deep learning model is the rotation posture of the reference marker system around its own rotation center (represented by rotation matrix, Euler angle or quaternion), which is then converted into the rotation posture of the imaging device through an equivalent rotation.

[0111] The above process mainly includes the following steps:

[0112] A) Size normalization: First, use the projection size of the marker in the two-dimensional projection image to normalize its position. The purpose of this step is to eliminate the difference in projection size caused by distance changes, so that subsequent calculations are more accurate. Taking the marker as a sphere as an example, the specific implementation method is as follows: the actual size of the sphere (i.e., the true diameter) is known, and the projection diameter of the specified sphere in the two-dimensional projection image is measured; 3. Use the projection diameter as a ruler and the projection of the center of the sphere at the center of rotation in the two-dimensional projection image as the coordinate origin to normalize the coordinates of each sphere on the two-dimensional image plane.

[0113] B) Target feature processing: The target features of the markers (such as color information) are used to distinguish and identify different markers. This is crucial for correctly matching 3D world coordinates and 2D image coordinates. Taking the color of a sphere as an example, the specific implementation method is as follows: perform color analysis on the 2D projection image to identify the color of each sphere; match the identified color with the pre-defined color identifier.

[0114] For the EPnP algorithm, taking the marker as a sphere as an example, the pose estimation process can be as follows:

[0115] 1. Input: a) 3D point set: the position of the sphere in the world coordinate system (the reference coordinate system preset for the reference marker system) with the rotation center as the origin (that is, the three-dimensional coordinate information of each sphere in the world coordinate system under the reference posture, which is known in advance). b) 2D point set: the position of the normalized sphere on the two-dimensional image plane. c) Intrinsic parameter matrix of the imaging device: describes the internal characteristics of the imaging device (such as focal length, principal point, etc.).

[0116] 2. EPnP algorithm steps: a) Select 4 control points to represent all 3D points. b) Represent the 3D points as the centroid coordinates of these 4 control points. c) Establish a linear equation system (related to the intrinsic parameter matrix of the imaging device) to describe the relationship between the 3D points and their 2D projections. d) Solve this linear equation system to obtain the positions of the control points in the imaging device coordinate system. e) Recover the rotation and translation of the imaging device from the positions of the control points.

[0117] 3. Output: The rotation matrix R and translation vector t of the imaging device relative to the world coordinate system. The rotation matrix R describes the direction of the imaging device; the translation vector t describes the position of the imaging device. The output of the EPnP algorithm is the precise posture of the imaging device.

[0118] Optionally, taking the marker as a sphere as an example, the calculation method of the translation vector between two two-dimensional projection images can be as follows:

[0119] 1. Normalize the size of the two 2D projection images: Use the diameter of the reference sphere as a ruler to unify the scales of the two 2D projection images.

[0120] 2. Calculate the two-dimensional coordinate change of the rotation center in the normalized image: suppose the normalized coordinates of the rotation center in the first image are (x1, y1), and the normalized coordinates of the rotation center in the second image are (x2, y2), then the coordinate change is: Δx = x2-x1, Δy = y2-y1.

[0121] 3. Calculate the translation vector based on the posture change of the imaging device: Assume that the rotation matrix between two shots is R, and the position of the rotation center in the world coordinate system is P (X, Y, Z). The translation vector T can be solved by the following equation: R×P+T=P+(Δx, Δy, Δz); where Δz can be estimated by other information (such as focal length change) or assumed to be 0 in some application scenarios.

[0122] 4. Solve the translation vector: T = (Δx, Δy, Δz) - (RI) × P, where I is the 3 × 3 identity matrix.

[0123] The embodiments of the present invention not only significantly improve the accuracy of reconstructing three-dimensional structures from single or sparse angle images, but also provide X-ray imaging with functions similar to ArUco marking in natural light images. This is of great value for medical surgery, CT geometry calibration, industrial inspection and augmented reality applications that require precise spatial positioning. In addition, since the system only requires a small number of projections for precise positioning and reconstruction in X-ray imaging, it provides the possibility of developing low-cost, portable 3D imaging devices based on existing X-ray systems, which can be quickly popularized in clinical practice, especially in areas with limited medical resources.

[0124] like Fig.12 As shown, an electronic device 1200 provided by an embodiment of the present invention includes: a processor 1201, a memory 1202 and a bus, the memory 1202 stores a computer program that can be run on the processor 1201, when the electronic device 1200 is running, the processor 1201 and the memory 1202 communicate through the bus, and the processor 1201 executes the computer program to implement the above-mentioned three-dimensional reconstruction method.

[0125] Specifically, the above-mentioned memory 1202 and processor 1201 can be general-purpose memory and processor, which are not specifically limited here.

[0126] In one possible implementation, the electronic device 1200 includes an input terminal, a host, and a display device; wherein the input terminal is used to record a projection image captured by an imaging device, such as a natural light image obtained by natural light imaging or an X-ray image obtained by X-ray imaging; the host is used to execute the above-mentioned computer program to realize three-dimensional reconstruction of the object to be imaged, and output the three-dimensional reconstructed image to the display device; the display device can be a display screen, an AR (augmented reality) device, a VR (virtual reality) device, etc., for displaying a three-dimensional reconstructed image.

[0127] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the three-dimensional reconstruction method in the above method embodiment is executed. The computer-readable storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disk, etc., various media that can store program codes.

[0128] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0129] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0130] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reference marking system, characterized in that: The reference marking system includes a marking assembly; The marking assembly includes at least four non-coplanar markers with different target features, and the target features of the markers have projection stability; The marking component has full-viewing projection specificity, and the two-dimensional projection image of the marking component corresponds one-to-one with its three-dimensional spatial posture.

2. The reference marking system according to claim 1, characterized in that The target features include one or more of relative size, color, grayscale and surface texture.

3. The reference marking system according to claim 1, characterized in that The marking component has a rotation center for providing a projection determination point, the rotation center is used as the coordinate origin of a normalized projection coordinate system, and the real size of the marker is used to provide normalization parameters for the normalized projection coordinate system.

4. The reference marking system according to claim 3, characterized in that The center of rotation includes a sphere or intersecting line segments.

5. The reference marking system according to claim 1, characterized in that The reference marking system is applied to X-ray imaging, and the marking assembly is a non-coplanar rigid structure composed of at least four spheres of different sizes, and the spheres are made of a material that is X-ray-developable but has a certain degree of transmittance.

6. The reference marking system according to claim 1, characterized in that The reference marking system is applied to natural light imaging. The marking component is a non-coplanar rigid structure composed of at least four spheres of different colors. Each of the spheres is fixed by a preset material. The transparency of the preset material reaches a preset transparency threshold and the refractive distortion rate is less than a preset distortion rate threshold.

7. The reference marking system according to any one of claims 1 to 6, characterized in that There are multiple marking components.

8. A three-dimensional reconstruction method based on the reference mark system according to any one of claims 1 to 7, characterized in that: include: Acquire a target projection image captured by a target imaging device for a scene to be measured where the reference mark system is placed; Estimating the position and posture of the target imaging device according to the target projection image to obtain the position and posture information of the target imaging device; The target projection image is subjected to three-dimensional reconstruction of the object to be imaged in the scene to be measured according to the position and posture information of the target imaging device to obtain a three-dimensional reconstruction result.

9. The three-dimensional reconstruction method according to claim 8, characterized in that: The marking component has a rotation center for providing a projection determination point; the pose estimation of the target imaging device according to the target projection image to obtain the pose information of the target imaging device includes: Extracting features from the target projection image to obtain original projection position information corresponding to each of the markers and the rotation center, and projection size information of a reference marker; wherein the reference marker is one of the markers; According to the original projection position information of the rotation center and the projection size information and true size information of the reference marker, coordinate system normalization and size normalization are performed on the original projection position information of each marker to obtain the target projection position information of each marker in the normalized projection coordinate system; wherein the coordinate origin of the normalized projection coordinate system is the rotation center; According to the target projection position information of each of the markers, the pose information of the target imaging device is calculated by a preset pose estimation algorithm; wherein the pose estimation algorithm includes a deep learning algorithm and / or a geometric algorithm.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the three-dimensional reconstruction method according to claim 8 or 9 is implemented.

Images