Single-target three-dimensional reconstruction method, device and equipment based on multi-view X-ray images

Abstract

The application provides a single-target three-dimensional reconstruction method, device and equipment based on multi-view X-ray images, comprising: for any X-ray image in multiple X-ray images, detecting the X-ray image by using a target detection model to determine the position coordinates and category label of the detection box of the X-ray image. Input the X-ray image and the position coordinates of the detection box into a segmentation model to obtain a mask image of the target to be detected in the detection box. Based on multiple mask images and multiple category labels corresponding to the multiple X-ray images, determine multiple target mask images belonging to the same target to be detected. Input the multiple target mask images belonging to the same target to be detected into a three-dimensional reconstruction model to obtain multiple three-dimensional coordinate points of the target to be detected. Compare the model composed of the multiple three-dimensional coordinate points with multiple preset three-dimensional model templates to determine a target three-dimensional model template matched with the model. Determine a target three-dimensional model of the target to be detected based on the model and the target three-dimensional model template.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a single-target three-dimensional reconstruction method, device and equipment based on multi-view X-ray images. BACKGROUND

[0002] Three-dimensional reconstruction is a core task in the field of computer vision, and the goal is to recover the three-dimensional structure of an object or scene based on two-dimensional images. There are currently two main methods for three-dimensional reconstruction: one is a geometry-based method: relying on the multi-view geometric information of the camera (such as camera internal parameters: focal length, principal point, etc.; external parameters: position, attitude, etc.), the three-dimensional structure is determined through projection geometry. The other is a deep learning-based method: with the development of neural networks, neural network models such as Neural Radiance Fields (NeRF) can be used to learn the implicit representation of images and generate three-dimensional structures from multi-view images.

[0003] In the field of industrial detection or security inspection, X-ray is usually used to detect objects to form X-ray images. The object in the X-ray image will have high perspective superposition and noise, and the target under multiple angles is difficult to accurately match, so the demand for single-target three-dimensional reconstruction of multi-view X-ray images is increasing. However, X-ray images lack geometric parameters of imaging equipment, and cannot use projection geometry methods for three-dimensional reconstruction of X-ray images. Since X-ray is a penetrating imaging, the projection shape of the same object under different angles differs greatly, and if a neural network model suitable for traditional camera imaging is directly used for three-dimensional reconstruction of multi-view X-ray images, the accuracy is low. SUMMARY

[0004] Therefore, the present application aims to provide a single-target three-dimensional reconstruction method, device and equipment based on multi-view X-ray images, in order to improve the accuracy of three-dimensional reconstruction of X-ray images.

[0005] In a first aspect, the present application provides a single-target three-dimensional reconstruction method based on multi-view X-ray images, which comprises:

[0006] For any X-ray image in the plurality of X-ray images, a target detection model is used to detect the X-ray image to determine the position coordinates and class labels of at least one detection box of the X-ray image;

[0007] The X-ray image and the position coordinates of the detection box are input into a segmentation model to obtain a mask image of the target to be detected in the detection box;

[0008] Based on the plurality of mask images corresponding to the plurality of X-ray images and the plurality of class labels, a plurality of target mask images belonging to the same target to be detected are determined.

[0009] inputting multiple target mask images belonging to the same to-be-detected target into a three-dimensional reconstruction model to obtain multiple three-dimensional coordinate points of the to-be-detected target;

[0010] comparing a model composed of the multiple three-dimensional coordinate points with multiple preset three-dimensional model templates to determine a target three-dimensional model template matched with the model;

[0011] determining a target three-dimensional model of the to-be-detected target based on the model and the target three-dimensional model template.

[0012] In a possible implementation, the determining of the multiple target mask images belonging to the same to-be-detected target based on the multiple mask images corresponding to the multiple X-ray images respectively and the multiple category labels comprises:

[0013] determining a first mask image and a second mask image belonging to the same category label based on the multiple category labels corresponding to the multiple X-ray images;

[0014] determining a first similarity between the first mask image and the second mask image;

[0015] when the first similarity is greater than a preset value, determining that the first mask image and the second mask image are target mask images belonging to the same to-be-detected target.

[0016] In a possible implementation, the determining of the first similarity between the first mask image and the second mask image comprises:

[0017] obtaining a first feature vector corresponding to the first mask image and a second feature vector corresponding to the second mask image, wherein the first feature vector is obtained by detecting an X-ray image corresponding to the first mask image by using the target detection model;

[0018] calculating a cosine similarity between the first feature vector and the second feature vector.

[0019] In a possible implementation, the determining process of the multiple preset three-dimensional model templates comprises:

[0020] performing three-dimensional modeling on the to-be-detected target to obtain multiple preset three-dimensional models;

[0021] projecting the preset three-dimensional models to a three-dimensional voxel grid to obtain the preset three-dimensional model templates.

[0022] In a possible implementation, the comparing of the model composed of the multiple three-dimensional coordinate points with the multiple preset three-dimensional model templates to determine a target three-dimensional model template matched with the model comprises:

[0023] projecting the plurality of three-dimensional coordinate points to the three-dimensional voxel grid to obtain the model;

[0024] calculating an intersection-over-union and a chamfer distance between the model and the preset three-dimensional model template;

[0025] determining a second similarity between the model and the preset three-dimensional model template based on the intersection-over-union and the chamfer distance;

[0026] determining the target three-dimensional model template based on a maximum value of a plurality of the second similarities between the model and a plurality of the preset three-dimensional model templates.

[0027] In a possible implementation, the determining, based on the model and the target three-dimensional model template, of the target three-dimensional model of the to-be-detected target comprises:

[0028] performing weighted fusion of the model and the target three-dimensional model template to obtain the target three-dimensional model.

[0029] In a possible implementation, the detecting, by the target detection model, of the X-ray image to determine the position coordinates and the category label of at least one detection frame of the X-ray image comprises:

[0030] inputting the X-ray image into the target detection model to output a position offset corresponding to at least one detection frame and a category probability;

[0031] determining the position coordinates of the detection frame based on the position offset, and determining the category label corresponding to the detection frame based on the category probability.

[0032] In a possible implementation, the three-dimensional reconstruction model comprises a dense unconstrained stereo three-dimensional reconstruction DUSt3R model.

[0033] In a second aspect, the present application provides a single-target three-dimensional reconstruction device based on multi-view X-ray images, the device comprising:

[0034] a target detection unit configured to, for any X-ray image in a plurality of X-ray images, detect the X-ray image by using a target detection model to determine the position coordinates and the category label of at least one detection frame of the X-ray image;

[0035] a segmentation unit configured to input the X-ray image and the position coordinates of the detection frame into a segmentation model to obtain a mask image of a to-be-detected target in the detection frame;

[0036] A classification unit is used to determine multiple target mask images belonging to the same target to be detected based on multiple mask images corresponding to multiple X-ray images and multiple category labels;

[0037] The three-dimensional reconstruction unit is used to input multiple target mask images belonging to the same target to the three-dimensional reconstruction model to obtain multiple three-dimensional coordinate points of the target to be detected;

[0038] A matching unit is used to compare the model composed of multiple three-dimensional coordinate points with multiple preset three-dimensional model templates to determine the target three-dimensional model template that matches the model.

[0039] The determining unit is used to determine the target 3D model of the target to be detected based on the model and the target 3D model template.

[0040] Thirdly, this application provides an electronic device, the device comprising: a memory and a processor;

[0041] The memory is used to store the relevant program code;

[0042] The processor is used to call the program code to execute the single-target three-dimensional reconstruction method based on multi-view X-ray images as described in any of the implementations of the first aspect above.

[0043] Fourthly, this application provides a computer-readable storage medium for storing a computer program for executing the single-target three-dimensional reconstruction method based on multi-view X-ray images as described in any implementation of the first aspect.

[0044] Fifthly, this application provides a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, it implements the single-target three-dimensional reconstruction method based on multi-view X-ray images as described in any of the implementations of the first aspect.

[0045] In the above implementation of the present application, for any X-ray image in the plurality of X-ray images, the target detection model is used to detect the X-ray image to determine the position coordinates and the class label of at least one detection box of the X-ray image. The X-ray image and the position coordinates of the detection box are input into the segmentation model to obtain a mask image of the target to be detected in the detection box. Based on the plurality of mask images respectively corresponding to the plurality of X-ray images and the plurality of class labels, a plurality of target mask images belonging to the same target to be detected are determined. The plurality of target mask images belonging to the same target to be detected are input into the three-dimensional reconstruction model to obtain a plurality of three-dimensional coordinate points of the target to be detected. The model composed of the plurality of three-dimensional coordinate points is compared with a plurality of preset three-dimensional model templates to determine a target three-dimensional model template matching the model. Based on the model and the target three-dimensional model template, a target three-dimensional model of the target to be detected is determined. Through the method provided in the present application, the mask images belonging to the same object can be determined based on the plurality of X-ray images, and the three-dimensional model can be directly reconstructed based on the mask images. The reconstructed three-dimensional model is fused with the three-dimensional model template determined in advance to obtain the target three-dimensional model of the target to be detected, thereby improving the accuracy of the reconstructed three-dimensional model. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments provided in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0047] Figure 1 A flowchart of a single-target three-dimensional reconstruction method based on multi-view X-ray images provided by an embodiment of the present application.

[0048] Figure 2 A schematic diagram of a single-target three-dimensional reconstruction device based on multi-view X-ray images provided by an embodiment of the present application.

[0049] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. The described embodiments are only exemplary embodiments of the present application, and are not all embodiments. Those skilled in the art can obtain other embodiments without creative labor based on the embodiments of the present application, and these embodiments are also within the protection scope of the present application.

[0051] The target of three-dimensional reconstruction is to recover the three-dimensional structure of an object or a scene based on two-dimensional images. There are mainly two types of three-dimensional reconstruction methods at present: one type is a geometry-based method: relying on the multi-view geometric information of the camera (such as camera internal parameters: focal length, principal point, etc.; external parameters: position, pose, etc.), the three-dimensional structure is determined through the projection geometry principle. Another type is a deep learning-based method: with the development of neural networks, neural network models such as NeRF can be used to learn the implicit representation of images and generate three-dimensional structures from multi-view images.

[0052] In the field of industrial detection or security inspection, X-ray is usually used to detect objects to form X-ray images. There are high perspective superposition and noise in the X-ray images, and it is difficult to accurately match the target under multiple angles, so the demand for single-target three-dimensional reconstruction of multi-view X-ray images is increasing. However, X-ray images lack the geometric parameters of the imaging device, and cannot use the projection geometry method for three-dimensional reconstruction of X-ray images. Since X-ray is a penetrating imaging, the projection shape of the same object under different angles is quite different, and if a neural network model suitable for traditional camera imaging is directly used for three-dimensional reconstruction of multi-view X-ray images, the accuracy is low.

[0053] Therefore, the embodiment of the present application provides a single-target three-dimensional reconstruction method based on multi-view X-ray images, so as to improve the accuracy of three-dimensional reconstruction of X-ray images. In specific implementation, for any X-ray image in the plurality of X-ray images, a target detection model is used to detect the X-ray image to determine the position coordinates and class labels of at least one detection box of the X-ray image. The X-ray image and the position coordinates of the detection box are input into a segmentation model to obtain a mask image of a target to be detected in the detection box. Based on a plurality of mask images corresponding to the plurality of X-ray images and a plurality of class labels, a plurality of target mask images belonging to the same target to be detected are determined. The plurality of target mask images belonging to the same target to be detected are input into a three-dimensional reconstruction model to obtain a plurality of three-dimensional coordinate points of the target to be detected. The model composed of the plurality of three-dimensional coordinate points is compared with a plurality of preset three-dimensional model templates to determine a target three-dimensional model template matched with the model. Based on the model and the target three-dimensional model template, a target three-dimensional model of the target to be detected is determined. Through the method provided in the embodiment of the present application, the mask images belonging to the same object can be determined based on the plurality of X-ray images, and the three-dimensional model can be directly reconstructed based on the mask images. The reconstructed three-dimensional model is fused with the pre-determined three-dimensional model template to obtain the target three-dimensional model of the target to be detected, so as to improve the accuracy of the reconstructed three-dimensional model.

[0054] In order to facilitate understanding of the technical solutions provided in the embodiments of the present application, specific descriptions will be made below in conjunction with the drawings in the embodiments.

[0055] Reference is made to Figure 1As shown, a flowchart of a single-target three-dimensional reconstruction method based on multi-view X-ray images is provided in the embodiments of the present application.

[0056] Optionally, the method can be performed by an image processing device. The method can include the following steps:

[0057] S101: For any X-ray image in the plurality of X-ray images, detecting the X-ray image by using a target detection model to determine the position coordinates and class label of at least one detection box of the X-ray image.

[0058] The X-ray image can represent an image obtained by taking an object by using X-rays. In the embodiments of the present application, the target detection model can be used to detect the X-ray image to obtain a target to be detected in the X-ray image. The target detection model can be a pre-trained model and can detect targets of a plurality of preset classes in the X-ray image. For example, the target detection model can be a Faster Region-based Convolutional Neural Network (Faster R-CNN).

[0059] In a specific implementation, after the X-ray image is input into the target detection model, feature extraction can be performed on the X-ray image to obtain at least one feature vector. The number of feature vectors corresponds to the number of targets to be detected in the X-ray image. For each feature vector, the feature vector is input into two parallel fully connected layers respectively, and the position offset of the detection box and the class probability are output. That is, when there are multiple targets to be detected in the X-ray image, multiple feature vectors can be obtained, and the position offset and the class probability corresponding to each detection box are output.

[0060] One of the fully connected layers of the target detection model is a classification branch, which is used to output the class probability of the detection box. The class probability includes a plurality of probabilities of the detection box belonging to a plurality of preset classes (including background) in the training data. Then, the class label corresponding to the detection box can be determined based on the class probability. For example, the class corresponding to the maximum probability value in the class probability can be determined as the class label of the detection box.

[0061] The other fully connected layer is a regression branch, which is used to output the position offset of each detection box. The position offset represents the position offset between the detection box and an anchor box. The anchor box is a preset box of the target detection model, and the position coordinates of the anchor box are known. Thus, the position coordinates of the detection box can be determined according to the position offset and the position coordinates of the anchor box.

[0062] S102: Input the X-ray image and the position coordinates of the detection box into a segmentation model to obtain a mask image of the target to be detected in the detection box.

[0063] After obtaining the position coordinates of one or more in the X-ray image, the X-ray image and the position coordinates of each detection box can be input into a segmentation model, which can output a mask image of the target to be detected in the detection box. For example, the segmentation model can be a segment anything model (SAM). Wherein the mask image represents a binary image of the same size as the detection box, and the region of the target to be detected in the mask image is pixel 1, and the pixel of the background region is 0.

[0064] S103: Based on the plurality of mask images respectively corresponding to the plurality of X-ray images and the plurality of class labels, a plurality of target mask images belonging to the same target to be detected are determined.

[0065] After performing the above steps S101 and S102 for each X-ray image, the class label and mask image corresponding to each X-ray image can be obtained. Since multiple targets to be detected can be included in multiple X-ray images, multiple target mask images belonging to the same target to be detected can be determined based on the multiple mask images and the class labels.

[0066] In one possible implementation, a first mask image and a second mask image belonging to the same class label can be determined based on the plurality of class labels. That is, a plurality of mask images belonging to the same class label can be determined first, and any two mask images belonging to the same class label are represented as the first mask image and the second mask image.

[0067] Then a first similarity between the first mask image and the second mask image is determined. The first similarity is compared with a preset value to measure the similarity of the first mask image and the second mask image. Wherein the preset value represents the minimum similarity that the two targets to be detected corresponding to the two mask images are the same target to be detected. The specific value of the preset value is not limited in the embodiments of the present application. For example, it can be set to 0.5. When the first similarity is greater than the preset value, it can be determined that the first mask image and the second mask image are target mask images belonging to the same target to be detected. Based on the above method, the first similarity of any two mask images is calculated, and then a plurality of target mask images belonging to the same target to be detected can be determined. It should be noted that based on the above method, multiple targets to be detected can be determined, and each target to be detected corresponds to multiple target mask images.

[0068] In a possible implementation, the first similarity between the first mask image and the second mask image can be determined in the following manner: according to the above embodiment, when the target detection model detects the X-ray image, at least one feature vector of the X-ray image can be obtained, each feature vector corresponds to a detection box, and each detection box can obtain a mask image. Then, a first feature vector corresponding to the first mask image and a second feature vector corresponding to the second mask image can be determined. A cosine similarity between the first feature vector and the second feature vector is calculated, which is taken as the first similarity between the first mask image and the second mask image.

[0069] S104: input multiple target mask images belonging to the same to-be-detected target into a three-dimensional reconstruction model to obtain multiple three-dimensional coordinate points of the to-be-detected target.

[0070] The three-dimensional reconstruction model can be a Dense and Unconstrained Stereo 3D Reconstruction (DUSt3R) model. The DUSt3R model does not need to use device parameters for shooting X-ray images, and can directly obtain multiple three-dimensional coordinate points of the to-be-detected target based on multiple target mask images.

[0071] Optionally, in order to visualize the three-dimensional structure of the to-be-detected target and facilitate subsequent analysis, the multiple three-dimensional coordinate points can be projected into a three-dimensional voxel grid. The three-dimensional voxel grid can be a cubic region including all three-dimensional coordinate points, and the cubic region is divided into multiple small cubes, i.e., voxel units, each of which corresponds to a three-dimensional coordinate. Each three-dimensional coordinate point is projected into a voxel unit corresponding to the three-dimensional voxel grid. When a three-dimensional coordinate point exists in the voxel unit, the value of the voxel unit is marked as 1. The values of the remaining voxel units without three-dimensional coordinate points are marked as 0. Then, a three-dimensional array corresponding to the three-dimensional voxel grid can be obtained as a preliminary model of the to-be-detected target.

[0072] S105: compare the model composed of the multiple three-dimensional coordinate points with multiple preset three-dimensional model templates to determine a target three-dimensional model template matched with the model.

[0073] The multiple preset three-dimensional model templates are determined in advance. Three-dimensional modeling can be performed on each to-be-detected target in advance to obtain multiple preset three-dimensional models corresponding to the to-be-detected target. For example, three-dimensional modeling can be performed on the to-be-detected target from different observation angles based on a Computer Aided Design (CAD), CT scanning, or the like to obtain multiple preset three-dimensional models of the to-be-detected target.

[0074] Similarly, each preset three-dimensional model can be projected into a three-dimensional voxel grid, and the obtained three-dimensional array can be taken as a preset three-dimensional model template. The three-dimensional voxel grid projected by the preset three-dimensional model and the three-dimensional voxel grid projected by the three-dimensional coordinate point have the same size.

[0075] Based on this, the intersection-over-union and the chamfer distance between the model composed of the plurality of three-dimensional coordinate points and each preset three-dimensional model template are calculated. The larger the intersection-over-union is, the higher the similarity between the model and the preset three-dimensional model template is. The larger the chamfer distance is, the lower the similarity between the model and the preset three-dimensional model template is. For example, when the model composed of the plurality of three-dimensional coordinate points and the preset three-dimensional model template can be represented as a three-dimensional array, the intersection-over-union can be represented as the ratio between the intersection of the two three-dimensional arrays and the union of the two three-dimensional arrays. The intersection of the two three-dimensional arrays can be represented as the number of positions in which the two corresponding values are both 1, and the union of the two three-dimensional arrays can be represented as the number of positions in which at least one of the two corresponding values is 1. The chamfer distance between the model composed of the plurality of three-dimensional coordinate points and the preset three-dimensional model template can be represented as the chamfer distance between the three-dimensional array corresponding to the plurality of three-dimensional coordinate points and the three-dimensional array corresponding to the preset three-dimensional model template.

[0076] Then, based on the intersection-over-union and the chamfer distance, the second similarity between the model composed of the plurality of three-dimensional coordinate points and the preset three-dimensional model template is determined. For example, a first weight can be assigned to the intersection-over-union, and a second weight can be assigned to the chamfer distance. Then, a first product of the intersection-over-union and the first weight and a second product of the chamfer distance and the second weight are calculated. Then, the second similarity can be obtained according to the difference between the first product and the second product. When the intersection-over-union is mainly used to determine the second similarity, the first weight can be set to be greater than the second weight. When the chamfer distance is mainly used to determine the second similarity, the first weight can be set to be less than the second weight. In the embodiments of the present application, the first weight and the second weight can be set to be 0.5.

[0077] After obtaining the plurality of second similarities between the model and the plurality of preset three-dimensional model templates, the preset three-dimensional model template corresponding to the maximum value in the plurality of second similarities can be determined as the target three-dimensional model template, which has the highest similarity with the model.

[0078] It should be noted that the above embodiments introduce a method for determining the second similarity based on the intersection-over-union and the chamfer distance. Alternatively, one of the intersection-over-union and the chamfer distance can be taken as the second similarity, which does not affect the implementation of the embodiments of the present application.

[0079] S106: Based on the model and the target three-dimensional model template, a target three-dimensional model of the to-be-detected target is determined.

[0080] After the target three-dimensional model template is determined, the model can be fused with the target three-dimensional model template by weighting to obtain a target three-dimensional model. For example, after the first three-dimensional array corresponding to the plurality of three-dimensional coordinate points and the second three-dimensional array corresponding to the target three-dimensional model template are determined, the two values at the corresponding positions in the first three-dimensional array and the second three-dimensional array are summed by weighting to obtain a weighted three-dimensional array as the target three-dimensional model.

[0081] Based on the method provided in the above embodiment, the mask images belonging to the same object can be determined based on the plurality of X-ray images, and the three-dimensional model can be directly reconstructed based on the mask images. The reconstructed three-dimensional model is fused with the pre-determined three-dimensional model template to obtain the target three-dimensional model of the to-be-detected target, so that the accuracy of the reconstructed three-dimensional model can be improved.

[0082] Based on the above method embodiment, the embodiment of the present application further provides a single-target three-dimensional reconstruction device based on multi-view X-ray images. For details, please refer to Figure 2 As shown in the figure, it is a schematic diagram of a single-target three-dimensional reconstruction device based on multi-view X-ray images provided by the embodiment of the present application.

[0083] The device 200 comprises:

[0084] The target detection unit 201 is configured to, for any X-ray image in the plurality of X-ray images, detect the X-ray image by using a target detection model, determine the position coordinates and the category label of at least one detection frame of the X-ray image.

[0085] The segmentation unit 202 is configured to input the X-ray image and the position coordinates of the detection frame into a segmentation model to obtain a mask image of the to-be-detected target in the detection frame.

[0086] The classification unit 203 is configured to determine a plurality of target mask images belonging to the same to-be-detected target based on a plurality of mask images respectively corresponding to the plurality of X-ray images and a plurality of category labels.

[0087] The three-dimensional reconstruction unit 204 is configured to input the plurality of target mask images belonging to the same to-be-detected target into a three-dimensional reconstruction model to obtain a plurality of three-dimensional coordinate points of the to-be-detected target.

[0088] The matching unit 205 is configured to compare a model composed of the plurality of three-dimensional coordinate points with a plurality of pre-set three-dimensional model templates to determine a target three-dimensional model template matched with the model.

[0089] The determination unit 206 is configured to determine a target three-dimensional model of the to-be-detected target based on the model and the target three-dimensional model template.

[0090] In a possible implementation, the classification unit 203 is specifically configured to determine a first mask image and a second mask image belonging to a same category label based on a plurality of category labels corresponding to a plurality of X-ray images; determine a first similarity between the first mask image and the second mask image; and determine that the first mask image and the second mask image are target mask images of a same target to be detected when the first similarity is greater than a preset value.

[0091] In a possible implementation, the classification unit 203 is specifically configured to obtain a first feature vector corresponding to the first mask image and a second feature vector corresponding to the second mask image, where the first feature vector is obtained by detecting an X-ray image corresponding to the first mask image by using the target detection model; and calculate a cosine similarity between the first feature vector and the second feature vector.

[0092] In a possible implementation, the determination of the plurality of preset three-dimensional model templates includes: performing three-dimensional modeling on the target to be detected to obtain a plurality of preset three-dimensional models; and projecting the preset three-dimensional models to a three-dimensional voxel grid to obtain the preset three-dimensional model templates.

[0093] In a possible implementation, the matching unit 205 is specifically configured to project a plurality of three-dimensional coordinate points to the three-dimensional voxel grid to obtain the model; calculate an intersection-over-union ratio and a chamfer distance between the model and the preset three-dimensional model template; determine a second similarity between the model and the preset three-dimensional model template based on the intersection-over-union ratio and the chamfer distance; and determine the target three-dimensional model template based on a maximum value of a plurality of second similarities between the model and the plurality of preset three-dimensional model templates.

[0094] In a possible implementation, the determination unit 206 is specifically configured to perform weighted fusion of the model and the target three-dimensional model template to obtain the target three-dimensional model.

[0095] In a possible implementation, the target detection unit 201 is specifically configured to input the X-ray image to the target detection model to output a position offset corresponding to at least one detection frame and a category probability; determine a position coordinate of the detection frame based on the position offset, and determine a category label corresponding to the detection frame based on the category probability.

[0096] In a possible implementation, the three-dimensional reconstruction model includes a dense unconstrained stereo three-dimensional reconstruction DUSt3R model.

[0097] Based on the method embodiments and the device embodiments above, the embodiments of the present application further provide an electronic device. The following will be introduced with reference to the accompanying drawings.

[0098] Referring to Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an electronic device provided by the embodiments of the present application.

[0099] The device 300 comprises a memory 301 and a processor 302.

[0100] The memory 301 is configured to store related program codes.

[0101] The processor 302 is configured to invoke the program codes and execute the method of single-target three-dimensional reconstruction based on multi-view X-ray images as described above.

[0102] In addition, the embodiments of the present application further provide a computer readable storage medium, which is configured to store a computer program, and the computer program is configured to execute the method of single-target three-dimensional reconstruction based on multi-view X-ray images as described above.

[0103] The embodiments of the present application further provide a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the method of single-target three-dimensional reconstruction based on multi-view X-ray images as described above.

[0104] It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection with one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0105] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. The embodiments were chosen and described in order to best explain the principles of the application and the practical application, and to allow those skilled in the art to understand the application for various embodiments and with various modifications that are suited to the particular use contemplated.

[0106] It should be noted that the various embodiments described in this specification are intended to be illustrative only and that the scope of the application is therefore intended to be broad. It will be understood by those within the art that, in light of the foregoing disclosure, the present application can be implemented in a variety of ways, and that the scope of embodiments of the present application can be measured by the breadth of the claims that follow. The embodiments described and pictured herein are presented by way of example only and are not intended to limit the scope of the application. Accordingly, the actual scope of the application is set forth and limited only by the following claims.

[0107] The flow diagrams and block diagrams in the drawings are illustrations of architectures, functions, and operations that can be implemented in software, hardware, or a combination thereof. In this regard, each block in the flow diagrams and block diagrams can represent a module, segment, or portion of code that comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0108] It should be understood that, in the application, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including single item or any combination of multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0109] It should also be noted that, in the application, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0110] The steps of the method or algorithm described in conjunction with the embodiments disclosed in the application can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0111] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the application can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the embodiments shown in the application, but will conform to the widest scope consistent with the principles and novel features disclosed in the application.

Claims

1. A single object 3D reconstruction method based on multi-view X-ray images, characterized in that, The method comprises: For any X-ray image in a plurality of X-ray images, detecting the X-ray image by using a target detection model to determine the position coordinates and category labels of at least one detection box of the X-ray image; inputting the X-ray image and the position coordinates of the detection box into a segmentation model to obtain a mask image of a target to be detected in the detection box; based on a plurality of mask images corresponding to the plurality of X-ray images and a plurality of category labels, determining a plurality of target mask images belonging to the same target to be detected; inputting the plurality of target mask images belonging to the same target to be detected into a three-dimensional reconstruction model to obtain a plurality of three-dimensional coordinate points of the target to be detected; comparing the model composed of the plurality of three-dimensional coordinate points with a plurality of preset three-dimensional model templates to determine a target three-dimensional model template matching the model; based on the model and the target three-dimensional model template, determining a target three-dimensional model of the target to be detected.

2. The method of claim 1, wherein, The method comprises: based on a plurality of category labels corresponding to a plurality of X-ray images, determining a first mask image and a second mask image belonging to the same category label; determining a first similarity between the first mask image and the second mask image; when the first similarity is greater than a preset value, determining that the first mask image and the second mask image are target mask images belonging to the same target to be detected.

3. The method of claim 2, wherein, The method comprises: obtaining a first feature vector corresponding to the first mask image and a second feature vector corresponding to the second mask image, wherein the first feature vector is obtained by detecting the X-ray image corresponding to the first mask image by using the target detection model; calculating the cosine similarity between the first feature vector and the second feature vector.

4. The method of claim 1, wherein, The determination process of the plurality of preset three-dimensional model templates comprises: three-dimensional modeling the target to be detected to obtain a plurality of preset three-dimensional models; projecting the preset three-dimensional models onto a three-dimensional voxel grid to obtain the preset three-dimensional model templates.

5. The method of claim 4, wherein, The method comprises: projecting the plurality of three-dimensional coordinate points onto the three-dimensional voxel grid to obtain the model; calculating the intersection over union and the chamfer distance between the model and the preset three-dimensional model template; based on the intersection over union and the chamfer distance, determining a second similarity between the model and the preset three-dimensional model template; based on the maximum value of the plurality of second similarities between the model and the plurality of preset three-dimensional model templates, determining the target three-dimensional model template.

6. The method of claim 1, wherein, The method comprises: weighting and fusing the model and the target three-dimensional model template to obtain the target three-dimensional model.

7. The method of claim 1, wherein, The target detection model is used to detect the X-ray image, and position coordinates and a category label of at least one detection frame of the X-ray image are determined, including: The X-ray image is input into the target detection model, and a position offset corresponding to at least one detection frame and a category probability are output; The position coordinates of the detection frame are determined based on the position offset, and the category label corresponding to the detection frame is determined based on the category probability.

8. The method of claim 1, wherein, The three-dimensional reconstruction model includes a dense unconstrained stereo three-dimensional reconstruction DUSt3R model.

9. An apparatus for single object three-dimensional reconstruction based on multi-view X-ray images, characterized by comprising: The device includes: A target detection unit is configured to use a target detection model to detect any X-ray image in a plurality of X-ray images, determine position coordinates and a category label of at least one detection frame of the X-ray image; A segmentation unit is configured to input the X-ray image and the position coordinates of the detection frame into a segmentation model, and obtain a mask image of a target to be detected in the detection frame; A classification unit is configured to determine a plurality of target mask images belonging to the same target to be detected based on a plurality of mask images corresponding to the plurality of X-ray images respectively and a plurality of category labels; A three-dimensional reconstruction unit is configured to input the plurality of target mask images belonging to the same target to be detected into a three-dimensional reconstruction model, and obtain a plurality of three-dimensional coordinate points of the target to be detected; A matching unit is configured to compare a model composed of the plurality of three-dimensional coordinate points with a plurality of preset three-dimensional model templates, and determine a target three-dimensional model template matched with the model; A determination unit is configured to determine a target three-dimensional model of the target to be detected based on the model and the target three-dimensional model template.

10. An electronic device, comprising: The device includes a memory and a processor; The memory is configured to store related program codes; The processor is configured to call the program codes and execute the single-target three-dimensional reconstruction method based on a plurality of X-ray images according to any one of claims 1 to 8.

Images