Single-target three-dimensional reconstruction method, device and equipment based on multi-view X-ray image

Abstract

The invention provides a single-target three-dimensional reconstruction method, device and equipment based on a multi-view X-ray image, and the method comprises the steps: carrying out the detection of an X-ray image through a target detection model for any one of a plurality of X-ray images, and determining the position coordinates and category labels of a detection frame of the X-ray image. And inputting the X-ray image and the position coordinates of the detection frame into the segmentation model, and obtaining a mask image of the to-be-detected target in the detection frame. And determining a plurality of target mask images belonging to the same to-be-detected target based on the plurality of mask images corresponding to the plurality of X-ray images and the plurality of category labels. And inputting a plurality of target mask images belonging to the same to-be-detected target into the three-dimensional reconstruction model, and obtaining a plurality of three-dimensional coordinate points of the to-be-detected target. And comparing a model formed by the plurality of three-dimensional coordinate points with a plurality of preset three-dimensional model templates, and determining a target three-dimensional model template matched with the model. And determining a target three-dimensional model of the to-be-detected target based on the model and the target three-dimensional model template.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a single-target three-dimensional reconstruction method, device and equipment based on multi-view X-ray images. BACKGROUND

[0002] Three-dimensional reconstruction is a core task in the field of computer vision, and the goal is to recover the three-dimensional structure of an object or scene based on two-dimensional images. There are currently two main methods for three-dimensional reconstruction: one is a geometry-based method that relies on the multi-view geometric information of the camera (such as camera intrinsic parameters: focal length, principal point, etc.; extrinsic parameters: position, pose, etc.) to determine the three-dimensional structure through projection geometry. The other is a deep learning-based method that can use neural network models such as Neural Radiance Fields (NeRF) to learn the implicit representation of images and generate three-dimensional structures from multi-view images.

[0003] In the field of industrial detection or security inspection, X-ray is often used to detect objects and form X-ray images. The object in the X-ray image may have high perspective superposition and noise, and the target under multiple angles is difficult to accurately match, so there is an increasing demand for single-target three-dimensional reconstruction of multi-view X-ray images. However, X-ray images lack geometric parameters of imaging devices, and cannot be used for three-dimensional reconstruction of X-ray images using projection geometry. Since X-ray is a penetrating imaging, the projected shape of the same object under different angles differs greatly, and if a neural network model suitable for traditional camera imaging is directly used to perform three-dimensional reconstruction on multi-view X-ray images, the accuracy is low. SUMMARY

[0004] Therefore, the present application aims to provide a single-target three-dimensional reconstruction method, device and equipment based on multi-view X-ray images to improve the accuracy of three-dimensional reconstruction of X-ray images.

[0005] In a first aspect, the present application provides a single-target three-dimensional reconstruction method based on multi-view X-ray images, which comprises: For any X-ray image in the plurality of X-ray images, a target detection model is used to detect the X-ray image to determine the position coordinates and class labels of at least one detection box of the X-ray image; The X-ray image and the position coordinates of the detection box are input into a segmentation model to obtain a mask image of the target to be detected in the detection box; Based on the plurality of mask images corresponding to the plurality of X-ray images and the plurality of class labels, a plurality of target mask images belonging to the same target to be detected are determined. inputting multiple target mask images belonging to the same to-be-detected target into a three-dimensional reconstruction model to obtain multiple three-dimensional coordinate points of the to-be-detected target; comparing the model composed of the multiple three-dimensional coordinate points with multiple preset three-dimensional model templates to determine a target three-dimensional model template matched with the model; determining a target three-dimensional model of the to-be-detected target based on the model and the target three-dimensional model template.

[0006] In a possible implementation, the determining of the multiple target mask images belonging to the same to-be-detected target based on the multiple mask images corresponding to the multiple X-ray images respectively and the multiple category labels comprises: determining a first mask image and a second mask image belonging to the same category label based on the multiple category labels corresponding to the multiple X-ray images; determining a first similarity between the first mask image and the second mask image; when the first similarity is greater than a preset value, determining that the first mask image and the second mask image are target mask images belonging to the same to-be-detected target.

[0007] In a possible implementation, the determining of the first similarity between the first mask image and the second mask image comprises: obtaining a first feature vector corresponding to the first mask image and a second feature vector corresponding to the second mask image, wherein the first feature vector is obtained by detecting an X-ray image corresponding to the first mask image by using the target detection model; calculating a cosine similarity between the first feature vector and the second feature vector.

[0008] In a possible implementation, the determining of the multiple preset three-dimensional model templates comprises: performing three-dimensional modeling on the to-be-detected target to obtain multiple preset three-dimensional models; projecting the preset three-dimensional models to a three-dimensional voxel grid to obtain the preset three-dimensional model templates.

[0009] In a possible implementation, the comparing of the model composed of the multiple three-dimensional coordinate points with the multiple preset three-dimensional model templates to determine a target three-dimensional model template matched with the model comprises: projecting the multiple three-dimensional coordinate points to the three-dimensional voxel grid to obtain the model; calculating an intersection-over-union ratio and an angle distance between the model and the preset three-dimensional model templates; determine a second similarity between the model and the preset three-dimensional model template based on the intersection-over-union ratio and the chamfer distance; determine the target three-dimensional model template based on a maximum value of a plurality of the second similarities between the model and a plurality of the preset three-dimensional model templates.

[0010] In a possible implementation, the determining the target three-dimensional model of the to-be-detected target based on the model and the target three-dimensional model template comprises: performing weighted fusion of the model and the target three-dimensional model template to obtain the target three-dimensional model.

[0011] In a possible implementation, the detecting the X-ray image by using the target detection model to determine the position coordinates and the category label of at least one detection frame of the X-ray image comprises: inputting the X-ray image into the target detection model to output a position offset corresponding to at least one detection frame and a category probability; determining the position coordinates of the detection frame based on the position offset, and determining the category label corresponding to the detection frame based on the category probability.

[0012] In a possible implementation, the three-dimensional reconstruction model comprises a dense unconstrained stereo three-dimensional reconstruction DUSt3R model.

[0013] In a second aspect, the present application provides a single-target three-dimensional reconstruction device based on multi-view X-ray images, the device comprising: a target detection unit configured to, for any X-ray image in a plurality of X-ray images, detect the X-ray image by using a target detection model to determine the position coordinates and the category label of at least one detection frame of the X-ray image; a segmentation unit configured to input the X-ray image and the position coordinates of the detection frame into a segmentation model to obtain a mask image of a to-be-detected target in the detection frame; a classification unit configured to determine a plurality of target mask images belonging to the same to-be-detected target based on a plurality of the mask images respectively corresponding to a plurality of the X-ray images and a plurality of the category labels; a three-dimensional reconstruction unit configured to input a plurality of the target mask images belonging to the same to-be-detected target into a three-dimensional reconstruction model to obtain a plurality of three-dimensional coordinate points of the to-be-detected target; a matching unit configured to compare a model composed of a plurality of the three-dimensional coordinate points with a plurality of preset three-dimensional model templates to determine a target three-dimensional model template matched with the model; a determination unit configured to determine a target three-dimensional model of the to-be-detected target based on the model and the target three-dimensional model template.

[0014] Thirdly, this application provides an electronic device, the device comprising: a memory and a processor; The memory is used to store the relevant program code; The processor is used to call the program code to execute the single-target three-dimensional reconstruction method based on multi-view X-ray images as described in any of the implementations of the first aspect above.

[0015] Fourthly, this application provides a computer-readable storage medium for storing a computer program for executing the single-target three-dimensional reconstruction method based on multi-view X-ray images as described in any implementation of the first aspect.

[0016] Fifthly, this application provides a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, it implements the single-target three-dimensional reconstruction method based on multi-view X-ray images as described in any of the implementations of the first aspect.

[0017] In the above implementation of this application, for any X-ray image among multiple X-ray images, a target detection model is used to detect the X-ray image, determining the position coordinates and category label of at least one detection box in the X-ray image. The X-ray image and the position coordinates of the detection box are input into a segmentation model to obtain a mask image of the target to be detected within the detection box. Based on multiple mask images corresponding to multiple X-ray images and multiple category labels, multiple target mask images belonging to the same target to be detected are determined. The multiple target mask images belonging to the same target to be detected are input into a 3D reconstruction model to obtain multiple 3D coordinate points of the target to be detected. The model composed of multiple 3D coordinate points is compared with multiple preset 3D model templates to determine the target 3D model template that matches the model. Based on the model and the target 3D model template, the target 3D model of the target to be detected is determined. Through the method provided by this application, mask images belonging to the same object can be determined based on multiple X-ray images, and a 3D model can be directly reconstructed based on the mask images. The reconstructed 3D model is fused with a preset 3D model template to obtain the target 3D model of the target to be detected, thereby improving the accuracy of the reconstructed 3D model. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments provided in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 A flowchart of a single-target three-dimensional reconstruction method based on multi-view X-ray images is provided for an embodiment of the present application.

[0020] Figure 2 A schematic diagram of a single-target three-dimensional reconstruction device based on multi-view X-ray images is provided for an embodiment of the present application.

[0021] Figure 3 A schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. The described embodiments are only exemplary implementations of the present application, and not all implementations. Those skilled in the art can obtain other embodiments without creative labor by combining the embodiments of the present application, and these embodiments are also within the protection scope of the present application.

[0023] The goal of three-dimensional reconstruction is to recover the three-dimensional structure of an object or scene based on two-dimensional images. There are mainly two categories of three-dimensional reconstruction methods at present: one is a geometry-based method: relying on the multi-view geometric information of the camera (such as camera intrinsic parameters: focal length, principal point, etc.; extrinsic parameters: position, pose, etc.), the three-dimensional structure is determined through the projection geometry principle. The other is a deep learning-based method: with the development of neural networks, neural network models such as NeRF can be used to learn the implicit representation of images and generate three-dimensional structures from multi-view images.

[0024] In the field of industrial detection or security inspection, X-ray is usually used to detect objects to form X-ray images. There are high perspective superposition and noise in the X-ray images, and the targets under multiple angles are difficult to accurately match, so the demand for single-target three-dimensional reconstruction of multi-view X-ray images is increasing. However, X-ray images lack the geometric parameters of imaging equipment, and cannot use the projection geometry method to reconstruct the three-dimensional structure of X-ray images. Since X-ray is a penetrating imaging, the projection shape of the same object under different angles is quite different, and if a neural network model suitable for traditional camera imaging is directly used to reconstruct the three-dimensional structure of multi-view X-ray images, the accuracy is low.

[0025] Based on this, the embodiment of the present application provides a single target three-dimensional reconstruction method based on multi-view X-ray images, so as to improve the accuracy of three-dimensional reconstruction of X-ray images. In specific implementation, for any X-ray image in the plurality of X-ray images, the target detection model is used to detect the X-ray image, and the position coordinates and category label of at least one detection box of the X-ray image are determined. The X-ray image and the position coordinates of the detection box are input into the segmentation model to obtain the mask image of the target to be detected in the detection box. Based on the plurality of mask images corresponding to the plurality of X-ray images and the plurality of category labels, the plurality of target mask images belonging to the same target to be detected are determined. The plurality of target mask images belonging to the same target to be detected are input into the three-dimensional reconstruction model to obtain a plurality of three-dimensional coordinate points of the target to be detected. The model composed of the plurality of three-dimensional coordinate points is compared with a plurality of preset three-dimensional model templates to determine a target three-dimensional model template matched with the model. Based on the model and the target three-dimensional model template, a target three-dimensional model of the target to be detected is determined. Through the method provided in the embodiment of the present application, the mask images belonging to the same object can be determined based on the plurality of X-ray images, and the three-dimensional model can be directly reconstructed based on the mask image. The reconstructed three-dimensional model is fused with the three-dimensional model template determined in advance to obtain the target three-dimensional model of the target to be detected, so as to improve the accuracy of the reconstructed three-dimensional model.

[0026] In order to facilitate understanding of the technical solutions provided in the embodiments of the present application, the following will be specifically introduced in conjunction with the drawings in the embodiments.

[0027] Referring to Figure 1 Fig. 1 shows a flowchart of a single target three-dimensional reconstruction method based on multi-view X-ray images provided in an embodiment of the present application.

[0028] Optionally, the method can be executed by an image processing device. The method can include the following steps: S101: For any X-ray image in the plurality of X-ray images, a target detection model is used to detect the X-ray image, and the position coordinates and category label of at least one detection box of the X-ray image are determined.

[0029] Wherein, the X-ray image can represent an image obtained by photographing an object with X-rays. In the embodiment of the present application, the target detection model can be used to detect the X-ray image to obtain the target to be detected in the X-ray image. Wherein, the target detection model can be a pre-trained model, which can detect targets of a plurality of preset categories in the X-ray image. For example, the target detection model can be a Faster Region-based Convolutional Neural Network (Faster R-CNN).

[0030] In a specific implementation, after inputting the X-ray image into the target detection model, feature extraction can be performed on the X-ray image to obtain at least one feature vector, and the number of feature vectors corresponds to the number of targets to be detected in the X-ray image. For each feature vector, the feature vector is input into two parallel fully connected layers respectively, and the position offset corresponding to the detection box and the class probability are output. That is, when there are multiple targets to be detected in the X-ray image, multiple feature vectors can be obtained, and the position offset corresponding to multiple detection boxes and the class probability are output.

[0031] Among them, one fully connected layer of the target detection model is a classification branch, which is used to output the class probability corresponding to the detection box. The class probability includes multiple probabilities that the detection box belongs to multiple preset classes (including background) in the training data. Then the class label corresponding to the detection box can be determined based on the class probability. For example, the class corresponding to the maximum probability value in the class probability can be determined as the class label of the detection box.

[0032] The other fully connected layer is a regression branch, which is used to output the position offset corresponding to each detection box. Among them, the position offset represents the position offset between the detection box and the anchor box. The anchor box is a preset box of the target detection model, and the position coordinates of the anchor box are known. Therefore, the position coordinates of the detection box can be determined according to the position offset and the position coordinates of the anchor box.

[0033] S102: input the X-ray image and the position coordinates of the detection box into the segmentation model to obtain the mask image of the target to be detected in the detection box.

[0034] After obtaining the position coordinates of one or more X-ray images, the X-ray image and the position coordinates of each detection box can be input into the segmentation model, which can output the mask image of the target to be detected in the detection box. For example, the segmentation model can be a segment anything model (SAM). Among them, the mask image represents a binary image of the same size as the detection box, and the region of the target to be detected in the mask image is 1, and the pixel of the background region is 0.

[0035] S103: based on the multiple mask images corresponding to the multiple X-ray images and the multiple class labels, determine multiple target mask images belonging to the same target to be detected.

[0036] After performing the above steps S101 and S102 for each X-ray image, the class label and the mask image corresponding to each X-ray image can be obtained. Since multiple X-ray images can include multiple targets to be detected, multiple target mask images belonging to the same target to be detected can be determined based on the multiple mask images and the class labels.

[0037] In a possible implementation, the first mask image and the second mask image belonging to the same category label can be determined based on the plurality of category labels. That is, the plurality of mask images belonging to the same category label can be determined first, and any two mask images belonging to the same category label are denoted as the first mask image and the second mask image.

[0038] Then, the first similarity between the first mask image and the second mask image is determined. The first similarity is compared with a preset value to measure the similarity of the first mask image and the second mask image. The preset value represents the minimum similarity of the two detection targets corresponding to the two mask images being the same detection target. The specific value of the preset value is not limited in the embodiment of the application. For example, the preset value can be set to 0.5. When the first similarity is greater than the preset value, it can be determined that the first mask image and the second mask image are target mask images belonging to the same detection target. Based on the above method, the first similarity of any two mask images is calculated, and then a plurality of target mask images belonging to the same detection target can be determined. It should be noted that based on the above method, a plurality of detection targets can be determined, and each detection target corresponds to a plurality of target mask images.

[0039] In a possible implementation, the first similarity between the first mask image and the second mask image can be determined in the following manner: According to the above embodiment, when the X-ray image is detected by the target detection model, at least one feature vector of the X-ray image can be obtained, each feature vector corresponds to a detection box, and each detection box can obtain a mask image. Therefore, the first feature vector corresponding to the first mask image and the second feature vector corresponding to the second mask image can be determined. The cosine similarity between the first feature vector and the second feature vector is calculated, which is taken as the first similarity between the first mask image and the second mask image.

[0040] S104: inputting the plurality of target mask images belonging to the same detection target into a three-dimensional reconstruction model to obtain a plurality of three-dimensional coordinate points of the detection target.

[0041] The three-dimensional reconstruction model can be a Dense and Unconstrained Stereo 3D Reconstruction (DUSt3R) model. The DUSt3R model does not need to use the device parameters for shooting the X-ray image, and can directly obtain the plurality of three-dimensional coordinate points of the detection target based on the plurality of target mask images.

[0042] Optionally, in order to visualize the three-dimensional structure of the target to be detected, facilitating subsequent analysis, the plurality of three-dimensional coordinate points can be projected into a three-dimensional voxel grid. The three-dimensional voxel grid can be a cubic region including all the three-dimensional coordinate points, and the cubic region is divided into a plurality of small cubes, i.e., voxel units, each of which corresponds to a three-dimensional coordinate. Each three-dimensional coordinate point is projected into the corresponding voxel unit of the three-dimensional voxel grid. When a three-dimensional coordinate point exists in a voxel unit, the value of the voxel unit is marked as 1. The values of the remaining voxel units without three-dimensional coordinate points are marked as 0, and then a three-dimensional array corresponding to the three-dimensional voxel grid can be obtained as a preliminary model of the target to be detected.

[0043] S105: comparing the model composed of the plurality of three-dimensional coordinate points with a plurality of preset three-dimensional model templates to determine a target three-dimensional model template matched with the model.

[0044] The plurality of preset three-dimensional model templates are determined in advance. Three-dimensional modeling can be performed on each target to be detected in advance to obtain a plurality of preset three-dimensional models corresponding to the target to be detected. For example, three-dimensional modeling can be performed on the target to be detected from different observation angles based on computer-aided design (CAD), CT scanning, or the like to obtain a plurality of preset three-dimensional models of the target to be detected.

[0045] Similarly, each preset three-dimensional model can be projected into a three-dimensional voxel grid, and the obtained three-dimensional array can be used as a preset three-dimensional model template. The three-dimensional voxel grid projected by the preset three-dimensional model and the three-dimensional voxel grid projected by the three-dimensional coordinate points are of the same size.

[0046] Based on this, the intersection-over-union and the chamfer distance between the model composed of the plurality of three-dimensional coordinate points and each preset three-dimensional model template are calculated. The larger the intersection-over-union, the higher the similarity between the model and the preset three-dimensional model template; the larger the chamfer distance, the lower the similarity between the model and the preset three-dimensional model template. For example, when the model composed of the plurality of three-dimensional coordinate points and the preset three-dimensional model template can be represented as three-dimensional arrays, the intersection-over-union can be represented as the ratio between the intersection of the two three-dimensional arrays and the union of the two three-dimensional arrays. The intersection of the two three-dimensional arrays can represent the number of positions where the two values at the corresponding positions are both 1, and the union of the two three-dimensional arrays can represent the number of positions where at least one of the two values at the corresponding positions is 1. The chamfer distance between the model composed of the plurality of three-dimensional coordinate points and the preset three-dimensional model template can be represented as the chamfer distance between the three-dimensional array corresponding to the plurality of three-dimensional coordinate points and the three-dimensional array corresponding to the preset three-dimensional model template.

[0047] Then, based on the intersection-over-union ratio and the chamfer distance, a second similarity between the model composed of the plurality of three-dimensional coordinate points and the preset three-dimensional model template is determined. For example, a first weight can be assigned to the intersection-over-union ratio and a second weight can be assigned to the chamfer distance, then a first product of the intersection-over-union ratio and the first weight and a second product of the chamfer distance and the second weight are calculated, and then the second similarity is obtained according to a difference between the first product and the second product. When the intersection-over-union ratio is mainly used to determine the second similarity, the first weight can be set to be greater than the second weight. When the chamfer distance is mainly used to determine the second similarity, the first weight can be set to be less than the second weight. In the embodiments of the present application, the first weight and the second weight can be set to be 0.5.

[0048] After obtaining the plurality of second similarities between the model and the plurality of preset three-dimensional model templates, the preset three-dimensional model template corresponding to the maximum value in the plurality of second similarities can be determined as the target three-dimensional model template, which has the highest similarity with the model.

[0049] It should be noted that the above embodiments introduce a method of determining the second similarity based on the intersection-over-union ratio and the chamfer distance. Alternatively, one of the intersection-over-union ratio and the chamfer distance can be used as the second similarity, which does not affect the implementation of the embodiments of the present application.

[0050] S106: Determine the target three-dimensional model of the target to be detected based on the model and the target three-dimensional model template.

[0051] After determining the target three-dimensional model template, the model and the target three-dimensional model template can be fused by weighting to obtain the target three-dimensional model. For example, after determining the first three-dimensional array corresponding to the plurality of three-dimensional coordinate points and the second three-dimensional array corresponding to the target three-dimensional model template, the two values at the corresponding positions in the first three-dimensional array and the second three-dimensional array can be weighted and summed to obtain a weighted three-dimensional array as the target three-dimensional model.

[0052] Based on the method provided in the above embodiments, a mask image belonging to the same object can be determined based on a plurality of X-ray images, and a three-dimensional model can be directly reconstructed based on the mask image. The reconstructed three-dimensional model is fused with a three-dimensional model template determined in advance to obtain a target three-dimensional model of the target to be detected, thereby improving the accuracy of the reconstructed three-dimensional model.

[0053] Based on the above method embodiments, the embodiments of the present application also provide a single-target three-dimensional reconstruction device based on multi-view X-ray images. For details, please refer to Figure 2 Fig. 1 shows a schematic diagram of a single-target three-dimensional reconstruction device based on multi-view X-ray images provided by the embodiments of the present application.

[0054] The device 200 comprises: The target detection unit 201 is configured to, for any X-ray image in the plurality of X-ray images, detect the X-ray image by using a target detection model to determine position coordinates and a category label of at least one detection frame of the X-ray image. The segmentation unit 202 is configured to input the X-ray image and the position coordinates of the detection frame into a segmentation model to obtain a mask image of a target to be detected in the detection frame. The classification unit 203 is configured to determine, based on a plurality of mask images respectively corresponding to the plurality of X-ray images and a plurality of category labels, a plurality of target mask images belonging to a same target to be detected. The three-dimensional reconstruction unit 204 is configured to input the plurality of target mask images belonging to the same target to be detected into a three-dimensional reconstruction model to obtain a plurality of three-dimensional coordinate points of the target to be detected. The matching unit 205 is configured to compare a model composed of the plurality of three-dimensional coordinate points with a plurality of preset three-dimensional model templates to determine a target three-dimensional model template matching the model. The determination unit 206 is configured to determine, based on the model and the target three-dimensional model template, a target three-dimensional model of the target to be detected.

[0055] In a possible implementation, the classification unit 203 is specifically configured to determine, based on a plurality of category labels corresponding to a plurality of X-ray images, a first mask image and a second mask image belonging to a same category label; determine a first similarity between the first mask image and the second mask image; and when the first similarity is greater than a preset value, determine that the first mask image and the second mask image are target mask images belonging to a same target to be detected.

[0056] In a possible implementation, the classification unit 203 is specifically configured to obtain a first feature vector corresponding to the first mask image and a second feature vector corresponding to the second mask image, wherein the first feature vector is obtained by detecting an X-ray image corresponding to the first mask image by using the target detection model; and calculate a cosine similarity between the first feature vector and the second feature vector.

[0057] In a possible implementation, the determination process of the plurality of preset three-dimensional model templates includes: performing three-dimensional modeling on the target to be detected to obtain a plurality of preset three-dimensional models; and projecting the preset three-dimensional models to a three-dimensional voxel grid to obtain the preset three-dimensional model templates.

[0058] In a possible implementation, the matching unit 205 is specifically configured to project the plurality of three-dimensional coordinate points to the three-dimensional voxel grid to obtain the model; calculate an intersection-over-union ratio and a chamfer distance between the model and the preset three-dimensional model template; determine a second similarity between the model and the preset three-dimensional model template based on the intersection-over-union ratio and the chamfer distance; and determine the target three-dimensional model template based on a maximum value of the plurality of second similarities between the model and the plurality of preset three-dimensional model templates.

[0059] In a possible implementation, the determining unit 206 is specifically configured to perform weighted fusion of the model and the target three-dimensional model to obtain the target three-dimensional model.

[0060] In a possible implementation, the target detection unit 201 is specifically configured to input the X-ray image into the target detection model to output a position offset corresponding to the at least one detection frame and a class probability; determine a position coordinate of the detection frame based on the position offset, and determine a class label corresponding to the detection frame based on the class probability.

[0061] In a possible implementation, the three-dimensional reconstruction model includes a dense unconstrained stereo three-dimensional reconstruction DUSt3R model.

[0062] Based on the above method embodiments and device embodiments, an electronic device is further provided in the embodiments of the present application. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0063] Referring to Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an electronic device provided in the embodiments of the present application.

[0064] The device 300 includes a memory 301 and a processor 302. The memory 301 is configured to store related program codes. The processor 302 is configured to invoke the program codes and perform the single-target three-dimensional reconstruction method based on multi-view X-ray images described in the above method embodiments.

[0065] In addition, the embodiments of the present application further provide a computer readable storage medium configured to store a computer program, and the computer program is configured to perform the single-target three-dimensional reconstruction method based on multi-view X-ray images described in the above method embodiments.

[0066] The embodiments of the present application further provide a computer program product including computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the single-target three-dimensional reconstruction method based on multi-view X-ray images described in the above method embodiments.

[0067] Note that the computer readable medium in the above embodiments can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination thereof. More specific examples of the computer readable storage medium include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0068] The program code of the computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0069] Note that the embodiments described in the specification are progressive, and each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be mutually referred to. Especially, the system or device embodiments are described more simply, and the relevant parts can be referred to the part of the method embodiments. The device embodiments described above are merely illustrative, and the units or modules described as separate components can or can not be physically separated, and the components shown as units or modules can or can not be physical modules, i.e. can be located in one place or distributed on multiple network units, and some or all of the units or modules can be selected to achieve the purpose of the embodiments. Those skilled in the art can understand and implement it without creative effort.

[0070] The flow diagrams and block diagrams in the drawings are illustrations of architectures, functions, and operations that can be implemented in methods, apparatus, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code that comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0071] It should be understood that, in this application, "at least one", "one or more", "multiple", "two or more" means one or more, "and / or" is used to describe the association between associated objects, and means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or the like means any combination of these items, including single item or any combination of multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0072] It should also be noted that the relationship terms such as first and second, etc. in this application are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0073] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this application can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0074] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A single object 3D reconstruction method based on multi-view X-ray images, characterized in that, The method comprises: For any X-ray image in a plurality of X-ray images, detecting the X-ray image by using a target detection model to determine the position coordinates and category labels of at least one detection box of the X-ray image; inputting the X-ray image and the position coordinates of the detection box into a segmentation model to obtain a mask image of a target to be detected in the detection box; based on a plurality of mask images corresponding to the plurality of X-ray images and a plurality of category labels, determining a plurality of target mask images belonging to the same target to be detected; inputting the plurality of target mask images belonging to the same target to be detected into a three-dimensional reconstruction model to obtain a plurality of three-dimensional coordinate points of the target to be detected; comparing the model composed of the plurality of three-dimensional coordinate points with a plurality of preset three-dimensional model templates to determine a target three-dimensional model template matching the model; based on the model and the target three-dimensional model template, determining a target three-dimensional model of the target to be detected.

2. The method of claim 1, wherein, The method comprises: based on a plurality of category labels corresponding to a plurality of X-ray images, determining a first mask image and a second mask image belonging to the same category label; determining a first similarity between the first mask image and the second mask image; when the first similarity is greater than a preset value, determining that the first mask image and the second mask image are target mask images belonging to the same target to be detected.

3. The method of claim 2, wherein, The method comprises: obtaining a first feature vector corresponding to the first mask image and a second feature vector corresponding to the second mask image, wherein the first feature vector is obtained by detecting the X-ray image corresponding to the first mask image by using the target detection model; calculating the cosine similarity between the first feature vector and the second feature vector.

4. The method of claim 1, wherein, The determination process of the plurality of preset three-dimensional model templates comprises: three-dimensional modeling the target to be detected to obtain a plurality of preset three-dimensional models; projecting the preset three-dimensional models onto a three-dimensional voxel grid to obtain the preset three-dimensional model templates.

5. The method of claim 4, wherein, The method comprises: projecting the plurality of three-dimensional coordinate points onto the three-dimensional voxel grid to obtain the model; calculating the intersection over union and the chamfer distance between the model and the preset three-dimensional model template; based on the intersection over union and the chamfer distance, determining a second similarity between the model and the preset three-dimensional model template; based on the maximum value of the plurality of second similarities between the model and the plurality of preset three-dimensional model templates, determining the target three-dimensional model template.

6. The method of claim 1, wherein, The method comprises: weighting and fusing the model and the target three-dimensional model template to obtain the target three-dimensional model.

7. The method of claim 1, wherein, The target detection model is used to detect the X-ray image, and position coordinates and a category label of at least one detection frame of the X-ray image are determined, including: The X-ray image is input into the target detection model, and a position offset corresponding to at least one detection frame and a category probability are output; The position coordinates of the detection frame are determined based on the position offset, and the category label corresponding to the detection frame is determined based on the category probability.

8. The method of claim 1, wherein, The three-dimensional reconstruction model includes a dense unconstrained stereo three-dimensional reconstruction DUSt3R model.

9. An apparatus for single object three-dimensional reconstruction based on multi-view X-ray images, characterized by comprising: The device includes: A target detection unit is configured to use a target detection model to detect any X-ray image in a plurality of X-ray images, determine position coordinates and a category label of at least one detection frame of the X-ray image; A segmentation unit is configured to input the X-ray image and the position coordinates of the detection frame into a segmentation model, and obtain a mask image of a target to be detected in the detection frame; A classification unit is configured to determine a plurality of target mask images belonging to the same target to be detected based on a plurality of mask images corresponding to the plurality of X-ray images respectively and a plurality of category labels; A three-dimensional reconstruction unit is configured to input the plurality of target mask images belonging to the same target to be detected into a three-dimensional reconstruction model, and obtain a plurality of three-dimensional coordinate points of the target to be detected; A matching unit is configured to compare a model composed of the plurality of three-dimensional coordinate points with a plurality of preset three-dimensional model templates, and determine a target three-dimensional model template matched with the model; A determination unit is configured to determine a target three-dimensional model of the target to be detected based on the model and the target three-dimensional model template.

10. An electronic device, comprising: The device includes a memory and a processor; The memory is configured to store related program codes; The processor is configured to call the program codes and execute the single-target three-dimensional reconstruction method based on a plurality of X-ray images according to any one of claims 1 to 8.

Images