Hip joint three-dimensional reconstruction method and device based on biplane X-ray film
Through the three-dimensional reconstruction method of hip joints with double-plane X-ray, downsampling and upsampling technology combined with CT image training data, the accuracy of the three-dimensional reconstruction generated by X-rays is improved, and the problem of low accuracy in the prior art is solved .
Patent Information
- Application Number
- CN202411163222.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-23
AI Technical Summary
The accuracy of the prior art to generate three-dimensional reconstruction results of hip joints through X-rays is not high.
Using a three-dimensional reconstruction method of hip joint based on double-plane X-ray, the hip joint positive and lateral images are obtained, and the three-dimensional reconstruction convolution model is input for downsampling feature extraction and transformation and fusion, and combined with CT image training data, upsampling is performed. to improve the accuracy of reconstruction results.
The accuracy of the three-dimensional reconstruction results generated by X-rays is significantly improved, the problem of insufficient training data is made up for, and the reconstruction accuracy is achieved.
Smart Images

Figure CN119068117A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of medical image processing, and in particular to a method and device for three-dimensional reconstruction of a hip joint based on dual-plane X-ray films. Background Art
[0002] 3D reconstruction of the hip joint is an important step for various clinical applications. It can be used for surgical planning, precise implant selection, patient-specific implant manufacturing or intraoperative jig printing that perfectly fits the anatomy. X-rays are often used due to their wide availability, low price, short scanning time and low ionization level, which results in less radiation compared to CT scanners.
[0003] Therefore, it is of great value to use X-rays to replace CT to generate three-dimensional reconstruction results of the hip joint. However, the accuracy of the three-dimensional reconstruction results generated by X-rays is currently not high. Summary of the invention
[0004] The problem solved by this application is that the accuracy of three-dimensional reconstruction results generated by current X-ray films is not high.
[0005] To solve the above problems, the first aspect of the present application provides a method for three-dimensional reconstruction of hip joint based on dual-plane X-ray films, comprising:
[0006] Obtain anteroposterior and lateral hip joint images;
[0007] Inputting the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively to obtain a corresponding plurality of down-sampled feature maps;
[0008] The down-sampled feature map of the hip joint frontal image and the down-sampled feature map corresponding to the hip joint lateral image are converted and fused to obtain corresponding multiple three-dimensional feature maps;
[0009] Inputting the multiple three-dimensional feature maps into an upsampling structure to obtain a three-dimensional reconstruction result of the hip joint;
[0010] The three-dimensional reconstruction convolution model is trained based on a first sample frontal image, a second sample lateral image and an annotated CT image, and the first sample frontal image or the second sample lateral image is generated by the annotated CT image.
[0011] The second aspect of the present application provides a hip joint three-dimensional reconstruction device based on a dual-plane X-ray film, which comprises:
[0012] An image acquisition module, which is used to acquire an anteroposterior image of the hip joint and a lateral image of the hip joint;
[0013] A downsampling module, which is used to input the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively, and obtain a corresponding plurality of down-sampled feature maps;
[0014] A conversion and fusion module is used to convert and fuse the downsampled feature map of the hip joint frontal image and the downsampled feature map corresponding to the hip joint lateral image to obtain a corresponding plurality of three-dimensional feature maps;
[0015] An upsampling module, which is used to input the multiple three-dimensional feature maps into an upsampling structure to obtain a three-dimensional reconstruction result of the hip joint;
[0016] The three-dimensional reconstruction convolution model is trained based on a first sample frontal image, a second sample lateral image and an annotated CT image, and the first sample frontal image or the second sample lateral image is generated by the annotated CT image.
[0017] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;
[0018] The memory is used to store programs;
[0019] The processor, coupled to the memory, is configured to execute the program to:
[0020] Obtain anteroposterior and lateral hip joint images;
[0021] Inputting the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively to obtain a corresponding plurality of down-sampled feature maps;
[0022] The down-sampled feature map of the hip joint frontal image and the down-sampled feature map corresponding to the hip joint lateral image are converted and fused to obtain corresponding multiple three-dimensional feature maps;
[0023] Inputting the multiple three-dimensional feature maps into an upsampling structure to obtain a three-dimensional reconstruction result of the hip joint;
[0024] The three-dimensional reconstruction convolution model is trained based on a first sample frontal image, a second sample lateral image and an annotated CT image, and the first sample frontal image or the second sample lateral image is generated by the annotated CT image.
[0025] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned method for three-dimensional reconstruction of the hip joint based on dual-plane X-ray films.
[0026] In the present application, the two-dimensional images of the anteroposterior and lateral positions of the hip joint are down-sampled respectively, the corresponding down-sampled images are converted and fused to obtain three-dimensional images, and the three-dimensional images are up-sampled respectively to obtain the final three-dimensional reconstruction results, thereby utilizing two-dimensional downsampling and three-dimensional upsampling to greatly increase the accuracy of reconstruction.
[0027] In the present application, training anteroposterior images or lateral images of the hip joint are generated through CT images, thereby making up for the current shortage of training data and improving the accuracy of the three-dimensional reconstruction convolution model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 is a schematic diagram of a hip joint three-dimensional reconstruction method according to an embodiment of the present application;
[0029] Figure 2 is a flow chart of a method for three-dimensional reconstruction of a hip joint according to an embodiment of the present application;
[0030] Figure 3 A flowchart of a hip joint three-dimensional reconstruction method model training according to an embodiment of the present application;
[0031] Figure 4 It is a structural diagram of a conversion fusion module of a hip joint three-dimensional reconstruction method according to an embodiment of the present application;
[0032] Figure 5 It is a schematic diagram of the architecture of the two-dimensional to three-dimensional conversion of the three-dimensional reconstruction method of the hip joint according to the embodiment of the present application;
[0033] Figure 6 is a structural block diagram of a hip joint three-dimensional reconstruction device according to an embodiment of the present application;
[0034] Figure 7 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0035] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0036] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.
[0037] In response to the above problems, the present application provides a new three-dimensional reconstruction scheme of the hip joint based on dual-plane X-rays, which solves the problem of low accuracy of three-dimensional reconstruction results generated by current X-rays by down-sampling the frontal and lateral two-dimensional images respectively, and up-sampling the three-dimensional images obtained by conversion and fusion.
[0038] The present application embodiment provides a method for three-dimensional reconstruction of hip joint based on dual-plane X-ray film, and the specific scheme of the method is as follows: Figure 1-Figure 5 As shown, the method can be performed by a hip joint three-dimensional reconstruction device based on a dual-plane X-ray film, and the hip joint three-dimensional reconstruction device based on a dual-plane X-ray film can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 , Figure 2 As shown, it is a flow chart of a method for three-dimensional reconstruction of hip joint based on dual-plane X-ray film according to an embodiment of the present application; wherein the method for three-dimensional reconstruction of hip joint based on dual-plane X-ray film comprises:
[0039] S101, acquiring an anteroposterior image and a lateral image of the hip joint;
[0040] The anteroposterior image of the hip joint and the lateral image of the hip joint are X-ray films of the hip joint area of any object.
[0041] The anteroposterior image of the hip joint and the lateral image of the hip joint are anteroposterior and lateral images of the same object in the same posture, and a three-dimensional reconstruction result in the position state is generated based on the image.
[0042] In the present application, the anteroposterior image of the hip joint is an X-ray taken from the front, and the lateral image is an X-ray taken from the left side of the photographed object. In this way, the input data of the three-dimensional reconstruction convolution model is limited, and the anteroposterior image of the hip joint and the lateral image of the hip joint, as well as the first sample anteroposterior image and the second sample lateral image are limited and unified to improve the accuracy during training and actual reconstruction.
[0043] S102, respectively inputting the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model to obtain a corresponding plurality of down-sampled feature maps;
[0044] In the present application, the first processing structure includes a preprocessing part and a downsampling part; by downsampling in sequence, a corresponding plurality of downsampled feature maps are obtained.
[0045] In the present application, there are two first processing structures (it can also be understood that the first processing structure has two branches with the same structure, and the branches are independent of each other), which respectively process the anteroposterior image of the hip joint and the lateral image of the hip joint, and respectively output the corresponding multiple down-sampled feature maps; the two first processing structures have the same structure, but different specific parameters.
[0046] S103, converting and fusing the downsampled feature map of the hip joint frontal image and the downsampled feature map corresponding to the hip joint lateral image to obtain a corresponding plurality of three-dimensional feature maps;
[0047] In the present application, the downsampled feature maps of the anteroposterior hip joint images of the same order and the downsampled feature maps corresponding to the lateral hip joint images of the same order are converted and fused to obtain the three-dimensional feature maps corresponding to the same order.
[0048] In the present application, the conversion fusion structure is composed of four identical structures, which are used to process the conversion fusion of downsampled feature maps of hip joint anteroposterior images and downsampled feature maps corresponding to hip joint lateral images of different orders; their specific structures are the same, but the parameters are different.
[0049] In this application, the sizes of feature maps of different orders are different.
[0050] In the present application, the height of the anteroposterior image of the hip joint and the lateral image of the hip joint are the same, but the width may be different.
[0051] S104, input the multiple three-dimensional feature maps into the upsampling structure to obtain a three-dimensional reconstruction result of the hip joint; the three-dimensional reconstruction convolution model is trained based on the first sample frontal image, the second sample lateral image and the annotated CT image, and the first sample frontal image or the second sample lateral image is generated by the annotated CT image.
[0052] In this application, the three-dimensional feature map obtained by transformation and fusion is added to the upsampling structure as a long connection, so as to achieve the superposition and highlighting of transformation and fusion features, and further improve the accuracy of three-dimensional reconstruction.
[0053] In the present application, the two-dimensional images of the anteroposterior and lateral positions of the hip joint are down-sampled respectively, the corresponding down-sampled images are converted and fused to obtain three-dimensional images, and the three-dimensional images are up-sampled respectively to obtain the final three-dimensional reconstruction results, thereby utilizing two-dimensional downsampling and three-dimensional upsampling to greatly increase the accuracy of reconstruction.
[0054] In the present application, training anteroposterior images or lateral images of the hip joint are generated through CT images, thereby making up for the current shortage of training data and improving the accuracy of the three-dimensional reconstruction convolution model training.
[0055] In one embodiment, in combination Figure 3 As shown, the training process of the 3D reconstruction convolutional model includes:
[0056] S201, acquiring a CT image and a corresponding first sample frontal image, or a second sample lateral image;
[0057] In this step, a CT image of the hip joint of any object and at least one X-ray film of the hip joint are obtained. The X-ray film can be an anteroposterior image or a lateral image.
[0058] It should be noted that for the training of the 3D reconstruction convolutional model, the training data requires anteroposterior and lateral X-rays, as well as CT images in the same posture; however, in the actual acquisition process, it is found that only a small amount of complete data can be obtained, and most of the data only contains one X-ray (anteroposterior or lateral).
[0059] Therefore, in this step, a CT image and a corresponding first sample frontal image, or a second sample lateral image, are obtained, which specifically means: obtaining a CT image and a corresponding first sample frontal image and a second sample lateral image, or a CT image and a corresponding first sample frontal image, or a CT image and a corresponding second sample lateral image.
[0060] S202, generating a first sample frontal image based on the CT image, or generating a second sample lateral image based on the CT image;
[0061] In this step, if the CT image has a corresponding first sample frontal image and a second sample lateral image, this step does not need to be performed; if the CT image only has a corresponding first sample frontal image, it is necessary to generate the second sample lateral image based on the CT image; if the CT image only has a corresponding second sample lateral image, it is necessary to generate the first sample frontal image based on the CT image.
[0062] S203, inputting the first sample frontal image and the second sample lateral image into a three-dimensional reconstruction convolution model to obtain a three-dimensional reconstruction prediction result;
[0063] S204, generating a three-dimensional reconstruction annotation result based on the CT image;
[0064] In this step, three-dimensional reconstruction is performed through the CT image, and the reconstruction result obtained is the three-dimensional reconstruction annotation result; the specific method of performing three-dimensional reconstruction of the CT image can be to load the CT image in DICOM format and then perform three-dimensional reconstruction through surface rendering (contour extraction-three-dimensional surface generation-surface smoothing), or it can be performed through volume rendering, or the reconstruction result can be directly obtained by using professional software such as 3D Slicer; the specific method is not limited in this application.
[0065] S205, calculating the overall loss of the 3D reconstruction convolution model according to the 3D reconstruction prediction result and the 3D reconstruction annotation result;
[0066] In the present application, the overall loss includes reconstruction loss and CE loss, and the two losses are associated through a weight coefficient.
[0067] The reconstruction loss is the mean square error loss.
[0068] S206, iterating the parameters of the 3D reconstruction convolution model according to the overall loss until the loss converges.
[0069] In this application, X-ray films are generated by projecting CT images, thereby making up for the problem of insufficient training data and improving the accuracy of training.
[0070] In one embodiment, generating a first sample normal image based on a CT image includes:
[0071] Acquire CT images, and generate three-dimensional point clouds based on the CT images;
[0072] Determine a projection line corresponding to the first sample positive image;
[0073] For each projection line, calculate the path of the projection line passing through the three-dimensional point cloud, and generate the grayscale value of the projection line in the two-dimensional projection image;
[0074] The grayscale image obtained by traversing all projection lines is the first sample positive image.
[0075] In this application, each pixel value (called Hounsfield unit, HU value) of the CT image represents the X-ray attenuation coefficient of that position. The CT image is three-dimensional volume data, including multiple slices; based on this, a three-dimensional point cloud can be generated, and each point has a pixel value.
[0076] In this application, the attenuation value of each ray is integrated by calculating the path of the ray passing through the three-dimensional CT data. This process simulates the physical process of X-rays passing through an object and calculates the attenuation image on the two-dimensional projection plane. Specifically, for each projection line, the attenuation values of all voxels (each small volume unit in the CT data) passing along this line are accumulated to generate the grayscale value of the point in the two-dimensional projection image.
[0077] In this application, through the above calculation, projections at different angles are superimposed to generate a two-dimensional projection image similar to a traditional X-ray. This image is very close to the effect of actual X-ray shooting, which is the first sample positive image.
[0078] In one embodiment, in combination Figure 1 As shown, the hip joint frontal image and the hip joint lateral image are respectively input into the first processing structure of the three-dimensional reconstruction convolution model to obtain corresponding multiple down-sampled feature maps, including:
[0079] Perform attention and global feature extraction on the anteroposterior image of the hip joint to obtain an anteroposterior extraction feature map;
[0080] The feature map extracted from the positive position is sequentially downsampled to obtain a corresponding positive position one-downsampled feature map, a positive position two-downsampled feature map, a positive position three-downsampled feature map, and a positive position four-downsampled feature map;
[0081] Perform attention and global feature extraction on the lateral image of the hip joint to obtain a lateral extraction feature map;
[0082] The side position extraction feature maps are downsampled in sequence to obtain corresponding side position one downsampled feature map, side position two downsampled feature map, side position three downsampled feature map, and side position four downsampled feature map.
[0083] In the present application, the attention extraction (CBM) can be spatial attention extraction and / or channel attention extraction; the corresponding spatial features and channel features are strengthened through attention extraction.
[0084] In this application, better global feature enhancement is achieved through the setting of attention extraction-global extraction.
[0085] Among them, the process of global extraction is:
[0086] The output feature map of the class attention is used as the input feature map of the global extraction;
[0087] Input the input feature map into a 1×1 convolution layer set in parallel to obtain a first extraction map, a second extraction map, and a third extraction map respectively; wherein the first extraction map and the third extraction map are in HW×N format, and the second extraction map is in N×HW format;
[0088] Multiply the first extracted image and the second extracted image to obtain a multiplied feature image;
[0089] Perform softmax processing on the multiplied feature map to obtain the multiplication coefficient;
[0090] Multiplying the multiplication coefficient with the third extraction map to obtain a coefficient multiplication map;
[0091] Add the coefficient multiplication map and the input feature map to obtain the globally extracted output feature map.
[0092] In the present application, the input feature map is divided into three branches for 1×1 convolution processing. The dimensions of the first branch and the third branch are adjusted to HW×N format, and the dimension of the second branch is N×HW format; then the first branch is multiplied by the third branch to obtain a feature map with a dimension of HW×HW, and a coefficient is obtained through softmax processing; the coefficient is multiplied with the third branch, and the multiplied feature map is added to the input feature map to obtain an output feature map.
[0093] In this application, the dimensions of the first branch and the second branch are reversed through 1×1 convolution, thereby realizing multiplication between feature maps; through this multiplication, long-range dependencies are captured; by setting coefficients, the dependencies are embedded in the feature maps; and on this basis, the original input is added and a residual connection is used to embed the global extraction into the model without destroying the parameters.
[0094] In this application, long-range dependencies between features are captured through global extraction, so that the model can extract global features, making up for the defect that the current model mainly extracts local features.
[0095] In the present application, the response at a position is calculated as the weighted sum of the features at all positions, so that when processing the information at each position, the processing is performed on the basis of considering all the position information that can be considered.
[0096] In one embodiment, in combination Figure 1 As shown, the down-sampled feature map of the hip joint frontal image and the down-sampled feature map corresponding to the hip joint lateral image are converted and fused to obtain corresponding multiple three-dimensional feature maps, including:
[0097] The down-sampling feature map in the frontal position and the down-sampling feature map in the lateral position are transformed and fused to obtain a corresponding first three-dimensional feature map;
[0098] The frontal two-downsampled feature map and the lateral two-downsampled feature map are transformed and fused to obtain a corresponding second three-dimensional feature map;
[0099] The three-downsampling feature map of the front position and the three-downsampling feature map of the lateral position are transformed and fused to obtain the corresponding third three-dimensional feature map;
[0100] The frontal four-downsampling feature map and the lateral four-downsampling feature map are transformed and fused to obtain the corresponding fourth three-dimensional feature map.
[0101] Among them, the sizes of the first, second, third, and fourth order feature maps and the three-dimensional feature map decrease successively.
[0102] In this application, after obtaining the first three-dimensional feature map, the second three-dimensional feature map, the third three-dimensional feature map, and the fourth three-dimensional feature map, they are input into the upsampling structure to obtain the three-dimensional reconstruction result of the hip joint, specifically:
[0103] The fourth three-dimensional feature map is set as the upsampled basic feature map, and upsampled to obtain a third upsampled three-dimensional map;
[0104] After merging the third upsampled three-dimensional image and the third three-dimensional feature image, upsampling is performed to obtain a second upsampled three-dimensional image;
[0105] After merging the second upsampled three-dimensional image and the second three-dimensional feature image, upsampling is performed to obtain a first upsampled three-dimensional image;
[0106] The first upsampled three-dimensional image is combined with the first three-dimensional feature image and then convolved to obtain a three-dimensional reconstruction result of the hip joint.
[0107] In the present application, the three-dimensional image is upsampled to obtain the corresponding three-dimensional reconstruction result.
[0108] In one embodiment, in combination Figure 4 As shown, the conversion and fusion of the down-sampling feature map in the front position and the down-sampling feature map in the side position to obtain the corresponding first three-dimensional feature map includes:
[0109] Perform a two-dimensional to three-dimensional transformation on the normal-position downsampling feature map to obtain a normal-position transformation feature map, wherein the size of the normal-position transformation feature map is H×W×N;
[0110] Performing a two-dimensional to three-dimensional conversion on a downsampled feature map of the lateral position to obtain a lateral position conversion feature map, wherein the size of the lateral position conversion feature map is H×N×W; wherein the width of the frontal position conversion feature map is the same as the depth of the lateral position conversion feature map, and the width of the lateral position conversion feature map is the same as the depth of the frontal position conversion feature map;
[0111] Performing azimuth transformation on the lateral position transformation characteristic map to obtain an azimuth transformation characteristic map, wherein the azimuth transformation characteristic map has the same size as the frontal position transformation characteristic map;
[0112] The azimuth conversion feature map is superimposed with the orthographic conversion feature map to obtain the first three-dimensional feature map.
[0113] In the present application, conversion and fusion (COF) is described in detail by taking the conversion and fusion of a down-sampling feature map in the front position and a down-sampling feature map in the side position as an example, wherein the processing process can also be applied to the down-sampling feature maps in the front position two, three, and four.
[0114] In the present application, assuming that the size of a downsampled feature map in the orthogonal position is H×A, and the size of a downsampled feature map in the lateral position is H×B, then the size of the generated orthogonal transformation feature map is H×A×B, and the size of the generated lateral transformation feature map is H×B×A; at this time, the sizes of the two transformation feature maps are different and cannot be superimposed, and the lateral transformation feature map needs to be adjusted to its size H×A×B; the size adjustment can be a matrix transformation, and the width and depth of the image are exchanged by applying a rotation matrix to the coordinate system of the image (it can also be considered as a 90-degree rotation around the height axis).
[0115] In the present application, the feature map superposition is to add the grayscale values of the corresponding coordinates, so as to deeply combine the feature map of the front position and the feature map of the side position and strengthen the corresponding features.
[0116] In one embodiment, in combination Figure 5 As shown, the two-dimensional to three-dimensional conversion of the normal down-sampling feature map is performed to obtain the normal conversion feature map, including:
[0117] Flatten the downsampled feature map in the normal position to obtain a corresponding flattened feature map;
[0118] Perform full connection, normalization and ReLU processing on the flattened feature map to obtain a normalized feature map;
[0119] Reshape the normalized feature map to obtain a three-dimensional reshaped feature map;
[0120] The three-dimensional reconstruction feature map is subjected to three-dimensional stabilization processing to obtain the orthotopic transformation feature map.
[0121] Among them, Figure 5 As shown in the figure, flattening is to flatten the multi-dimensional convolution feature map into a one-dimensional vector; FC is full connection processing, IN is normalization processing; Reshape is reshaping processing, which converts the tensor from one shape to another without changing the order of the data; three-dimensional stabilization processing (Basic3D Block) can be multiple three-dimensional convolutions (set in sequence) to process the reshaped three-dimensional image to improve the overall stability of the network.
[0122] In a specific implementation, the softmax process is a SoftMax-like operation, and its activation function is:
[0123]
[0124] Among them, SoftMax1(x) is the activation function, x i 、x j is the element in the input vector, i and j are the element numbers.
[0125] In this application, SoftMax is a mathematical function that is usually used to convert a set of arbitrary real numbers into real numbers representing probability distributions. It is essentially a normalization function that can convert a set of arbitrary real values into probability values between [0,1], because SoftMax converts them to values between 0 and 1, so they can be interpreted as probabilities. If one of the inputs is small or negative, SoftMax turns it into a small probability, and if the input is large, it turns it into a large probability, but it will always remain between 0 and 1.
[0126] However, for the standard SoftMax function, since the input is mapped between 0 and 1, and the sum of all output values is 1, this means that even if some input values are very small, they will have a non-zero output value after being processed by the SoftMax function. This will also cause the noise to be amplified, resulting in the final output result being affected by greater noise.
[0127] In this application, a 1 is added to the denominator of the SoftMax-like function; this change means that when the input value is very small, their output value can be closer to zero. This allows the corresponding output to tend to zero when there is no valuable information to add, thus greatly reducing unnecessary noise.
[0128] An embodiment of the present application provides a three-dimensional reconstruction device for a hip joint based on a dual-plane X-ray film, which is used to execute the three-dimensional reconstruction method for a hip joint based on a dual-plane X-ray film described in the above content of the present application. The three-dimensional reconstruction device for a hip joint based on a dual-plane X-ray film is described in detail below.
[0129] like Figure 6 As shown, the hip joint three-dimensional reconstruction device based on dual-plane X-ray film includes:
[0130] An image acquisition module 101 is used to acquire an anteroposterior image of the hip joint and a lateral image of the hip joint;
[0131] A downsampling module 102, which is used to input the hip joint anteroposterior image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively, and obtain a corresponding plurality of down-sampled feature maps;
[0132] A conversion and fusion module 103 is used to convert and fuse the downsampled feature map of the hip joint anteroposterior image and the downsampled feature map corresponding to the hip joint lateral image to obtain a corresponding plurality of three-dimensional feature maps;
[0133] An upsampling module 104 is used to input the multiple three-dimensional feature maps into an upsampling structure to obtain a three-dimensional reconstruction result of the hip joint; the three-dimensional reconstruction convolution model is trained based on a first sample anteroposterior image, a second sample lateral image and an annotated CT image, and the first sample anteroposterior image or the second sample lateral image is generated by the annotated CT image.
[0134] In a specific embodiment, a model training module is also included, which is used to:
[0135] Acquire a CT image and a corresponding first sample frontal image, or a second sample lateral image; generate a first sample frontal image based on the CT image, or generate a second sample lateral image based on the CT image; input the first sample frontal image and the second sample lateral image into a three-dimensional reconstruction convolution model to obtain a three-dimensional reconstruction prediction result; generate a three-dimensional reconstruction annotation result based on the CT image; calculate the overall loss of the three-dimensional reconstruction convolution model according to the three-dimensional reconstruction prediction result and the three-dimensional reconstruction annotation result; iterate the parameters of the three-dimensional reconstruction convolution model according to the overall loss until the loss converges.
[0136] In a specific implementation, the model training module is also used to:
[0137] Acquire a CT image and generate a three-dimensional point cloud based on the CT image; determine the projection line corresponding to the first sample positive image; for each projection line, calculate the path of the projection line passing through the three-dimensional point cloud, and generate the grayscale value of the projection line in the two-dimensional projection image; traverse all projection lines, and the grayscale value image obtained is the first sample positive image.
[0138] In a specific implementation, the downsampling module 102 is further configured to:
[0139] Attention and global feature extraction are performed on the anteroposterior image of the hip joint to obtain an anteroposterior extraction feature map; the anteroposterior extraction feature map is sequentially down-sampled to obtain the corresponding anteroposterior first down-sampling feature map, anteroposterior second down-sampling feature map, anteroposterior third down-sampling feature map, and anteroposterior fourth down-sampling feature map; attention and global feature extraction are performed on the lateral image of the hip joint to obtain a lateral extraction feature map; the lateral extraction feature map is sequentially down-sampled to obtain the corresponding lateral first down-sampling feature map, lateral second down-sampling feature map, lateral third down-sampling feature map, and lateral fourth down-sampling feature map.
[0140] In a specific implementation, the conversion and fusion module 103 is further used for:
[0141] The first sampling feature map of the front position is transformed and fused with the first sampling feature map of the side position to obtain the corresponding first three-dimensional feature map; the second sampling feature map of the front position is transformed and fused with the second sampling feature map of the side position to obtain the corresponding second three-dimensional feature map; the third sampling feature map of the front position is transformed and fused with the third sampling feature map of the side position to obtain the corresponding third three-dimensional feature map; the fourth sampling feature map of the front position is transformed and fused with the fourth sampling feature map of the side position to obtain the corresponding fourth three-dimensional feature map.
[0142] In a specific implementation, the conversion and fusion module 103 is further used for:
[0143] A two-dimensional and three-dimensional conversion is performed on a down-sampled feature map in the orthogonal position to obtain an orthogonal conversion feature map, and the size of the orthogonal conversion feature map is H×W×N; a two-dimensional and three-dimensional conversion is performed on a down-sampled feature map in the lateral position to obtain a lateral conversion feature map, and the size of the lateral conversion feature map is H×N×W; wherein the width of the orthogonal conversion feature map is the same as the depth of the lateral conversion feature map, and the width of the lateral conversion feature map is the same as the depth of the orthogonal conversion feature map; the lateral conversion feature map is azimuthally converted to obtain an azimuth conversion feature map, and the size of the azimuth conversion feature map is the same as that of the orthogonal conversion feature map; the azimuth conversion feature map is superimposed with the orthogonal conversion feature map to obtain the first three-dimensional feature map.
[0144] In a specific implementation, the conversion and fusion module 103 is further used for:
[0145] The downsampled feature map in the orthoposition is flattened to obtain a corresponding flattened feature map; the flattened feature map is fully connected, normalized and ReLU processed to obtain a normalized feature map; the normalized feature map is reshaped to obtain a three-dimensional reshaped feature map; the three-dimensional reshaped feature map is three-dimensionally stabilized to obtain the orthoposition conversion feature map.
[0146] The three-dimensional reconstruction device of the hip joint based on dual-plane X-ray films provided in the above-mentioned embodiment of the present application has a corresponding relationship with the three-dimensional reconstruction method of the hip joint based on dual-plane X-ray films provided in the embodiment of the present application. Therefore, the specific content in the device has a corresponding relationship with the three-dimensional reconstruction method of the hip joint. The specific content can be referred to the records in the three-dimensional reconstruction method of the hip joint, and will not be repeated in this application.
[0147] The three-dimensional reconstruction device of the hip joint based on dual-plane X-ray films provided in the above-mentioned embodiments of the present application and the three-dimensional reconstruction method of the hip joint based on dual-plane X-ray films provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0148] The above describes the internal functions and structure of the hip joint three-dimensional reconstruction device based on dual-plane X-ray films, such as Figure 7 As shown, in practice, the hip joint three-dimensional reconstruction device based on dual-plane X-rays can be realized as an electronic device, including: a memory 301 and a processor 303.
[0149] The memory 301 may be configured to store programs.
[0150] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.
[0151] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0152] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:
[0153] Obtain anteroposterior and lateral hip joint images;
[0154] Inputting the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively to obtain a corresponding plurality of down-sampled feature maps;
[0155] The down-sampled feature map of the hip joint frontal image and the down-sampled feature map corresponding to the hip joint lateral image are converted and fused to obtain corresponding multiple three-dimensional feature maps;
[0156] Inputting the multiple three-dimensional feature maps into an upsampling structure to obtain a three-dimensional reconstruction result of the hip joint;
[0157] The three-dimensional reconstruction convolution model is trained based on a first sample frontal image, a second sample lateral image and an annotated CT image, and the first sample frontal image or the second sample lateral image is generated by the annotated CT image.
[0158] In a specific implementation, the processor 303 is further configured to:
[0159] Acquire a CT image and a corresponding first sample frontal image, or a second sample lateral image; generate a first sample frontal image based on the CT image, or generate a second sample lateral image based on the CT image; input the first sample frontal image and the second sample lateral image into a three-dimensional reconstruction convolution model to obtain a three-dimensional reconstruction prediction result; generate a three-dimensional reconstruction annotation result based on the CT image; calculate the overall loss of the three-dimensional reconstruction convolution model according to the three-dimensional reconstruction prediction result and the three-dimensional reconstruction annotation result; iterate the parameters of the three-dimensional reconstruction convolution model according to the overall loss until the loss converges.
[0160] In a specific implementation, the processor 303 is further configured to:
[0161] Acquire a CT image and generate a three-dimensional point cloud based on the CT image; determine the projection line corresponding to the first sample positive image; for each projection line, calculate the path of the projection line passing through the three-dimensional point cloud, and generate the grayscale value of the projection line in the two-dimensional projection image; traverse all projection lines, and the grayscale value image obtained is the first sample positive image.
[0162] In a specific implementation, the processor 303 is further configured to:
[0163] Attention and global feature extraction are performed on the anteroposterior image of the hip joint to obtain an anteroposterior extraction feature map; the anteroposterior extraction feature map is sequentially down-sampled to obtain the corresponding anteroposterior first down-sampling feature map, anteroposterior second down-sampling feature map, anteroposterior third down-sampling feature map, and anteroposterior fourth down-sampling feature map; attention and global feature extraction are performed on the lateral image of the hip joint to obtain a lateral extraction feature map; the lateral extraction feature map is sequentially down-sampled to obtain the corresponding lateral first down-sampling feature map, lateral second down-sampling feature map, lateral third down-sampling feature map, and lateral fourth down-sampling feature map.
[0164] In a specific implementation, the processor 303 is further configured to:
[0165] The first sampling feature map of the front position is transformed and fused with the first sampling feature map of the side position to obtain the corresponding first three-dimensional feature map; the second sampling feature map of the front position is transformed and fused with the second sampling feature map of the side position to obtain the corresponding second three-dimensional feature map; the third sampling feature map of the front position is transformed and fused with the third sampling feature map of the side position to obtain the corresponding third three-dimensional feature map; the fourth sampling feature map of the front position is transformed and fused with the fourth sampling feature map of the side position to obtain the corresponding fourth three-dimensional feature map.
[0166] In a specific implementation, the processor 303 is further configured to:
[0167] A two-dimensional and three-dimensional conversion is performed on a down-sampled feature map in the orthogonal position to obtain an orthogonal conversion feature map, and the size of the orthogonal conversion feature map is H×W×N; a two-dimensional and three-dimensional conversion is performed on a down-sampled feature map in the lateral position to obtain a lateral conversion feature map, and the size of the lateral conversion feature map is H×N×W; wherein the width of the orthogonal conversion feature map is the same as the depth of the lateral conversion feature map, and the width of the lateral conversion feature map is the same as the depth of the orthogonal conversion feature map; the lateral conversion feature map is azimuthally converted to obtain an azimuth conversion feature map, and the size of the azimuth conversion feature map is the same as that of the orthogonal conversion feature map; the azimuth conversion feature map is superimposed with the orthogonal conversion feature map to obtain the first three-dimensional feature map.
[0168] In a specific implementation, the processor 303 is further configured to:
[0169] The downsampled feature map in the orthoposition is flattened to obtain a corresponding flattened feature map; the flattened feature map is fully connected, normalized and ReLU processed to obtain a normalized feature map; the normalized feature map is reshaped to obtain a three-dimensional reshaped feature map; the three-dimensional reshaped feature map is three-dimensionally stabilized to obtain the orthoposition conversion feature map.
[0170] In the present application, the processor is also specifically used to execute all processes and steps of the above-mentioned three-dimensional reconstruction method of the hip joint based on dual-plane X-rays. The specific content can refer to the records in the three-dimensional reconstruction method of the hip joint, and will not be repeated in this application.
[0171] In this application, Figure 7 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 7 Components shown.
[0172] The electronic device provided in this embodiment is based on the same inventive concept as the method for three-dimensional reconstruction of the hip joint based on dual-plane X-rays provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored in it.
[0173] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0174] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0175] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0177] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0178] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0179] The present application also provides a computer-readable storage medium corresponding to the three-dimensional reconstruction method of the hip joint based on dual-plane X-ray films provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the three-dimensional reconstruction method of the hip joint based on dual-plane X-ray films provided in any of the aforementioned embodiments.
[0180] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0181] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the hip joint three-dimensional reconstruction method based on dual-plane X-rays provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0182] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.
[0183] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0184] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A method for three-dimensional reconstruction of hip joint based on dual-plane X-ray film, characterized in that: include: Obtain anteroposterior and lateral hip joint images; Inputting the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively to obtain a corresponding plurality of down-sampled feature maps; The down-sampled feature map of the hip joint frontal image and the down-sampled feature map corresponding to the hip joint lateral image are converted and fused to obtain corresponding multiple three-dimensional feature maps; Inputting the multiple three-dimensional feature maps into an upsampling structure to obtain a three-dimensional reconstruction result of the hip joint; The three-dimensional reconstruction convolution model is trained based on a first sample frontal image, a second sample lateral image and an annotated CT image, and the first sample frontal image or the second sample lateral image is generated by the annotated CT image.
2. The method for three-dimensional reconstruction of hip joint based on dual-plane X-ray film according to claim 1, characterized in that: The training process of the 3D reconstruction convolutional model includes: Acquire a CT image and a corresponding first sample frontal image or a second sample lateral image; Generate a first sample frontal image based on the CT image, or generate a second sample lateral image based on the CT image; Inputting the first sample frontal image and the second sample lateral image into a three-dimensional reconstruction convolution model to obtain a three-dimensional reconstruction prediction result; Generate 3D reconstruction annotation results based on CT images; Calculate the overall loss of the 3D reconstruction convolutional model based on the 3D reconstruction prediction results and the 3D reconstruction annotation results; The parameters of the 3D reconstruction convolution model are iterated according to the overall loss until the loss converges.
3. The method for three-dimensional reconstruction of hip joint based on dual-plane X-ray film according to claim 2, characterized in that: The step of generating a first sample normal image based on the CT image comprises: Acquire CT images, and generate three-dimensional point clouds based on the CT images; Determine a projection line corresponding to the first sample positive image; For each projection line, calculate the path of the projection line passing through the three-dimensional point cloud, and generate the grayscale value of the projection line in the two-dimensional projection image; The grayscale image obtained by traversing all projection lines is the first sample positive image.
4. The method for three-dimensional reconstruction of hip joint based on biplane X-ray films according to any one of claims 1 to 3, characterized in that: The method of inputting the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively to obtain a corresponding plurality of down-sampled feature maps includes: Perform attention and global feature extraction on the anteroposterior image of the hip joint to obtain an anteroposterior extraction feature map; The feature map extracted from the positive position is sequentially downsampled to obtain a corresponding positive position one-downsampled feature map, a positive position two-downsampled feature map, a positive position three-downsampled feature map, and a positive position four-downsampled feature map; Perform attention and global feature extraction on the lateral image of the hip joint to obtain a lateral extraction feature map; The side position extraction feature maps are downsampled in sequence to obtain corresponding side position one downsampled feature map, side position two downsampled feature map, side position three downsampled feature map, and side position four downsampled feature map.
5. The method for three-dimensional reconstruction of hip joint based on dual-plane X-ray films according to claim 4, characterized in that: The down-sampled feature map of the hip joint frontal image and the down-sampled feature map corresponding to the hip joint lateral image are converted and fused to obtain corresponding multiple three-dimensional feature maps, including: The down-sampling feature map in the frontal position and the down-sampling feature map in the lateral position are transformed and fused to obtain a corresponding first three-dimensional feature map; The frontal two-downsampled feature map and the lateral two-downsampled feature map are transformed and fused to obtain a corresponding second three-dimensional feature map; The three-downsampling feature map of the front position and the three-downsampling feature map of the lateral position are transformed and fused to obtain the corresponding third three-dimensional feature map; The frontal four-downsampling feature map and the lateral four-downsampling feature map are transformed and fused to obtain the corresponding fourth three-dimensional feature map.
6. The method for three-dimensional reconstruction of hip joint based on dual-plane X-ray films according to claim 5, characterized in that: The converting and fusing the down-sampling feature map in the frontal position and the down-sampling feature map in the lateral position to obtain the corresponding first three-dimensional feature map includes: Perform a two-dimensional to three-dimensional transformation on the normal-position downsampling feature map to obtain a normal-position transformation feature map, wherein the size of the normal-position transformation feature map is H×W×N; Performing a two-dimensional to three-dimensional conversion on a downsampled feature map of the lateral position to obtain a lateral position conversion feature map, wherein the size of the lateral position conversion feature map is H×N×W; wherein the width of the frontal position conversion feature map is the same as the depth of the lateral position conversion feature map, and the width of the lateral position conversion feature map is the same as the depth of the frontal position conversion feature map; Performing azimuth transformation on the lateral position transformation characteristic map to obtain an azimuth transformation characteristic map, wherein the azimuth transformation characteristic map has the same size as the frontal position transformation characteristic map; The azimuth conversion feature map is superimposed with the orthographic conversion feature map to obtain the first three-dimensional feature map.
7. The method for three-dimensional reconstruction of hip joint based on dual-plane X-ray films according to claim 6, characterized in that: The two-dimensional to three-dimensional conversion of the orthogonal down-sampling feature map to obtain the orthogonal conversion feature map includes: Flatten the downsampled feature map in the normal position to obtain a corresponding flattened feature map; Perform full connection, normalization and ReLU processing on the flattened feature map to obtain a normalized feature map; Reshape the normalized feature map to obtain a three-dimensional reshaped feature map; The three-dimensional reconstruction feature map is subjected to three-dimensional stabilization processing to obtain the orthotopic transformation feature map.
8. A hip joint three-dimensional reconstruction device based on dual-plane X-ray film, characterized in that: include: An image acquisition module, which is used to acquire an anteroposterior image of the hip joint and a lateral image of the hip joint; A downsampling module, which is used to input the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively, and obtain a corresponding plurality of down-sampled feature maps; A conversion and fusion module is used to convert and fuse the downsampled feature map of the hip joint frontal image and the downsampled feature map corresponding to the hip joint lateral image to obtain a corresponding plurality of three-dimensional feature maps; An upsampling module, which is used to input the multiple three-dimensional feature maps into an upsampling structure to obtain a three-dimensional reconstruction result of the hip joint; The three-dimensional reconstruction convolution model is trained based on a first sample frontal image, a second sample lateral image and an annotated CT image, and the first sample frontal image or the second sample lateral image is generated by the annotated CT image.
9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Obtain anteroposterior and lateral hip joint images; Inputting the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively to obtain a corresponding plurality of down-sampled feature maps; The down-sampled feature map of the hip joint frontal image and the down-sampled feature map corresponding to the hip joint lateral image are converted and fused to obtain corresponding multiple three-dimensional feature maps; Inputting the multiple three-dimensional feature maps into an upsampling structure to obtain a three-dimensional reconstruction result of the hip joint; The three-dimensional reconstruction convolution model is trained based on a first sample frontal image, a second sample lateral image and an annotated CT image, and the first sample frontal image or the second sample lateral image is generated by the annotated CT image.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the three-dimensional reconstruction method of hip joint based on dual-plane X-ray films as described in any one of claims 1-7.
Citation Information
Patent Citations
Hip joint image processing method based on deep learning and computing equipment
CN111179350A
Attention feature map acquisition method and device and target detection method and device
CN112101456A
Three-dimensional image reconstruction method and device, equipment and storage medium
CN113554742A
Medical image three-dimensional reconstruction method, medium and electronic equipment
CN113744387A
Edge width extraction method based on series-parallel network feature fusion
CN116129143A
