A method and device for three-dimensional reconstruction of hip joint based on double plane X-ray film

Abstract

The application provides a hip joint three-dimensional reconstruction method and device based on double-plane X-ray films, and the method comprises the following steps: acquiring a hip joint frontal image and a hip joint lateral image; inputting a first processing structure of a three-dimensional reconstruction convolution model to obtain a plurality of corresponding down-sampling feature maps; converting and fusing the down-sampling feature map of the frontal image and the down-sampling feature map of the lateral image to obtain a plurality of corresponding three-dimensional feature maps; inputting the plurality of three-dimensional feature maps into an up-sampling structure to obtain a three-dimensional reconstruction result of the hip joint; the three-dimensional reconstruction convolution model is obtained by training based on a first sample frontal image, a second sample lateral image and a labeled CT image, and the first sample frontal image or the second sample lateral image is generated from the labeled CT image. In the application, the two-dimensional down-sampling and three-dimensional up-sampling mode is used to increase the accuracy of reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a hip joint three-dimensional reconstruction method and device based on double-plane X-ray films. BACKGROUND

[0002] 3D reconstruction of the hip joint is an important step for various clinical applications. It can be used for surgical planning, precise implant selection, patient-specific implant manufacturing or intraoperative jig printing, perfect fitting of anatomical structure. X-ray films are usually used because of their wide availability, lower price, short scanning time and lower ionization level, and smaller radiation compared with CT scanners.

[0003] Therefore, it has high value to replace CT with X-ray films to generate hip joint three-dimensional reconstruction results, but the accuracy of the three-dimensional reconstruction results generated by the X-ray films is not high at present. SUMMARY

[0004] The problem solved by the present application is that the accuracy of the three-dimensional reconstruction results generated by the X-ray films is not high at present.

[0005] To solve the above problems, the first aspect of the present application provides a hip joint three-dimensional reconstruction method based on double-plane X-ray films, comprising:

[0006] obtaining a hip joint frontal image and a hip joint lateral image;

[0007] respectively inputting the hip joint frontal image and the hip joint lateral image into a first processing structure of a three-dimensional reconstruction convolution model to obtain a plurality of corresponding down-sampling feature maps;

[0008] transforming and fusing the down-sampling feature map of the hip joint frontal image and the corresponding down-sampling feature map of the hip joint lateral image to obtain a plurality of corresponding three-dimensional feature maps;

[0009] inputting the plurality of three-dimensional feature maps into an up-sampling structure to obtain a three-dimensional reconstruction result of the hip joint;

[0010] The three-dimensional reconstruction convolution model is trained based on a first sample frontal image and a second sample lateral image and a labeled CT image, and the first sample frontal image or the second sample lateral image is generated from the labeled CT image.

[0011] The second aspect of the present application provides a hip joint three-dimensional reconstruction device based on double-plane X-ray films, comprising:

[0012] an image acquisition module for acquiring a hip joint frontal image and a hip joint lateral image;

[0013] a downsampling module, configured to input the frontal view hip joint image and the lateral view hip joint image into a first processing structure of a three-dimensional reconstruction convolution model respectively, to obtain a plurality of corresponding downsampled feature maps;

[0014] a conversion fusion module, configured to perform conversion fusion on the downsampled feature map of the frontal view hip joint image and the downsampled feature map corresponding to the lateral view hip joint image, to obtain a plurality of corresponding three-dimensional feature maps;

[0015] an upsampling module, configured to input the plurality of three-dimensional feature maps into an upsampling structure, to obtain a three-dimensional reconstruction result of the hip joint;

[0016] The three-dimensional reconstruction convolution model is trained based on a first sample frontal view image, a second sample lateral view image, and a labeled CT image, and the first sample frontal view image or the second sample lateral view image is generated from the labeled CT image.

[0017] The third aspect of the present application provides an electronic device, comprising a memory and a processor;

[0018] The memory is configured to store a program;

[0019] The processor is coupled to the memory and configured to execute the program, so as to:

[0020] obtain a frontal view hip joint image and a lateral view hip joint image;

[0021] input the frontal view hip joint image and the lateral view hip joint image into a first processing structure of a three-dimensional reconstruction convolution model respectively, to obtain a plurality of corresponding downsampled feature maps;

[0022] perform conversion fusion on the downsampled feature map of the frontal view hip joint image and the downsampled feature map corresponding to the lateral view hip joint image, to obtain a plurality of corresponding three-dimensional feature maps;

[0023] input the plurality of three-dimensional feature maps into an upsampling structure, to obtain a three-dimensional reconstruction result of the hip joint;

[0024] The three-dimensional reconstruction convolution model is trained based on a first sample frontal view image, a second sample lateral view image, and a labeled CT image, and the first sample frontal view image or the second sample lateral view image is generated from the labeled CT image.

[0025] The fourth aspect of the present application provides a computer readable storage medium, having a computer program stored thereon, wherein the program is executed by a processor to implement the three-dimensional reconstruction method of a hip joint based on double-plane X-ray films.

[0026] In the present application, the corresponding down-sampled images are converted and fused to obtain a three-dimensional image by down-sampling the frontal and lateral two-dimensional images of the hip joint, and the three-dimensional image is up-sampled to obtain the final three-dimensional reconstruction result, thereby greatly increasing the accuracy of reconstruction by using two-dimensional down-sampling and three-dimensional up-sampling.

[0027] In the present application, the trained frontal hip joint image or lateral hip joint image is generated from the CT image, thereby making up for the current lack of training data and improving the accuracy of training the three-dimensional reconstruction convolution model. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The architecture diagram of the three-dimensional reconstruction method of the hip joint according to the embodiment of the present application is shown.

[0029] Figure 2 The flowchart of the three-dimensional reconstruction method of the hip joint according to the embodiment of the present application is shown.

[0030] Figure 3 The flowchart of the model training of the three-dimensional reconstruction method of the hip joint according to the embodiment of the present application is shown.

[0031] Figure 4 The architecture diagram of the conversion and fusion module of the three-dimensional reconstruction method of the hip joint according to the embodiment of the present application is shown.

[0032] Figure 5 The architecture diagram of the two-dimensional to three-dimensional conversion of the three-dimensional reconstruction method of the hip joint according to the embodiment of the present application is shown.

[0033] Figure 6 The structure block diagram of the three-dimensional reconstruction device of the hip joint according to the embodiment of the present application is shown.

[0034] Figure 7 The structure block diagram of the electronic device according to the embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0036] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as the usual meaning understood by those skilled in the art to which the present application belongs.

[0037] To solve the above problems, the application provides a new hip joint three-dimensional reconstruction scheme based on double-plane X-ray films. By down-sampling the front and lateral two-dimensional images respectively and up-sampling the three-dimensional image obtained by conversion and fusion, the problem of low accuracy of three-dimensional reconstruction results generated by current X-ray films is solved.

[0038] The application embodiment provides a hip joint three-dimensional reconstruction method based on double-plane X-ray films. The specific scheme of the method is shown in Figures 1-5 The method can be executed by a hip joint three-dimensional reconstruction device based on double-plane X-ray films, which can be integrated in a computer, a server, a computer cluster, a data center, or other electronic devices. As shown in Figure 1 , Figure 2 The application provides a hip joint three-dimensional reconstruction method based on double-plane X-ray films. The specific scheme of the method is shown in the flowchart of the hip joint three-dimensional reconstruction method based on double-plane X-ray films according to an embodiment of the application. The hip joint three-dimensional reconstruction method based on double-plane X-ray films includes:

[0039] S101, acquiring a hip joint front image and a hip joint lateral image;

[0040] The hip joint front image and the hip joint lateral image are X-ray films containing the hip joint part of any object.

[0041] The hip joint front image and the hip joint lateral image are front and lateral images of the same object in the same pose, so that a three-dimensional reconstruction result in the position state is generated based on the images.

[0042] In the application, the hip joint front image is an X-ray film taken from the front, and the lateral image is an X-ray film taken from the left side of the object. In this way, the input data of the three-dimensional reconstruction convolution model is limited, and the hip joint front image and the hip joint lateral image, as well as the first sample front image and the second sample lateral image, are limited and unified to improve the accuracy during training and actual reconstruction.

[0043] S102, respectively inputting the hip joint front image and the hip joint lateral image into a first processing structure of a three-dimensional reconstruction convolution model to obtain a plurality of corresponding down-sampling feature maps;

[0044] In the application, the first processing structure includes a preprocessing part and a down-sampling part. By sequentially down-sampling, a plurality of corresponding down-sampling feature maps are obtained.

[0045] In the application, the first processing structure is two (it can also be understood that the first processing structure has two branches with the same structure, and the branches are independent of each other). The hip joint front image and the hip joint lateral image are processed respectively, and a plurality of corresponding down-sampling feature maps are output respectively. The structures of the two first processing structures are the same, but the specific parameters are different.

[0046] S103, convert and fuse the down-sampling feature maps of the hip joint frontal view image and the down-sampling feature maps corresponding to the hip joint lateral view image to obtain a plurality of three-dimensional feature maps;

[0047] In the present application, the down-sampling feature maps of the hip joint frontal view image of the same order and the down-sampling feature maps corresponding to the hip joint lateral view image of the same order are converted and fused to obtain three-dimensional feature maps of the same order.

[0048] In the present application, the conversion and fusion structure is composed of four identical structures, which are respectively used to process the conversion and fusion of the down-sampling feature maps of the hip joint frontal view image and the down-sampling feature maps corresponding to the hip joint lateral view image of different orders; the specific structures are the same, but the parameters are different.

[0049] In the present application, the sizes of the feature maps of different orders are different.

[0050] In the present application, the height of the hip joint frontal view image and the hip joint lateral view image is the same, but the width can be different.

[0051] S104, input the plurality of three-dimensional feature maps into an up-sampling structure to obtain a three-dimensional reconstruction result of the hip joint; the three-dimensional reconstruction convolutional model is obtained by training based on a first sample frontal view image and a second sample lateral view image and a labeled CT image, and the first sample frontal view image or the second sample lateral view image is generated from the labeled CT image.

[0052] In the present application, the three-dimensional feature maps obtained by conversion and fusion are added to the up-sampling structure as long connections, so as to realize the superposition and highlighting of the converted and fused features, and further improve the accuracy of three-dimensional reconstruction.

[0053] In the present application, the hip joint frontal view image and the hip joint lateral view image are respectively down-sampled, the corresponding down-sampled images are converted and fused to obtain a three-dimensional image, and the three-dimensional image is respectively up-sampled to obtain a final three-dimensional reconstruction result, so as to utilize the two-dimensional down-sampling and three-dimensional up-sampling manner to greatly increase the accuracy of reconstruction.

[0054] In the present application, the training hip joint frontal view image or hip joint lateral view image is generated from the CT image, so as to make up for the current insufficient training data, and improve the accuracy of training of the three-dimensional reconstruction convolutional model.

[0055] In one embodiment, the three-dimensional reconstruction convolutional model is trained by using a CT image and a first sample frontal view image corresponding to the CT image, or a second sample lateral view image corresponding to the CT image. Figure 3 As shown in the figure, the training process of the three-dimensional reconstruction convolutional model includes:

[0056] S201, obtaining a CT image and a corresponding first sample frontal view image, or a second sample lateral view image;

[0057] In this step, the CT image of the hip joint of any object and at least one X-ray film of the hip joint are obtained, which can be an anteroposterior image or a lateral image.

[0058] It should be noted that for training of the three-dimensional reconstruction convolution model, the training data needs to include the anteroposterior X-ray film and the lateral X-ray film under the same pose, as well as the CT image; however, in the actual acquisition process, it is found that only a small amount of complete data can be obtained, and most of the data only include one X-ray film (anteroposterior or lateral).

[0059] Therefore, in this step, the CT image and the corresponding first sample anteroposterior image or the second sample lateral image are obtained, which specifically means that the CT image and the corresponding first sample anteroposterior image and the second sample lateral image are obtained, or the CT image and the corresponding first sample anteroposterior image are obtained, or the CT image and the corresponding second sample lateral image are obtained.

[0060] S202, generating a first sample anteroposterior image based on the CT image, or generating a second sample lateral image based on the CT image;

[0061] In this step, if the CT image has the corresponding first sample anteroposterior image and the second sample lateral image, this step does not need to be performed; if the CT image only has the corresponding first sample anteroposterior image, the second sample lateral image needs to be generated based on the CT image; if the CT image only has the corresponding second sample lateral image, the first sample anteroposterior image needs to be generated based on the CT image.

[0062] S203, inputting the first sample anteroposterior image and the second sample lateral image into the three-dimensional reconstruction convolution model to obtain a three-dimensional reconstruction prediction result;

[0063] S204, generating a three-dimensional reconstruction annotation result based on the CT image;

[0064] In this step, the three-dimensional reconstruction result obtained by the CT image is the three-dimensional reconstruction annotation result; the specific way of three-dimensional reconstruction of the CT image can be to load the CT image in DICOM format, and then perform three-dimensional reconstruction through surface rendering (contour extraction-three-dimensional surface generation-surface smoothing processing), or through volume rendering, or directly obtain the reconstruction result by using professional software such as 3D Slicer; the specific way is not limited in the present application.

[0065] S205, calculating the overall loss of the three-dimensional reconstruction convolution model according to the three-dimensional reconstruction prediction result and the three-dimensional reconstruction annotation result;

[0066] In the present application, the overall loss includes a reconstruction loss and a CE loss, and the two losses are associated through a weight coefficient.

[0067] where the reconstruction loss is mean square error loss.

[0068] S206, iteratively adjusting parameters of the three-dimensional reconstruction convolutional model according to the overall loss until the loss converges.

[0069] In the present application, the X-ray film is generated by CT image projection, thereby making up for the problem of insufficient training data and improving the accuracy of training.

[0070] In one embodiment, the first sample frontal image is generated based on the CT image, comprising:

[0071] Obtaining a CT image, and generating a three-dimensional point cloud based on the CT image;

[0072] Determining a projection line corresponding to the first sample frontal image;

[0073] For each projection line, calculating a path of the projection line through the three-dimensional point cloud, and generating a gray value of the projection line in the two-dimensional projection image;

[0074] Traversing all the projection lines to obtain a gray value image, which is the first sample frontal image.

[0075] In the present application, each pixel value (referred to as Hounsfield unit, HU value) of the CT image represents the X-ray attenuation coefficient at that position. The CT image is a three-dimensional volume data containing multiple slices; based on this, a three-dimensional point cloud can be generated, each point having a pixel value.

[0076] In the present application, the attenuation value of each ray is integrated by calculating the path of the ray through the three-dimensional CT data. This process simulates the physical process of X-rays passing through an object and calculates the attenuation image in the two-dimensional projection plane. Specifically, for each projection line, the attenuation values of all voxels (each small volume unit in the CT data) along the line are accumulated to generate the gray value of the point in the two-dimensional projection image.

[0077] In the present application, by the above calculation, the projections at different angles are superimposed to generate a two-dimensional projection image similar to a traditional X-ray. This image is very close to the effect of actual X-ray shooting. That is, the first sample frontal image.

[0078] In one embodiment, the three-dimensional reconstruction convolutional model is trained based on the first sample frontal image and the first sample lateral image. Figure 1 As shown in the figure, the first processing structure for respectively inputting the frontal image of the hip joint and the lateral image of the hip joint into the three-dimensional reconstruction convolutional model comprises:

[0079] The attention and global feature extraction is performed on the frontal image of the hip joint to obtain a frontal extracted feature map;

[0080] The positive position extraction feature map is sequentially down-sampled to obtain a corresponding positive position first down-sampling feature map, a positive position second down-sampling feature map, a positive position third down-sampling feature map, and a positive position fourth down-sampling feature map.

[0081] The hip joint lateral image is subjected to attention and global feature extraction to obtain a lateral extraction feature map.

[0082] The lateral extraction feature map is sequentially down-sampled to obtain a corresponding lateral first down-sampling feature map, a lateral second down-sampling feature map, a lateral third down-sampling feature map, and a lateral fourth down-sampling feature map.

[0083] In the present application, the attention extraction (CBM) can be spatial attention extraction and / or channel attention extraction; the corresponding spatial features and channel features are strengthened through attention extraction.

[0084] In the present application, through the setting of attention extraction-global extraction, better global feature strengthening is achieved.

[0085] The process of global extraction is as follows:

[0086] The class attention output feature map is used as the input feature map of global extraction.

[0087] The input feature map is input into a 1x1 convolution layer arranged in parallel to obtain a first extraction map, a second extraction map, and a third extraction map; wherein the first extraction map and the third extraction map are in the format of HWxN, and the second extraction map is in the format of NxHW.

[0088] The first extraction map and the second extraction map are multiplied to obtain a multiplication feature map.

[0089] The multiplication feature map is subjected to softmax processing to obtain a multiplication coefficient.

[0090] The multiplication coefficient is multiplied with the third extraction map to obtain a coefficient multiplication map.

[0091] The coefficient multiplication map and the input feature map are added to obtain an output feature map of global extraction.

[0092] In the present application, the input feature map is divided into three branches for 1x1 convolution processing; through the convolution, the dimensions of the first branch and the third branch are adjusted to be in the format of HWxN, and the dimension of the second branch is adjusted to be in the format of NxHW; then the first branch and the third branch are multiplied to obtain a feature map with a dimension of HWxHW, and a coefficient is obtained through softmax processing; the coefficient is multiplied with the third branch, and the multiplied feature map is added with the input feature map to obtain an output feature map.

[0093] In the present application, the dimensions of the first branch and the second branch are reversed by 1*1 convolution to realize the multiplication between the feature maps; through the multiplication, the long-range dependence relationship is captured; through the setting of the coefficient, the dependence relationship is embedded in the feature map; and on this basis, the original input is added, and the residual connection is used to embed the global extraction in the model without destroying the parameters.

[0094] In the present application, the long-range dependence relationship between the features is captured by global extraction, so that the model can extract global features to make up for the defect that the current model mainly extracts local features.

[0095] In the present application, the response at a position is calculated as the weighted sum of the features at all positions, so that when processing the information of each position, all position information that can be considered is considered.

[0096] In one embodiment, in combination with Figure 1 As shown in the figure, the down-sampled feature map of the frontal view image of the hip joint and the down-sampled feature map corresponding to the lateral view image of the hip joint are converted and fused to obtain a plurality of corresponding three-dimensional feature maps, including:

[0097] The down-sampled feature map of the frontal view and the down-sampled feature map of the lateral view are converted and fused to obtain a corresponding first three-dimensional feature map;

[0098] The down-sampled feature map of the frontal view and the down-sampled feature map of the lateral view are converted and fused to obtain a corresponding first three-dimensional feature map;

[0099] The down-sampled feature map of the frontal view and the down-sampled feature map of the lateral view are converted and fused to obtain a corresponding first three-dimensional feature map;

[0100] The down-sampled feature map of the frontal view and the down-sampled feature map of the lateral view are converted and fused to obtain a corresponding first three-dimensional feature map.

[0101] Among them, the size of the feature map and the three-dimensional feature map of the first, second, third and fourth order decreases in turn.

[0102] In the present application, after obtaining the first three-dimensional feature map, the second three-dimensional feature map, the third three-dimensional feature map and the fourth three-dimensional feature map, they are input into the up-sampling structure to obtain the three-dimensional reconstruction result of the hip joint, specifically:

[0103] The fourth three-dimensional feature map is set as the basic feature map of up-sampling, and is up-sampled to obtain a third up-sampling three-dimensional map;

[0104] The third up-sampling three-dimensional map and the third three-dimensional feature map are merged and then up-sampled to obtain a second up-sampling three-dimensional map;

[0105] After the second up-sampling three-dimensional image is merged with the second three-dimensional feature map, up-sampling is performed to obtain a first up-sampling three-dimensional image;

[0106] After the first up-sampling three-dimensional image is merged with the first three-dimensional feature map, convolution processing is performed to obtain a three-dimensional reconstruction result of the hip joint.

[0107] In the present application, the corresponding three-dimensional reconstruction result is obtained by up-sampling the three-dimensional image.

[0108] In one embodiment, the present application is combined with Figure 4 As shown in the figure, the conversion and fusion of the frontal down-sampling feature map and the lateral down-sampling feature map to obtain the corresponding first three-dimensional feature map comprises:

[0109] The two-dimensional three-dimensional conversion is performed on the frontal down-sampling feature map to obtain a frontal conversion feature map, and the size of the frontal conversion feature map is HxWxN;

[0110] The two-dimensional three-dimensional conversion is performed on the lateral down-sampling feature map to obtain a lateral conversion feature map, and the size of the lateral conversion feature map is HxNxW; wherein the width of the frontal conversion feature map is the same as the depth of the lateral conversion feature map, and the width of the lateral conversion feature map is the same as the depth of the frontal conversion feature map;

[0111] The orientation conversion is performed on the lateral conversion feature map to obtain an orientation conversion feature map, and the size of the orientation conversion feature map is the same as that of the frontal conversion feature map;

[0112] The orientation conversion feature map and the frontal conversion feature map are superimposed to obtain the first three-dimensional feature map.

[0113] In the present application, the conversion and fusion (COF) is described in detail by taking the conversion and fusion of the frontal down-sampling feature map and the lateral down-sampling feature map as an example, wherein the processing process can also be applied to the frontal down-sampling feature map, the second down-sampling feature map, and the third down-sampling feature map.

[0114] In the present application, it is assumed that the size of the frontal down-sampling feature map is HxA, and the size of the lateral down-sampling feature map is HxB, then the size of the generated frontal conversion feature map is HxAxB, and the size of the generated lateral conversion feature map is HxBxA; at this time, the sizes of the two conversion feature maps are different, and the lateral conversion feature map needs to be adjusted to adjust its size to HxAxB; the size adjustment can be matrix transformation, which exchanges the width and depth of the image (or can be considered as rotating 90 degrees around the height axis) by applying a rotation matrix to the coordinate system of the image.

[0115] In this application, the feature map overlay is a process of adding the gray values ​​of corresponding coordinates, thereby combining the feature map in the frontal position and the feature map in the lateral position to enhance the corresponding features.

[0116] In one implementation method, combining Figure 5 As shown, the step of performing a two-dimensional to three-dimensional transformation on the orthogonal subsampled feature map to obtain the orthogonal transformed feature map includes:

[0117] Flatten the subsampled feature map in the positive position to obtain the corresponding flattened feature map;

[0118] The flattened feature map is processed by fully connected processing, normalization processing, and ReLU processing to obtain a normalized feature map.

[0119] The normalized feature map is reshaped to obtain a three-dimensional reshaped feature map.

[0120] The three-dimensional reconstructed feature map is subjected to three-dimensional stabilization processing to obtain the positive position transformation feature map.

[0121] Among them, such as Figure 5 As shown, flattening flattens multidimensional convolutional feature maps into one-dimensional vectors; FC stands for fully connected processing; IN stands for normalization; Reshape transforms tensors from one shape to another without changing the order of the data; and Basic3D Block can be used to process the reshaped 3D image by multiple 3D convolutions (set sequentially) to improve the overall stability of the network.

[0122] In one specific implementation, the softmax processing is a SoftMax-like operation, and its activation function is:

[0123]

[0124] Where SoftMax1(x) is the activation function, x i x j Let i and j be the elements in the input vector, where i and j are the element indices.

[0125] In this application, SoftMax is a mathematical function typically used to convert a set of arbitrary real numbers into real numbers representing a probability distribution. Essentially, it is a normalization function that transforms a set of arbitrary real values ​​into probability values ​​between [0,1]. Because SoftMax converts them to values ​​between 0 and 1, they can be interpreted as probabilities. If one of the inputs is small or negative, SoftMax transforms it into a low probability; if the input is large, it transforms it into a high probability, but it will always remain between 0 and 1.

[0126] However, for the standard SoftMax function, since the input is mapped to 0 to 1, and the sum of all output values is 1, it means that even if some input values are very small, they will have a non-zero output value after the SoftMax function processing. This will also cause the noise to be amplified, resulting in the final output result being affected by more noise.

[0127] In this application, a 1 is added in the denominator of the SoftMax function; this change means that when the input value is very small, its output value can be closer to zero. This allows the corresponding output to tend to zero when there is no valuable information to add, thus greatly reducing unnecessary noise.

[0128] The embodiment of the present application provides a hip joint three-dimensional reconstruction device based on double plane X-ray films, which is used for executing the hip joint three-dimensional reconstruction method based on double plane X-ray films described in the foregoing of the present application. The hip joint three-dimensional reconstruction device based on double plane X-ray films is described in detail as follows.

[0129] As shown in Figure 6 The hip joint three-dimensional reconstruction device based on double plane X-ray films comprises:

[0130] An image acquisition module 101 is configured to acquire a hip joint frontal image and a hip joint lateral image.

[0131] A down-sampling module 102 is configured to input the hip joint frontal image and the hip joint lateral image into a first processing structure of a three-dimensional reconstruction convolution model respectively to obtain a plurality of corresponding down-sampled feature maps.

[0132] A conversion fusion module 103 is configured to convert and fuse the down-sampled feature map of the hip joint frontal image and the corresponding down-sampled feature map of the hip joint lateral image to obtain a plurality of corresponding three-dimensional feature maps.

[0133] An up-sampling module 104 is configured to input the plurality of three-dimensional feature maps into an up-sampling structure to obtain a three-dimensional reconstruction result of the hip joint. The three-dimensional reconstruction convolution model is trained based on a first sample frontal image and a second sample lateral image and a labeled CT image. The first sample frontal image or the second sample lateral image is generated from the labeled CT image.

[0134] In a specific embodiment, the device further comprises a model training module configured to:

[0135] obtaining a CT image and a corresponding first sample frontal image, or a second sample lateral image; generating the first sample frontal image based on the CT image, or generating the second sample lateral image based on the CT image; inputting the first sample frontal image and the second sample lateral image into a three-dimensional reconstruction convolution model to obtain a three-dimensional reconstruction prediction result; generating a three-dimensional reconstruction annotation result based on the CT image; calculating an overall loss of the three-dimensional reconstruction convolution model according to the three-dimensional reconstruction prediction result and the three-dimensional reconstruction annotation result; and iterating parameters of the three-dimensional reconstruction convolution model according to the overall loss until the loss converges.

[0136] In a specific embodiment, the model training module is further configured to:

[0137] obtaining a CT image, generating a three-dimensional point cloud based on the CT image; determining a projection line corresponding to a first sample frontal image; for each projection line, calculating a path of the projection line passing through the three-dimensional point cloud, and generating a gray value of the projection line in a two-dimensional projection image; and traversing all the projection lines to obtain a gray value image, which is the first sample frontal image.

[0138] In a specific embodiment, the down-sampling module 102 is further configured to:

[0139] extracting attention and global features from the frontal image of the hip joint to obtain a frontal extracted feature map; sequentially down-sampling the frontal extracted feature map to obtain a corresponding first down-sampled feature map, a second down-sampled feature map, a third down-sampled feature map, and a fourth down-sampled feature map; extracting attention and global features from the lateral image of the hip joint to obtain a lateral extracted feature map; and sequentially down-sampling the lateral extracted feature map to obtain a corresponding first down-sampled feature map, a second down-sampled feature map, a third down-sampled feature map, and a fourth down-sampled feature map.

[0140] In a specific embodiment, the conversion and fusion module 103 is further configured to:

[0141] converting and fusing the first down-sampled feature map of the frontal image and the first down-sampled feature map of the lateral image to obtain a corresponding first three-dimensional feature map; converting and fusing the second down-sampled feature map of the frontal image and the second down-sampled feature map of the lateral image to obtain a corresponding second three-dimensional feature map; converting and fusing the third down-sampled feature map of the frontal image and the third down-sampled feature map of the lateral image to obtain a corresponding third three-dimensional feature map; and converting and fusing the fourth down-sampled feature map of the frontal image and the fourth down-sampled feature map of the lateral image to obtain a corresponding fourth three-dimensional feature map.

[0142] In a specific embodiment, the conversion and fusion module 103 is further configured to:

[0143] The orthosteric down-sampling feature map is subjected to two-dimensional to three-dimensional conversion to obtain an orthosteric conversion feature map, the size of the orthosteric conversion feature map being HxWxN; the parasteric down-sampling feature map is subjected to two-dimensional to three-dimensional conversion to obtain a parasteric conversion feature map, the size of the parasteric conversion feature map being HxNxW; wherein the width of the orthosteric conversion feature map is the same as the depth of the parasteric conversion feature map, and the width of the parasteric conversion feature map is the same as the depth of the orthosteric conversion feature map; the parasteric conversion feature map is subjected to azimuth conversion to obtain an azimuth conversion feature map, the size of the azimuth conversion feature map being the same as that of the orthosteric conversion feature map; and the azimuth conversion feature map and the orthosteric conversion feature map are superimposed to obtain the first three-dimensional feature map.

[0144] In a specific implementation, the conversion fusion module 103 is further configured to:

[0145] The orthosteric down-sampling feature map is subjected to flattening processing to obtain a corresponding flattened feature map; the flattened feature map is subjected to full connection processing, normalization processing and Relu processing to obtain a normalized feature map; the normalized feature map is subjected to reshaping processing to obtain a three-dimensional reshaped feature map; and the three-dimensional reshaped feature map is subjected to three-dimensional stabilization processing to obtain the orthosteric conversion feature map.

[0146] The device provided in the above embodiments of the application has a corresponding relationship with the method for three-dimensional reconstruction of a hip joint based on double-plane X-ray films, and thus the specific content in the device has a corresponding relationship with the method for three-dimensional reconstruction of a hip joint, and the specific content can be referred to the records in the method for three-dimensional reconstruction of a hip joint, which will not be described herein again.

[0147] The device provided in the above embodiments of the application has the same beneficial effects as the method adopted, run or implemented by the application program stored therein, based on the same inventive concept as the method for three-dimensional reconstruction of a hip joint based on double-plane X-ray films.

[0148] The internal functions and structures of the device for three-dimensional reconstruction of a hip joint based on double-plane X-ray films are described above, as shown in Figure 7 The device for three-dimensional reconstruction of a hip joint based on double-plane X-ray films can be implemented as an electronic device, including a memory 301 and a processor 303.

[0149] The memory 301 can be configured to store programs.

[0150] In addition, the memory 301 can also be configured to store other various data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0151] The memory 301 can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0152] The processor 303, coupled to the memory 301, is configured to execute programs in the memory 301 for:

[0153] obtaining an anteroposterior image of a hip joint and a lateral image of the hip joint;

[0154] inputting the anteroposterior image of the hip joint and the lateral image of the hip joint into a first processing structure of a three-dimensional reconstruction convolution model respectively to obtain a plurality of corresponding down-sampling feature maps;

[0155] performing conversion fusion on the down-sampling feature map of the anteroposterior image of the hip joint and the corresponding down-sampling feature map of the lateral image of the hip joint to obtain a plurality of corresponding three-dimensional feature maps;

[0156] inputting the plurality of three-dimensional feature maps into an up-sampling structure to obtain a three-dimensional reconstruction result of the hip joint;

[0157] The three-dimensional reconstruction convolution model is trained based on a first sample anteroposterior image and a second sample lateral image and a labeled CT image, and the first sample anteroposterior image or the second sample lateral image is generated from the labeled CT image.

[0158] In a specific embodiment, the processor 303 is further configured to:

[0159] obtain a CT image and a corresponding first sample anteroposterior image or a second sample lateral image, generate the first sample anteroposterior image based on the CT image or generate the second sample lateral image based on the CT image, input the first sample anteroposterior image and the second sample lateral image into a three-dimensional reconstruction convolution model to obtain a three-dimensional reconstruction prediction result, generate a three-dimensional reconstruction labeling result based on the CT image, calculate an overall loss of the three-dimensional reconstruction convolution model based on the three-dimensional reconstruction prediction result and the three-dimensional reconstruction labeling result, and iterate parameters of the three-dimensional reconstruction convolution model according to the overall loss until the loss converges.

[0160] In a specific embodiment, the processor 303 is further configured to:

[0161] Obtaining a CT image, generating a three-dimensional point cloud based on the CT image; determining a projection line corresponding to a first sample frontal image; for each projection line, calculating a path of the projection line passing through the three-dimensional point cloud, and generating a gray value of the projection line in a two-dimensional projection image; traversing all the projection lines to obtain a gray value image, which is the first sample frontal image.

[0162] In a specific embodiment, the processor 303 is further configured to:

[0163] extracting attention and global features from the frontal image of the hip joint to obtain a frontal extracted feature map; sequentially downsampling the frontal extracted feature map to obtain a corresponding first downsampled frontal feature map, a second downsampled frontal feature map, a third downsampled frontal feature map, and a fourth downsampled frontal feature map; extracting attention and global features from the lateral image of the hip joint to obtain a lateral extracted feature map; sequentially downsampling the lateral extracted feature map to obtain a corresponding first downsampled lateral feature map, a second downsampled lateral feature map, a third downsampled lateral feature map, and a fourth downsampled lateral feature map.

[0164] In a specific embodiment, the processor 303 is further configured to:

[0165] transforming and fusing the first downsampled frontal feature map and the first downsampled lateral feature map to obtain a corresponding first three-dimensional feature map; transforming and fusing the second downsampled frontal feature map and the second downsampled lateral feature map to obtain a corresponding second three-dimensional feature map; transforming and fusing the third downsampled frontal feature map and the third downsampled lateral feature map to obtain a corresponding third three-dimensional feature map; transforming and fusing the fourth downsampled frontal feature map and the fourth downsampled lateral feature map to obtain a corresponding fourth three-dimensional feature map.

[0166] In a specific embodiment, the processor 303 is further configured to:

[0167] performing two-dimensional to three-dimensional transformation on the first downsampled frontal feature map to obtain a frontal transformed feature map, the size of the frontal transformed feature map being HxWxN; performing two-dimensional to three-dimensional transformation on the first downsampled lateral feature map to obtain a lateral transformed feature map, the size of the lateral transformed feature map being HxNxW; wherein the width of the frontal transformed feature map is the same as the depth of the lateral transformed feature map, and the width of the lateral transformed feature map is the same as the depth of the frontal transformed feature map; performing orientation transformation on the lateral transformed feature map to obtain an orientation transformed feature map, the size of the orientation transformed feature map being the same as that of the frontal transformed feature map; superimposing the orientation transformed feature map and the frontal transformed feature map to obtain the first three-dimensional feature map.

[0168] In a specific embodiment, the processor 303 is further configured to:

[0169] The flattened feature map is obtained by flattening the positive position lower sampling feature map; the normalized feature map is obtained by full connection processing, normalization processing and Relu processing on the flattened feature map; the three-dimensional reshaped feature map is obtained by reshaping processing on the normalized feature map; and the positive position conversion feature map is obtained by three-dimensional stabilization processing on the three-dimensional reshaped feature map.

[0170] In the present application, the processor is also specifically configured to execute all processes and steps of the above-mentioned hip joint three-dimensional reconstruction method based on double-plane X-ray films, and the specific content can be referred to the record in the hip joint three-dimensional reconstruction method, which will not be described here.

[0171] In the present application, Figure 7 Only some components are shown in the present application, and it does not mean that the electronic device only includes Figure 7 The components shown.

[0172] The electronic device provided by the embodiment of the present application has the same beneficial effects as the method adopted, run or implemented by the application program stored in the electronic device, based on the same inventive concept as the hip joint three-dimensional reconstruction method based on double-plane X-ray films provided by the embodiment of the present application.

[0173] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0174] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0175] These computer program instructions can also be stored in a computer readable storage medium which can guide the computer or other programmable data processing apparatus to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams.Figure 1 one or more processes and / or blocks Figure 1 the function(s) specified in the flow block or blocks.

[0176] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable devices provide steps for implementing the function(s) specified in the flow block or blocks. Figure 1 one or more processes and / or blocks Figure 1 Figure 1 the function(s) specified in the flow block or blocks.

[0177] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0178] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or read-only memory (ROM) 1820, and non-volatile memory, such as flash memory, hard drives, or other forms of persistent memory. The memory is an example of computer-readable media.

[0179] The application also provides a computer readable storage medium corresponding to the method for three-dimensional reconstruction of hip joint based on double plane X-ray film provided by the foregoing embodiments, and a computer program (i.e., program product) is stored on the computer readable storage medium, and when the computer program is executed by a processor, the method for three-dimensional reconstruction of hip joint based on double plane X-ray film provided by any of the foregoing embodiments is executed.

[0180] Computer readable media includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carriers.

[0181] The computer readable storage medium provided by the above embodiments of the present application has the same beneficial effects as the method adopted, run or implemented by the application program stored therein, based on the same inventive concept as the method for three-dimensional reconstruction of hip joints based on double plane X-ray films provided by the embodiments of the present application.

[0182] It should be noted that in the specification provided herein, a large number of specific details are illustrated. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail in order not to obscure the understanding of the present specification.

[0183] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0184] The above only describes the embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for three-dimensional reconstruction of a hip joint based on biplane X-ray images, characterized in that, The method comprises the following steps: obtaining a hip joint frontal image and a hip joint lateral image; inputting the hip joint frontal image and the hip joint lateral image into a first processing structure of a three-dimensional reconstruction convolution model respectively to obtain a plurality of corresponding down-sampling feature maps; performing conversion fusion on the down-sampling feature map of the hip joint frontal image and the corresponding down-sampling feature map of the hip joint lateral image to obtain a plurality of corresponding three-dimensional feature maps; inputting the plurality of three-dimensional feature maps into an up-sampling structure to obtain a three-dimensional reconstruction result of the hip joint; the three-dimensional reconstruction convolution model is trained based on a first sample frontal image and a second sample lateral image and a labeled CT image, and the first sample frontal image or the second sample lateral image is generated from the labeled CT image; the step of inputting the hip joint frontal image and the hip joint lateral image into the first processing structure of the three-dimensional reconstruction convolution model respectively to obtain a plurality of corresponding down-sampling feature maps comprises the following steps: performing attention and global feature extraction on the hip joint frontal image to obtain a frontal extraction feature map; sequentially down-sampling the frontal extraction feature map to obtain a first down-sampling feature map, a second down-sampling feature map, a third down-sampling feature map and a fourth down-sampling feature map of the frontal image; performing attention and global feature extraction on the hip joint lateral image to obtain a lateral extraction feature map; sequentially down-sampling the lateral extraction feature map to obtain a first down-sampling feature map, a second down-sampling feature map, a third down-sampling feature map and a fourth down-sampling feature map of the lateral image; the process of global feature extraction comprises the following steps: taking the output feature map of the attention-like feature as an input feature map of global extraction; inputting the input feature map into a 1×1 convolution layer arranged in parallel to obtain a first extraction map, a second extraction map and a third extraction map respectively; wherein the first extraction map and the third extraction map are in an HW×N format, and the second extraction map is in an N×HW format; multiplying the first extraction map and the second extraction map to obtain a multiplication feature map; performing softmax processing on the multiplication feature map to obtain a multiplication coefficient; multiplying the multiplication coefficient and the third extraction map to obtain a coefficient multiplication map; adding the coefficient multiplication map and the input feature map to obtain an output feature map of global extraction; the softmax processing is a SoftMax operation, and an activation function thereof is: where SoftMax1(x) is an activation function, x i , x j is an element in the input vector, and i, j are element serial numbers.

2. The method for three-dimensional reconstruction of the hip joint based on biplane X-ray images according to claim 1, characterized in that, the training process of the three-dimensional reconstruction convolution model comprises the following steps: obtaining a CT image and a corresponding first sample frontal image, or a second sample lateral image; generating the first sample frontal image based on the CT image, or generating the second sample lateral image based on the CT image; inputting the first sample frontal image and the second sample lateral image into the three-dimensional reconstruction convolution model to obtain a three-dimensional reconstruction prediction result; generating a three-dimensional reconstruction labeling result based on the CT image; calculating an overall loss of the three-dimensional reconstruction convolution model according to the three-dimensional reconstruction prediction result and the three-dimensional reconstruction labeling result; iterating parameters of the three-dimensional reconstruction convolution model according to the overall loss until the loss converges.

3. The method for three-dimensional reconstruction of the hip joint based on biplane X-ray images according to claim 2, characterized in that, the step of generating the first sample frontal image based on the CT image comprises the following steps: obtaining a CT image, and generating a three-dimensional point cloud based on the CT image; determining a projection line corresponding to the first sample frontal image; For each projection line, a path of the projection line through the three-dimensional point cloud is calculated, and a gray value of the projection line in the two-dimensional projection image is generated; The gray value image obtained by traversing all the projection lines is the first sample orthographic image.

4. The method for three-dimensional reconstruction of the hip joint on the basis of biplane X-ray images according to any of claims 1 - 3, characterized in that, The conversion and fusion of the down-sampling feature map of the hip joint orthographic image and the down-sampling feature map corresponding to the hip joint lateral image are performed to obtain a plurality of corresponding three-dimensional feature maps, including: The conversion and fusion of the down-sampling feature map of the orthographic image and the down-sampling feature map of the lateral image are performed to obtain a corresponding first three-dimensional feature map; The conversion and fusion of the down-sampling feature map of the orthographic image and the down-sampling feature map of the lateral image are performed to obtain a corresponding first three-dimensional feature map; The conversion and fusion of the down-sampling feature map of the orthographic image and the down-sampling feature map of the lateral image are performed to obtain a corresponding first three-dimensional feature map; The conversion and fusion of the down-sampling feature map of the orthographic image and the down-sampling feature map of the lateral image are performed to obtain a corresponding first three-dimensional feature map.

5. The method for three-dimensional reconstruction of the hip joint based on biplane X-ray images according to claim 4, characterized in that, The conversion and fusion of the down-sampling feature map of the orthographic image and the down-sampling feature map of the lateral image are performed to obtain a corresponding first three-dimensional feature map, including: The two-dimensional three-dimensional conversion of the down-sampling feature map of the orthographic image is performed to obtain an orthographic conversion feature map, and the size of the orthographic conversion feature map is HxWxN; The two-dimensional three-dimensional conversion of the down-sampling feature map of the lateral image is performed to obtain a lateral conversion feature map, and the size of the lateral conversion feature map is HxNxW; wherein the width of the orthographic conversion feature map is the same as the depth of the lateral conversion feature map, and the width of the lateral conversion feature map is the same as the depth of the orthographic conversion feature map; The orientation conversion of the lateral conversion feature map is performed to obtain an orientation conversion feature map, and the size of the orientation conversion feature map is the same as that of the orthographic conversion feature map; The superposition of the orientation conversion feature map and the orthographic conversion feature map is performed to obtain the first three-dimensional feature map.

6. The method of three-dimensional reconstruction of the hip joint on the basis of double plane X-ray images according to claim 5, characterized in that, The conversion and fusion of the down-sampling feature map of the orthographic image and the down-sampling feature map of the lateral image are performed to obtain a corresponding first three-dimensional feature map, including: The flattening processing of the down-sampling feature map of the orthographic image is performed to obtain a corresponding flattened feature map; The full connection processing, normalization processing and Relu processing of the flattened feature map are performed to obtain a normalized feature map; The reshaping processing of the normalized feature map is performed to obtain a three-dimensional reshaped feature map; The three-dimensional stable processing of the three-dimensional reshaped feature map is performed to obtain the orthographic conversion feature map.

7. A device for three-dimensional reconstruction of a hip joint based on biplane X-ray images, characterized in that It includes: An image acquisition module for acquiring a hip joint orthographic image and a hip joint lateral image; A down-sampling module for inputting the hip joint orthographic image and the hip joint lateral image into a first processing structure of a three-dimensional reconstruction convolution model respectively to obtain a plurality of corresponding down-sampling feature maps; A conversion and fusion module for converting and fusing the down-sampling feature map of the hip joint orthographic image and the down-sampling feature map corresponding to the hip joint lateral image to obtain a plurality of corresponding three-dimensional feature maps; An up-sampling module for inputting the plurality of three-dimensional feature maps into an up-sampling structure to obtain a three-dimensional reconstruction result of the hip joint; The three-dimensional reconstruction convolution model is trained based on a first sample orthographic image and a second sample lateral image and a labeled CT image, and the first sample orthographic image or the second sample lateral image is generated from the labeled CT image; The first processing structure respectively inputs the hip joint frontal image and the hip joint lateral image into a three-dimensional reconstruction convolution model, to obtain a plurality of corresponding down-sampling feature maps, comprising: extracting attention and global features from the hip joint frontal image to obtain a frontal extraction feature map; sequentially down-sampling the frontal extraction feature map to obtain a corresponding first down-sampling feature map, a second down-sampling feature map, a third down-sampling feature map, and a fourth down-sampling feature map; extracting attention and global features from the hip joint lateral image to obtain a lateral extraction feature map; sequentially down-sampling the lateral extraction feature map to obtain a corresponding first down-sampling feature map, a second down-sampling feature map, a third down-sampling feature map, and a fourth down-sampling feature map; the process of global feature extraction is: taking the output feature map of the attention-like as the input feature map of global extraction; inputting the input feature map into a 1×1 convolution layer arranged in parallel to obtain a first extraction map, a second extraction map, and a third extraction map; wherein the first extraction map and the third extraction map are in the format of HW×N, and the second extraction map is in the format of N×HW; multiplying the first extraction map and the second extraction map to obtain a multiplication feature map; performing softmax processing on the multiplication feature map to obtain a multiplication coefficient; multiplying the multiplication coefficient and the third extraction map to obtain a coefficient multiplication map; adding the coefficient multiplication map and the input feature map to obtain the output feature map of global extraction; the softmax processing is a SoftMax operation, and the activation function thereof is: where SoftMax1(x) is an activation function, x i , x j is an element in the input vector, i, j is an element sequence number.

8. An electronic device, comprising: comprising: a memory and a processor; the memory is used to store a program; the processor is coupled to the memory and is used to execute the program, so as to: obtain a hip joint frontal image and a hip joint lateral image; respectively input the hip joint frontal image and the hip joint lateral image into a first processing structure of a three-dimensional reconstruction convolution model to obtain a plurality of corresponding down-sampling feature maps; convert and fuse the down-sampling feature map of the hip joint frontal image and the corresponding down-sampling feature map of the hip joint lateral image to obtain a plurality of corresponding three-dimensional feature maps; input the plurality of three-dimensional feature maps into an up-sampling structure to obtain a three-dimensional reconstruction result of the hip joint; the three-dimensional reconstruction convolution model is obtained based on a first sample frontal image, a second sample lateral image, and a labeled CT image, and the first sample frontal image or the second sample lateral image is generated from the labeled CT image; the first processing structure respectively inputs the hip joint frontal image and the hip joint lateral image into the three-dimensional reconstruction convolution model to obtain the plurality of corresponding down-sampling feature maps, comprising: extracting attention and global features from the hip joint frontal image to obtain a frontal extraction feature map; sequentially down-sampling the frontal extraction feature map to obtain a corresponding first down-sampling feature map, a second down-sampling feature map, a third down-sampling feature map, and a fourth down-sampling feature map; extracting attention and global features from the hip joint lateral image to obtain a lateral extraction feature map; sequentially down-sampling the lateral extraction feature map to obtain a corresponding first down-sampling feature map, a second down-sampling feature map, a third down-sampling feature map, and a fourth down-sampling feature map; The side position extraction feature map is sequentially down-sampled to obtain a corresponding side position first down-sampling feature map, a side position second down-sampling feature map, a side position third down-sampling feature map and a side position fourth down-sampling feature map; The process of global feature extraction is as follows: The output feature map of the class attention is taken as the input feature map of global extraction; The input feature map is input into a 1x1 convolution layer arranged in parallel to obtain a first extraction map, a second extraction map and a third extraction map; the first extraction map and the third extraction map are in an HWxN format, and the second extraction map is in an NxHW format; The first extraction map and the second extraction map are multiplied to obtain a multiplication feature map; The multiplication feature map is subjected to softmax processing to obtain a multiplication coefficient; The multiplication coefficient is multiplied with the third extraction map to obtain a coefficient multiplication map; The coefficient multiplication map and the input feature map are added to obtain an output feature map of global extraction; The softmax processing is a class SoftMax operation, and an activation function thereof is as follows: where SoftMax1(x) is an activation function, x i , x j is an element in the input vector, and i, j are element serial numbers.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the hip joint three-dimensional reconstruction method based on double-plane X-ray films according to any one of claims 1-6.

Images