Nasal bone fracture image diagnosis and evaluation system
Through the nasal fracture imaging diagnosis and evaluation system, combined with CT images and three-dimensional grid data, the problem of insufficient accuracy in the diagnosis and evaluation of nasal fractures is solved, and more accurate fracture type classification and reduction effect evaluation is achieved.
Patent Information
- Application Number
- CN202411962286.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-18
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
There is a problem of insufficient accuracy in the diagnosis and evaluation of nasal fractures, especially due to the complex geometric structure of nasal bones and the plane characteristics of CT images, which leads to misdiagnosis and lack of objective evaluation methods.
A nasal fracture imaging diagnosis and evaluation system is developed, combining the fracture image acquisition module, three-dimensional reconstruction module and classification module, CT images and three-dimensional grid data are used to classify nasal fracture types, and the reduction effect is evaluated through the evaluation module.
It improves the diagnostic accuracy of nasal fractures, can more accurately classify nasal fracture types, and effectively evaluates the reduction effect of nasal fractures in the absence of normal nasal bone CT images.
Smart Images
Figure CN119851881A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of computer-aided medical treatment, and in particular to an imaging diagnosis and evaluation system for nasal bone fractures. Background Art
[0002] Nasal bone fracture is the most common type of facial injury (epidemiology) and one of the most common ENT emergencies. According to the Global Burden of Disease (GBD) study, there were 7,538,663 new cases of facial fractures in 2017, with the country with the highest incidence reporting an incidence of 376 cases per 100,000 people. In the past, the diagnosis of nasal bone fractures was mainly based on physical examination and plain imaging findings. As an emergency, nasal bone fractures are recommended to be reduced as soon as possible after diagnosis, generally no more than 10 days, otherwise the reduction success rate will be seriously affected, causing mental and economic losses to patients. However, due to the complex geometric structure of the nasal bone, the seams between adjacent bone structures may overlap on plain X-rays, which will affect the accuracy of the diagnosis and lead to misdiagnosis. With the advancement of imaging technology, nasal bone CT can be used to diagnose various types of nasal bone fractures, which is superior to X-rays in terms of high positioning ability, high resolution and image reconstruction ability. At present, on the one hand, the judgment of nasal bone CT images mainly relies on the doctor's experience, which may lead to missed diagnosis or misdiagnosis of subtle hidden fractures and complex fractures, thus delaying diagnosis; on the other hand, due to the planar characteristics of CT images and the swelling of tissues around the fractures, the three-dimensional concept of nasal bone fractures is affected.
[0003] In addition, there is currently a lack of effective and intuitive methods for evaluating the effect of nasal bone fracture reduction. The postoperative results of nasal bone fractures are currently evaluated by complications or patient satisfaction. However, complications and patient satisfaction are subjective, and there is no evaluation of objective results after nasal bone fracture surgery. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a nasal bone fracture imaging diagnosis and evaluation system, which can improve the accuracy of nasal bone fracture diagnosis.
[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a nasal bone fracture imaging diagnosis and evaluation system, comprising:
[0006] A fracture image acquisition module, used for acquiring CT images of nasal bone fractures;
[0007] A three-dimensional reconstruction module, used for generating three-dimensional mesh data of nasal bone fracture based on the nasal bone fracture CT image;
[0008] A classification module is used to input the nasal bone fracture CT image and the nasal bone fracture three-dimensional grid data into the nasal bone fracture classification model to obtain a classification result of the nasal bone fracture type; wherein the nasal bone fracture classification model includes:
[0009] A CT image feature extraction module, used to extract feature information of the nasal bone fracture CT image to obtain flattened image features;
[0010] A three-dimensional structural feature extraction module is used to extract feature information of the three-dimensional mesh data of the nasal bone fracture to obtain shape and structural features;
[0011] A feature fusion module, used for fusing the image features and shape structure features to obtain fused features;
[0012] A classification module is used to classify the type of nasal bone fracture based on the fusion features.
[0013] The CT image feature extraction module includes an input layer, four residual modules and a global average pooling layer connected in sequence, wherein the input layer includes a 3D convolution layer and a 3D pooling layer; each of the four residual modules includes three convolution kernels, wherein the first convolution kernel is 1×1×1, which is used to increase and reduce the dimension of the feature; the second convolution kernel is 1×3×3, which is used to apply two-dimensional convolution in the length and width directions; the third convolution kernel is 3×1×1, which is used to apply one-dimensional convolution in the slice direction; the 1×3×3 convolution kernel and the 3×1×1 convolution kernel both adopt a channel-separated convolution method, and after performing convolution on each channel separately, the 1×1×1 convolution kernel is used to combine the channels; the global average pooling layer is used to flatten the output of the last residual module to obtain flattened image features.
[0014] The three-dimensional structure feature extraction module is constructed based on the MeshCNN network. Specifically, a multi-head graph attention mechanism is added on the basis of the edge convolution of the MeshCNN network, which is expressed as: Among them, e i represents the edge, e i+1 ~e i+4 Indicates e i , concat represents the concatenation operation, K is the number of heads of the multi-head graph attention mechanism, σ( ) represents the activation function, represents the edge e in the k-th graph attention mechanism j For edge e i The attention coefficient is expressed as: Softmax() represents a normalization operation. is the learnable parameter vector, [W k e i ,W k e j ] indicates that W k e i and W k e jFor splicing, W k Represents the learnable weight matrix in the k-th graph attention mechanism. In addition, only the Encoder part of MeshCNN is retained and the Decoder part is removed to reduce the number of parameters.
[0015] The feature fusion module adopts a self-attention mechanism to dynamically determine the weights of image features and shape structure features.
[0016] When training the nasal bone fracture classification model, the CT image feature extraction module and the three-dimensional structure feature extraction module are first trained separately without the feature fusion module to achieve nasal bone fracture classification, and then the trained weights are loaded on the CT image feature extraction module and the three-dimensional structure feature extraction module, and then the feature fusion module is connected to train again until convergence.
[0017] The fracture image acquisition module is also used to acquire a nasal bone fracture repair CT image, wherein the nasal bone fracture repair CT image and the nasal bone fracture CT image are CT images of the same person; the nasal bone fracture imaging diagnosis and evaluation system also includes:
[0018] A normal image acquisition module, used for acquiring a normal nasal bone CT image;
[0019] An evaluation module is used to input the nasal bone fracture CT image, the nasal bone fracture repair CT image and the normal nasal bone CT image into a nasal bone fracture reduction effect evaluation model to obtain a nasal bone fracture reduction effect; wherein the nasal bone fracture reduction effect evaluation model includes:
[0020] A normal image feature extraction module, used to extract image features of the normal nasal bone CT image to obtain normal image features;
[0021] A fracture image feature extraction module, used to extract image features of the nasal bone fracture CT image to obtain fracture image features;
[0022] A repair image feature extraction module is used to extract image features of the nasal bone fracture repair CT image to obtain repair image features;
[0023] A feature splicing module, used for splicing the normal image features, the fracture image features and the repair image features to obtain splicing features;
[0024] The evaluation effect classification module is used to obtain the nasal bone fracture reduction effect based on the splicing features.
[0025] The normal image feature extraction module, the fracture image feature extraction module and the repair image feature extraction module have the same structure, and all include: an input layer, four residual modules and a global average pooling layer connected in sequence, the input layer includes a convolution layer and a pooling layer; the four residual modules, each of which includes three convolution kernels, wherein the first convolution kernel is 1×1×1, which is used to increase and reduce the dimension of the feature; the second convolution kernel is 1×3×3, which is used to apply two-dimensional convolution in the length and width directions; the third convolution kernel is 3×1×1, which is used to apply one-dimensional convolution in the slice direction; the 1×3×3 convolution kernel and the 3×1×1 convolution kernel both adopt a channel-separated convolution method, and after performing convolution on each channel separately, the 1×1×1 convolution kernel is used to combine the channels; the global average pooling layer is used to flatten the output of the last residual module to obtain flattened image features.
[0026] When the nasal bone fracture reduction effect evaluation model is trained, the normal image feature extraction module, the fracture image feature extraction module and the repair image feature extraction module are trained simultaneously, and a softmax cross entropy loss function is used.
[0027] Beneficial Effects
[0028] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: the present invention combines the original CT image data and the three-dimensional mesh data after three-dimensional reconstruction to determine the type of nasal bone fracture, and can simultaneously utilize the bone density and other information in the CT image and the three-dimensional structure information in the three-dimensional mesh, so that the fracture type classification effect is more accurate. The present invention can also realize the fracture reduction effect evaluation through a three-branch network structure, so that the current reduction effect level can be determined in the absence of a patient's normal nasal bone CT image as a control. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a schematic diagram of the structure of a nasal bone fracture imaging diagnosis and evaluation system according to an embodiment of the present invention;
[0030] Figure 2 is a schematic diagram of the structure of a nasal bone fracture classification model in an embodiment of the present invention;
[0031] Figure 3 is a schematic diagram of the structure of a CT image feature extraction module in an embodiment of the present invention;
[0032] Figure 4 is a schematic diagram of equivalently implementing three-dimensional convolution in an embodiment of the present invention;
[0033] Figure 5 is a schematic diagram of the structure of a three-dimensional structure feature extraction module in an embodiment of the present invention;
[0034] Figure 6 Schematic diagram of the relationship between adjacent edges of edge convolution in an embodiment of the present invention;
[0035] Figure 7 is a schematic diagram of a feature fusion module in an embodiment of the present invention;
[0036] Figure 8 is a schematic diagram of the structure of a nasal bone fracture reduction effect evaluation model according to an embodiment of the present invention;
[0037] Fig. 9 It is a structural diagram of the evaluation effect classification module in an implementation manner of the present invention. DETAILED DESCRIPTION
[0038] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.
[0039] The embodiments of the present invention relate to a nasal bone fracture imaging diagnosis and evaluation system, such as Figure 1 As shown, including:
[0040] The fracture image acquisition module is used to acquire CT images of nasal bone fractures.
[0041] A three-dimensional reconstruction module is used to generate three-dimensional mesh data of nasal bone fracture based on the nasal bone fracture CT image. The three-dimensional reconstruction module can use an existing three-dimensional reconstruction algorithm library, such as VTK (Visualization Toolkit) in Python, which can generate corresponding three-dimensional data based on the nasal bone fracture CT image and save it in a three-dimensional mesh format (such as obj format). The reconstructed three-dimensional data can be visualized in medical image processing software (such as 3D Slicer).
[0042] The classification module is used to input the nasal bone fracture CT image and the nasal bone fracture three-dimensional grid data into the nasal bone fracture classification model to obtain the classification result of the nasal bone fracture type.
[0043] like Figure 2 As shown, the nasal bone fracture classification model in this embodiment includes:
[0044] The CT image feature extraction module is used to extract the feature information of the nasal bone fracture CT image and obtain the flattened image features. The CT image feature extraction module is a 3D CNN with a residual network structure, which takes 3D ResNet-50 as the main structure, and decomposes the 3D convolution kernel into 2D convolution + 1D convolution, and then equivalently implements the original 3D convolution calculation by channel separation convolution, and adds an additional nonlinear layer between the 2D convolution and the 1D convolution, which can reduce the amount of calculation while improving the nonlinear representation ability of the network.
[0045] like Figure 3 As shown, the CT image feature extraction module includes an input layer, four residual modules and a global average pooling layer connected in sequence.
[0046] The input layer includes a convolution layer and a pooling layer. The convolution layer uses a 3×7×7 convolution kernel, which means that its size in the length and width directions is 7, and its size in the CT slice direction is 3. The pooling layer uses a 3×3×3 maximum pooling, which means that its size in the length, width, and slice directions is 3.
[0047] The four residual modules adopt a design similar to ResNet50, which contains 3, 4, 6, and 3 basic residual blocks respectively. Each basic residual block contains two convolution operations: 1×1×1 convolution and improved 3D convolution. The 1×1×1 convolution is set at the input and output positions of the residual block, which is responsible for increasing and reducing the dimension of the features; while the improved 3D convolution consists of 3 convolution kernels (see Figure 4 ): 1×3×3 convolution kernel, 3×1×1 convolution kernel, and 1×1×1 convolution kernel. Compared with ordinary 3D convolution, this 3D convolution has two improvements: (1) The 3D convolution kernel is decomposed into 2D convolution kernel + 1D convolution kernel, that is, the 1×3×3 convolution kernel is first used to perform convolution calculations in the length and width directions, and then the 3×1×1 convolution kernel is used to perform convolution calculations in the slice direction. (2) Channel separation convolution is used instead of ordinary convolution, that is, the two convolution kernels described in point (1) are only calculated within the channel, and channel fusion is not performed. Finally, the convolution results calculated separately for each channel are input into the 1×1×1 convolution kernel to achieve channel fusion. Through these two improvements, on the one hand, the amount of calculation can be reduced, and on the other hand, since nonlinear layers can be inserted between these three types of convolution kernels, the nonlinear representation ability of the network is improved.
[0048] The output of the last residual module will enter the global average pooling layer and finally flattened into a 512-dimensional feature vector, which represents the information extracted by the model from the CT image and serves as the input of the feature fusion module.
[0049] In addition, other details of the CT image feature extraction module in this embodiment, such as convolution step size, ShortCut connection and BatchNorm layer, are the same as the settings in ResNet50 and are not repeated here.
[0050] The three-dimensional structure feature extraction module is used to extract the feature information of the three-dimensional grid data of the nasal bone fracture to obtain the shape structure feature.
[0051] In order to identify the three-dimensional mesh data, the three-dimensional structure feature extraction module in this embodiment uses a network structure similar to MeshCNN, which can receive a three-dimensional mesh data and output the corresponding nasal bone fracture category. Its network structure is as follows: Figure 5 shown.
[0052] The convolution calculation method of the three-dimensional structure feature extraction module in this embodiment is improved from the original MeshCNN. Since the input data type is three-dimensional mesh data, it is necessary to implement a convolution operation different from that of ordinary images. The convolution operation implemented in the original MeshCNN is "edge convolution", that is, each edge of the mesh is regarded as an element, and the four adjacent edges of this edge are the neighborhood of the element. For example, for Figure 6 The edge e in the 1 ,e 2 ,e 3 ,e 4 , the edge convolution is calculated as:
[0053]
[0054] Among them, [k0, k1, k2, k3, k4] is equivalent to a convolution kernel. The edge e uses the 5 geometric features of the edge as its specific feature value, which is equivalent to the pixel value in the image. The 5 geometric features used in this implementation are the same as those of the original MeshCNN, and will not be repeated here.
[0055] In the scenario of nasal bone fracture, the fracture area may account for a small proportion in the 3D mesh data, and the differences between different fracture types are also small. In order to enable the model to locate the effective area for classification, this implementation adds a graph attention mechanism based on the edge convolution of the original MeshCNN. The specific operations are as follows:
[0056]
[0057] Among them, W is a learnable weight matrix, α ij It is edge j For edge e i The attention coefficient represents "importance", σ() is the activation function, which can be implemented using the ReLU function. ijThe calculation method of is similar to the self-attention mechanism, and the specific calculation method is:
[0058]
[0059] Among them, [We i ,We j ] represents the two vectors We i and We j Put it together, It is a learnable parameter vector, which is equivalent to linearly mapping the concatenated vector. σ() is still the activation function. Finally, the calculated attention coefficient is normalized to between 0 and 1 through softmax.
[0060] Therefore, in the graph attention mechanism, when calculating the convolution of each edge in the three-dimensional grid, the "importance" of its neighboring edges to this edge is calculated, and each neighboring edge is weighted according to the "importance". These are all trainable parameters. As training progresses, the network will gradually give greater weight to "important" edges, making it easier for the model to locate areas that are effective for classification. This implementation can use a multi-head graph attention mechanism, which is equivalent to multiple convolution kernels in CNN, so that the network can learn richer and more diverse features, as shown below:
[0061]
[0062] Among them, K is the number of heads of the multi-head graph attention mechanism, which is equivalent to the K convolution kernels in the ordinary CNN convolution layer. represents the edge e in the k-th graph attention mechanism j For edge e i The attention coefficient is expressed as: W k represents the learnable weight matrix in the k-th graph attention mechanism.
[0063] In MeshCNN, the overall structure is similar to U-Net, that is, the deep features are dimensionalized by unpooling and then fused with the shallow features, so that the feature dimension of the network's final output is the same as the input to achieve the segmentation task. However, this implementation only needs to implement the classification task, so the unpooling part can be trimmed off, and a series of convolutional layers based on the graph attention mechanism can be used to complete the classification, which can offset the increase in parameters brought by the graph attention mechanism to a certain extent. The last convolutional layer will use global average pooling to merge the features output by the K attention heads, and finally output a 512-dimensional vector, which represents the information extracted by the model from the 3D reconstructed mesh data and serves as the input of the feature fusion module.
[0064] The feature fusion module is used to fuse the image features and shape structure features to obtain fused features.
[0065] In this embodiment, no fixed weights are assigned to the image features and shape structure features, but the importance of the two feature vectors is dynamically determined by the attention mechanism. In this embodiment, the feature fusion module is implemented by the self-attention mechanism, and there are two elements involved in the attention mechanism operation. Figure 7 As shown, the input feature X = [x1x2], X∈R 512×2 It is a matrix composed of image features x1 and shape structure features x2. and There are three learnable matrices that map the input features into Corresponding to the query vector matrix, key vector matrix and value vector matrix in the self-attention mechanism. A represents the attention coefficient matrix, that is, the mutual "importance" between input features. A is calculated as:
[0066]
[0067] Where, divided by the square root Mainly to prevent K T The absolute value of the dot product result of Q is too large, and the softmax function in the above formula is calculated in the column direction. The final output H is:
[0068] H=VA
[0069] It contains two feature vectors x1′ and x′2, which respectively represent the results of weighting the image features and shape structure features according to their mutual "importance". Finally, the two feature vectors x1′ and x′2 are concatenated and then passed to the classification module.
[0070] The classification module is used to complete the classification of the nasal bone fracture type based on the fusion feature. The classification module in this embodiment is a fully connected layer, which receives the two vectors spliced by the feature fusion module, with a dimension of 1×1024, and then maps them into logits with a dimension of 1×6, corresponding to the 6 categories to be classified, and then inputs the logits into the softmax function to calculate the final classification score, and takes the category with the largest score as the classification result.
[0071] The training process of the nasal bone fracture classification model in this embodiment is as follows:
[0072] First, nasal bone CT images are collected. During the collection, normal nasal bone CT images, nasal bone fracture CT images and CT images after fracture repair (cross-section, coronal and sagittal) need to be collected. Among them, the nasal bone fracture CT images and the CT images after fracture repair need to be CT images taken of the same person before and after the fracture reduction surgery, and organized in the form of image pairs.
[0073] Then the collected nasal bone CT images are preprocessed, which mainly includes two parts: category labeling and three-dimensional data generation.
[0074] When performing category labeling, the collected CT images were divided into 1 normal category and 5 fracture categories, and each group of CT images was manually labeled as one of the 6 categories. The 5 fracture categories are: frontal impact group type I (FI): only the lower end of the nasal bone; frontal impact group type II (FII): the proximal part of the nasal bone and the frontal process of the maxilla; lateral impact group type I (LI): unilateral displacement of the nasal bone into the nasal cavity; lateral impact group type II (LII): moderate inward displacement of the ipsilateral nasal bone, accompanied by partial outward displacement of the contralateral nasal bone; comminuted fracture type (C): multi-segmental fracture with telescoping and depression.
[0075] When generating three-dimensional data, a mature three-dimensional reconstruction algorithm library is used, which is VTK (Visualization Toolkit) in Python in this embodiment, to generate corresponding three-dimensional data according to the original CT image and save it in a three-dimensional mesh format (such as obj format).
[0076] For CT images, since the pixel values scanned are often stored as Hounsfield units (HU), the value range varies greatly, so they need to be normalized to between 0 and 1 before entering the model. In addition, the width, height, and depth of different CT images need to be standardized to the same size to meet the needs of the network structure. For 3D mesh data, since the number of 3D mesh facets after reconstruction of different CT images may vary greatly, they also need to be standardized into meshes with the same number of facets to simplify the training process.
[0077] The nasal bone fracture classification model in this embodiment includes two branches, namely, a CT image feature extraction module and a three-dimensional structure feature extraction module. Each branch extracts information of different modalities respectively. Theoretically, both branches can independently complete the classification task. Therefore, in order to reduce the difficulty of training, when training the nasal bone fracture classification model, the feature fusion module is removed first, and the CT image feature extraction module and the three-dimensional structure feature extraction module are trained separately to realize the nasal bone fracture classification. Then, the two branches load their trained weights respectively, and then connect to the feature fusion module, and train again until convergence.
[0078] The fracture image acquisition module in this embodiment is also used to acquire a nasal bone fracture repair CT image, wherein the nasal bone fracture repair CT image and the nasal bone fracture CT image are CT images of the same person. The nasal bone fracture imaging diagnosis and evaluation system in this embodiment also includes:
[0079] A normal image acquisition module, used for acquiring a normal nasal bone CT image;
[0080] The evaluation module is used to input the nasal bone fracture CT image, the nasal bone fracture repair CT image and the normal nasal bone CT image into the nasal bone fracture reduction effect evaluation model to obtain the nasal bone fracture reduction effect.
[0081] like Figure 8 As shown in the figure, the nasal bone fracture reduction effect evaluation model includes:
[0082] A normal image feature extraction module, used to extract image features of the normal nasal bone CT image to obtain normal image features;
[0083] A fracture image feature extraction module, used to extract image features of the nasal bone fracture CT image to obtain fracture image features;
[0084] A repair image feature extraction module is used to extract image features of the nasal bone fracture repair CT image to obtain repair image features;
[0085] A feature splicing module, used for splicing the normal image features, the fracture image features and the repair image features to obtain splicing features;
[0086] The evaluation effect classification module is used to obtain the nasal bone fracture reduction effect based on the splicing features.
[0087] In this embodiment, the normal image feature extraction module, the fracture image feature extraction module and the repair image feature extraction module respectively receive the normal nasal bone CT image, the nasal bone fracture CT image and the corresponding nasal bone fracture repair CT image. Each feature extraction module can extract a 512-dimensional feature vector, and then splice the three 512-dimensional feature vectors together and pass them into the evaluation effect classification module for classification.
[0088] In this embodiment, the normal image feature extraction module, the fracture image feature extraction module and the repair image feature extraction module have the same structure as the CT image feature extraction module in the nasal bone fracture classification model, which is to decompose the 3D convolution kernel into 2D convolution kernel + 1D convolution kernel, and use channel separation convolution to improve the 3D ResNet50.
[0089] like Fig. 9As shown in the figure, the evaluation effect classification module is a 3-layer MLP, and the feature dimensions of the 3 layers are 1536 (512×3), 512 and 4 respectively. The output of the 3rd layer is the category logits input to the softmax function. The 4 logits correspond to the 4 levels of repair effect. ReLU is used as a nonlinear activation function between layers. During training, the normal image feature extraction module, fracture image feature extraction module and repair image feature extraction module still use the softmax cross entropy loss function, but the difference from the nasal bone fracture classification model is that the normal image feature extraction module, fracture image feature extraction module and repair image feature extraction module in the nasal bone fracture reduction effect evaluation model are trained simultaneously, because the fracture reduction effect requires the comprehensive three features to be judged.
[0090] Before training the nasal bone fracture reduction effect evaluation model, this embodiment simultaneously collects the nasal bone fracture repair CT images after reduction and repair for each nasal bone fracture CT image. In the nasal bone fracture reduction effect evaluation model, these CT images need to be paired into image pairs, and each image pair is labeled with its repair effect. This embodiment divides the repair effect into the following four levels, namely: Excellent: no nasal deviation, smooth arch shape, and no observed fracture segment misalignment; Good: no nasal deviation, smooth arch shape, poor force line, but accompanied by single segment irregularity or displacement; General: no nasal deviation, smooth arch shape, poor force line, accompanied by bone irregularity and displacement; Poor: nasal deviation, uneven arch shape, and two segments of bone irregularity and displacement.
[0091] It is worth mentioning that the nasal bone fracture CT image and the nasal bone fracture repair CT image used in this embodiment are two CT images of the same person before and after the fracture reduction, while the normal nasal bone CT image is a nasal bone CT image of any person. Because in clinical practice, it is often not easy to obtain CT images of patients with nasal bone fractures in normal periods, but in the process of model training, the features of normal nasal bone CT images can also be learned from normal nasal bone CT images, and the quality of the reduction effect can be judged from the three features after splicing.
[0092] It is not difficult to find that the present invention combines the original CT image data and the three-dimensional mesh data after three-dimensional reconstruction to determine the type of nasal bone fracture. It can simultaneously use the bone density and other information in the CT image and the three-dimensional structure information in the three-dimensional mesh, making the fracture type classification effect more accurate. The present invention can also realize the fracture reduction effect evaluation through a three-branch network structure, so that the current reduction effect level can be determined in the absence of a patient's normal nasal bone CT image as a control.
Claims
1. A nasal bone fracture imaging diagnosis and evaluation system, characterized in that: include: A fracture image acquisition module, used for acquiring CT images of nasal bone fractures; A three-dimensional reconstruction module, used for generating three-dimensional mesh data of nasal bone fracture based on the nasal bone fracture CT image; A classification module is used to input the nasal bone fracture CT image and the nasal bone fracture three-dimensional grid data into the nasal bone fracture classification model to obtain a classification result of the nasal bone fracture type; wherein the nasal bone fracture classification model includes: A CT image feature extraction module, used to extract feature information of the nasal bone fracture CT image to obtain flattened image features; A three-dimensional structural feature extraction module is used to extract feature information of the three-dimensional mesh data of the nasal bone fracture to obtain shape and structural features; A feature fusion module, used for fusing the image features and shape structure features to obtain fused features; A classification module is used to classify the type of nasal bone fracture based on the fusion features.
2. The nasal bone fracture imaging diagnosis and evaluation system according to claim 1, characterized in that: The CT image feature extraction module includes an input layer, four residual modules and a global average pooling layer connected in sequence, wherein the input layer includes a 3D convolution layer and a 3D pooling layer; each of the four residual modules includes three convolution kernels, wherein the first convolution kernel is 1×1×1, which is used to increase and reduce the dimension of the feature; the second convolution kernel is 1×3×3, which is used to apply two-dimensional convolution in the length and width directions; the third convolution kernel is 3×1×1, which is used to apply one-dimensional convolution in the slice direction; the 1×3×3 convolution kernel and the 3×1×1 convolution kernel both adopt a channel-separated convolution method, and after performing convolution on each channel separately, the 1×1×1 convolution kernel is used to combine the channels; the global average pooling layer is used to flatten the output of the last residual module to obtain flattened image features.
3. The nasal bone fracture imaging diagnosis and evaluation system according to claim 1, characterized in that: The three-dimensional structure feature extraction module is constructed based on the MeshCNN network. Specifically, a multi-head graph attention mechanism is added on the basis of the edge convolution of the MeshCNN network, which is expressed as: Among them, e i represents the edge, e i+1 ~e i+4 Indicates e i , concat represents the concatenation operation, K is the number of heads of the multi-head graph attention mechanism, σ() represents the activation function, represents the edge e in the k-th graph attention mechanism j For edge e i The attention coefficient is expressed as: Softmax() represents a normalization operation. is the learnable parameter vector, [W k e i ,W k e j ] indicates that W k e i and W k e j For splicing, W k represents the learnable weight matrix in the k-th graph attention mechanism.
4. The nasal bone fracture imaging diagnosis and evaluation system according to claim 1, characterized in that: The feature fusion module adopts a self-attention mechanism to dynamically determine the weights of image features and shape structure features.
5. The nasal bone fracture imaging diagnosis and evaluation system according to claim 1, characterized in that: When training the nasal bone fracture classification model, the CT image feature extraction module and the three-dimensional structure feature extraction module are first trained separately without the feature fusion module to achieve nasal bone fracture classification, and then the trained weights are loaded on the CT image feature extraction module and the three-dimensional structure feature extraction module, and then the feature fusion module is connected to train again until convergence.
6. The nasal bone fracture imaging diagnosis and evaluation system according to claim 1, characterized in that: The fracture image acquisition module is also used to acquire a nasal bone fracture repair CT image, wherein the nasal bone fracture repair CT image and the nasal bone fracture CT image are CT images of the same person; the nasal bone fracture imaging diagnosis and evaluation system also includes: A normal image acquisition module, used for acquiring a normal nasal bone CT image; An evaluation module is used to input the nasal bone fracture CT image, the nasal bone fracture repair CT image and the normal nasal bone CT image into a nasal bone fracture reduction effect evaluation model to obtain a nasal bone fracture reduction effect; wherein the nasal bone fracture reduction effect evaluation model includes: A normal image feature extraction module, used to extract image features of the normal nasal bone CT image to obtain normal image features; A fracture image feature extraction module, used to extract image features of the nasal bone fracture CT image to obtain fracture image features; A repair image feature extraction module is used to extract image features of the nasal bone fracture repair CT image to obtain repair image features; A feature splicing module, used for splicing the normal image features, the fracture image features and the repair image features to obtain splicing features; The evaluation effect classification module is used to obtain the nasal bone fracture reduction effect based on the splicing features.
7. The nasal bone fracture imaging diagnosis and evaluation system according to claim 6, characterized in that: The normal image feature extraction module, the fracture image feature extraction module and the repair image feature extraction module have the same structure, and all include: an input layer, four residual modules and a global average pooling layer connected in sequence, the input layer includes a convolution layer and a pooling layer; the four residual modules, each of which includes three convolution kernels, wherein the first convolution kernel is 1×1×1, which is used to increase and reduce the dimension of the feature; the second convolution kernel is 1×3×3, which is used to apply two-dimensional convolution in the length and width directions; the third convolution kernel is 3×1×1, which is used to apply one-dimensional convolution in the slice direction; the 1×3×3 convolution kernel and the 3×1×1 convolution kernel both adopt a channel-separated convolution method, and after performing convolution on each channel separately, the 1×1×1 convolution kernel is used to combine the channels; the global average pooling layer is used to flatten the output of the last residual module to obtain flattened image features.
8. The nasal bone fracture imaging diagnosis and evaluation system according to claim 6, characterized in that: When the nasal bone fracture reduction effect evaluation model is trained, the normal image feature extraction module, the fracture image feature extraction module and the repair image feature extraction module are trained simultaneously, and a softmax cross entropy loss function is used.