Bone healing time sequence evaluation method, system, equipment and medium

Abstract

The invention discloses a bone healing time sequence evaluation method, system and device and a medium, and the method comprises the steps: obtaining a multi-time-point CT image data set through an image preprocessing and data enhancement method; performing feature extraction on the CT image at each time point by using a convolutional neural network to obtain spatial features of bone healing; and inputting the extracted features into an LSTM network, capturing time sequence changes between different periods, and finally outputting an evaluation result of the bone healing state through a full connection layer. According to the method, the limitation of single-time-point image analysis in the prior art can be overcome, more accurate and reliable bone healing evaluation is provided, and important decision support is provided for clinicians.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a bone healing timing evaluation method, system, equipment and medium. Background Art

[0002] After a fracture, the healing process of bones is a complex physiological process that usually needs to be evaluated through medical images (such as CT or X-rays). At present, most clinicians rely on the experience and visual analysis of doctors to judge the degree of bone healing. However, this evaluation method is highly subjective and susceptible to human error. With the development of medical image processing technology and deep learning technology, computer-assisted bone healing evaluation methods provide clinicians with a more objective and accurate auxiliary tool.

[0003] Most existing studies evaluate bone healing based on medical imaging data at a single time point, ignoring the importance of time series information in the fracture healing process. However, the fracture healing process has significant time dependence, and the formation of callus and the blurring of fracture lines change over time. Image analysis based on a single time point cannot accurately evaluate the dynamic changes of bone healing.

[0004] Therefore, it is urgent to solve the above problems. Summary of the invention

[0005] Purpose of the invention: The first purpose of the present invention is to provide a method for timing assessment of bone healing, which can accurately assess the bone healing status by comparing CT image data of multiple time periods combined with timing information, thereby overcoming the problem of inaccurate assessment caused by single-point analysis in the prior art.

[0006] The second object of the present invention is to provide a method system for evaluating the timing of bone healing.

[0007] A third object of the present invention is to provide an electronic device.

[0008] A fourth object of the present invention is to provide a computer-readable storage medium.

[0009] Technical solution: To achieve the above purpose, the present invention discloses a method for evaluating the timing of bone healing, comprising the following steps:

[0010] (1) Obtaining a number of original CT image data at different periods after fracture, and preprocessing them to obtain a multi-time point CT image dataset; labeling the degree of bone healing for each time point, where 0 represents nonunion, 1 represents bone healing, and 2 represents bone nonunion, to form a label file; dividing the multi-time point CT image dataset into a training dataset and a validation dataset, and combining them with the label file;

[0011] (2) Construct a CNN-FPN-LSTM network, which includes a ResNet50 convolutional neural network, an FPN feature pyramid network, an average pooling layer, at least one LSTM layer, and a fully connected layer;

[0012] Import the training data set obtained in step (1) into the CNN-FPN-LSTM network, stack the CT images at multiple time points into an input tensor of shape (B, T, H, W, C), and convert the CT images into 2D slices; where B is the batch size, B is initially set to 1, T is the number of time steps, representing the number of CT images at different time points, H is the height of the CT image, W is the width of the CT image, and C is the number of channels, C is the single channel corresponding to the grayscale;

[0013] The 2D slices are input into the ResNet50 convolutional neural network for feature extraction, and multiple feature maps of different scales are output. The multiple feature maps of different scales include the maximum resolution feature map P2, the high resolution feature map P3, the medium resolution feature map P4 and the low resolution feature map P5;

[0014] Input multiple feature maps of different scales into the FPN feature pyramid network for fusion to obtain the fused feature map;

[0015] The fused feature map is input into the average pooling layer, and the converted output is a fixed-length feature vector X=[x1,x2,...,x T ]; The feature vector extracted at each time point is input into the LSTM layer to capture the temporal relationship between features at different time points, and the temporal features in the hidden state of the last time step are output for classification;

[0016] The time series features output by the LSTM layer are converted into the evaluation results of the bone healing state through the fully connected layer and the Softmax activation function, and the current bone healing state is output; assuming that the bone healing state is divided into three categories, the CNN-FPN-LSTM network outputs a three-class classification, which are bone nonunion 0, bone healing 1 and bone nonunion 2, and the predicted probability distribution is calculated based on the three-classification results;

[0017] Repeat the training cycle several times until the evaluation accuracy of the validation data set of the CNN-FPN-LSTM network converges, check the evaluation performance of the CNN-FPN-LSTM network, and save the trained CNN-FPN-LSTM network;

[0018] (3) Obtain the original post-fracture CT image data to be evaluated, perform preprocessing, obtain preprocessed CT image data, and use the CNN-FPN-LSTM network trained in step S2 to evaluate the preprocessed CT image data to obtain the current state of bone healing.

[0019] Optionally, the preprocessing in step (1) refers to image normalization, region of interest (ROI) cropping and data enhancement preprocessing, that is, converting the DICOM file of the original CT image data into a 3D or 2D array format, and normalizing the grayscale value of the CT image or converting the HU value into a value within the standard CT range of [-1000, 4000], cropping the CT slice containing the fracture site according to the fracture area, removing redundant background information, and performing random rotation, translation, scaling and noise addition operations.

[0020] Optionally, the formula of the Softmax activation function in step (2) is:

[0021] y pred =Softmax(FC(LSTM(X)))

[0022] Where: y pred is the predicted probability distribution, X = [x1, x2, ..., x T ], LSTM(X), is the result of the LSTM layer aggregating the temporal features of the feature vector; FC is a fully connected layer used for classification tasks.

[0023] Optionally, in step (3), image normalization, region of interest (ROI) cropping and data enhancement preprocessing are performed, the DICOM file of the original CT image data is converted into a 3D or 2D array format, and the grayscale value of the CT image is normalized or the HU value is converted into a value within the standard CT range of [-1000, 4000], and the CT slice containing the fracture site is cropped according to the fracture area, and redundant background information is removed, and random rotation, translation, scaling and noise addition operations are performed.

[0024] Based on the same inventive concept, the present invention discloses a bone healing timing evaluation system, comprising:

[0025] A data preprocessing unit is used to obtain the original CT image data of several different periods after the fracture, and perform preprocessing to obtain a multi-time point CT image data set; annotate the bone healing degree label for each time point, where 0 represents bone nonunion, 1 represents bone healing, and 2 represents bone nonunion, to form a label file; divide the multi-time point CT image data set into a training data set and a validation data set, and combine them with the label file;

[0026] Model training unit, used to build a CNN-FPN-LSTM network, which includes a ResNet50 convolutional neural network, an FPN feature pyramid network, an average pooling layer, at least one LSTM layer, and a fully connected layer;

[0027] Import the acquired training data set into the CNN-FPN-LSTM network, stack the CT images at multiple time points into an input tensor of shape (B, T, H, W, C), and convert the CT images into 2D slices; where B is the batch size, B is initially set to 1, T is the number of time steps, representing the number of CT images at different time points, H is the height of the CT image, W is the width of the CT image, and C is the number of channels, C is the single channel corresponding to the grayscale;

[0028] The 2D slices are input into the ResNet50 convolutional neural network for feature extraction, and multiple feature maps of different scales are output. The multiple feature maps of different scales include the maximum resolution feature map P2, the high resolution feature map P3, the medium resolution feature map P4 and the low resolution feature map P5;

[0029] Input multiple feature maps of different scales into the FPN feature pyramid network for fusion to obtain the fused feature map;

[0030] The fused feature map is input into the average pooling layer, and the converted output is a fixed-length feature vector X=[x1,x2,...,x T ]; The feature vector extracted at each time point is input into the LSTM layer to capture the temporal relationship between features at different time points, and the temporal features in the hidden state of the last time step are output for classification;

[0031] The time series features output by the LSTM layer are converted into the evaluation results of the bone healing state through the fully connected layer and the Softmax activation function, and the current bone healing state is output; assuming that the bone healing state is divided into three categories, the CNN-FPN-LSTM network outputs a three-class classification, which are bone nonunion 0, bone healing 1 and bone nonunion 2, and the predicted probability distribution is calculated based on the three-classification results;

[0032] Repeat the training cycle several times until the evaluation accuracy of the validation data set of the CNN-FPN-LSTM network converges, check the evaluation performance of the CNN-FPN-LSTM network, and save the trained CNN-FPN-LSTM network;

[0033] A bone healing assessment unit is used to obtain the original CT image data after the fracture to be assessed, perform preprocessing, obtain the preprocessed CT image data, and use the CNN-FPN-LSTM network trained in step S2 to evaluate the preprocessed CT image data to obtain the current state of bone healing.

[0034] Optionally, preprocessing in the data preprocessing unit refers to image normalization, region of interest ROI cropping and data enhancement preprocessing, that is, converting the DICOM file of the original CT image data into a 3D or 2D array format, and normalizing the grayscale value of the CT image or converting the HU value into a value within the standard CT range of [-1000,4000], cropping the CT slice containing the fracture site according to the fracture area, removing redundant background information, and performing random rotation, translation, scaling and noise addition operations.

[0035] Optionally, the formula for the Softmax activation function in the model training unit is:

[0036] y pred =Softmax(FC(LSTM(X)))

[0037] Where: y pred is the predicted probability distribution, X = [x1, x2, ..., x T ], LSTM(X), is the result of the LSTM layer aggregating the temporal features of the feature vector; FC is a fully connected layer used for classification tasks.

[0038] Optionally, the bone healing assessment unit performs image normalization, region of interest (ROI) cropping, and data enhancement preprocessing, converts the DICOM file of the original CT image data into a 3D or 2D array format, and normalizes the grayscale value of the CT image or converts the HU value into a value within the standard CT range of [-1000, 4000]. The CT slices containing the fracture site are cropped according to the fracture area, and redundant background information is removed. Random rotation, translation, scaling, and noise addition operations are performed.

[0039] Based on the same inventive concept, the present invention discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a bone healing timing evaluation method as described above.

[0040] Based on the same inventive concept, the present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program is executed by a processor to implement a bone healing timing evaluation method as described above.

[0041] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: the present invention can capture important detail features of fracture lines, calluses, etc. at different scales through multi-layer convolution feature extraction of the CNN network, and provide more accurate spatial features; the present invention uses the LSTM network to effectively process the time-dependent information in the fracture healing process, making up for the shortcomings of traditional single-point analysis; combining the advantages of CNN convolutional neural network and LSTM, it can simultaneously consider spatial features and time series information, and significantly improve the accuracy of bone healing evaluation; the present invention overcomes the limitations of the existing methods based on single-point data analysis, and can effectively capture the dynamic changes in the bone healing process; the present invention improves the generalization ability and application performance of the evaluation method through steps such as data enhancement and multi-scale feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a partial structural diagram of the CNN-FPN-LSTM network in the present invention. Figure 1 ;

[0043] Figure 2 This is a partial structural diagram of the CNN-FPN-LSTM network in the present invention. Figure 2 . DETAILED DESCRIPTION

[0044] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0045] like Figure 1 and Figure 2 As shown, a bone healing timing evaluation method of the present invention comprises the following steps:

[0046] (1) Obtaining the original CT image data of several beagle dogs at different periods after fracture, or obtaining the multi-time point CT image data of patients or animal experiments, and performing image normalization, region of interest (ROI) cropping and data enhancement preprocessing, and then labeling to obtain the multi-time point CT image data set of several beagle dogs; using medical image processing tools to convert the DICOM file of the original CT image data into a 3D or 2D array format, and normalize the grayscale value of the CT image or convert the HU value into a value within the standard CT range of [-1000, 4000], cropping the CT slice containing the fracture site according to the fracture area, removing redundant background information, performing random rotation, translation, scaling and noise addition operations to increase data diversity, and annotating the bone healing degree score for each time point, where 0 represents bone nonunion, 1 represents bone healing, and 2 represents bone nonunion, and the label file is stored in text format; dividing the multi-time point CT image data set into a training data set and a validation data set, and combining them with the label file;

[0047] (2) Construct a CNN-FPN-LSTM network. The CNN-FPN-LSTM network model includes a ResNet50 convolutional neural network, an FPN feature pyramid network, an average pooling layer, at least one LSTM layer, and a fully connected layer;

[0048] Import the acquired training data set into the CNN-FPN-LSTM network, stack the CT images at multiple time points into an input tensor of shape (B, T, H, W, C), where B is the batch size, B is initially set to 1, T is the number of time steps, representing the number of CT images at different time points, H is the height of the CT image, W is the width of the CT image, C is the number of channels, and C is a single channel corresponding to grayscale; convert the CT image into a 2D slice format, and the size of each slice is standardized to 512×512 pixels; if the CT image is 3D, cut the CT image into multiple 2D slices;

[0049] The 2D slice is input into the ResNet50 convolutional neural network for feature extraction, and multiple feature maps of different scales are output, which include a maximum resolution feature map P2, a high resolution feature map P3, a medium resolution feature map P4 and a low resolution feature map P5; the maximum resolution feature map P2 is the largest feature map, which provides the most detailed spatial information, and is particularly effective in extracting the blurring degree of the fracture line at the detail level; the high resolution feature map P3 is a feature map of higher resolution, which captures the local features of the fracture area in a fine-grained manner, such as the formation process of the callus; the medium resolution feature map P4 is a feature map of medium resolution, which is mainly used to extract medium-scale features; the low resolution feature map P5 is a smaller feature map, which contains global semantic information and is suitable for extracting the overall trend of fracture healing; multiple feature maps of different scales jointly describe the features of the bone structure at different levels and scales, and the feature maps contain information that is crucial to the bone healing process, such as the blurring degree of the fracture, the growth of the callus and the possible healing state; the present invention uses a convolutional neural network CNN to extract features from the CT images at each time point to obtain the deep features of the bone structure at that time point, such as the blurring degree of the fracture line, the formation of the callus and the like.

[0050] Multiple feature maps of different scales are input into the FPN feature pyramid network for fusion to obtain the fused feature map; the FPN feature pyramid network is used to extract multi-scale features to help the model capture bone features at different sizes, especially the details of callus and fracture line; FPN is used to combine the multi-level features output by the convolutional neural network CNN to help the model better capture the fracture details of different sizes in bone CT images.

[0051] The fused feature map is input into the global average pooling layer, and the converted output is a fixed-length feature vector X=[x1,x2,...,xT ], x T ∈R n , x T ∈R n is the n-dimensional feature vector extracted by CNN at the Tth time point; the fixed length is 512, and the size of the feature vector is (B, T, 512); the role of global average pooling is to integrate spatial features to generate a fixed-length feature vector, which can be used as the input of the LSTM layer for subsequent classification or regression tasks. The advantage of global average pooling is that by averaging the feature maps of each channel, it avoids the introduction of a large number of parameters in the fully connected layer, reduces the complexity and computational overhead of the model, and is particularly suitable for processing high-resolution input such as medical imaging data. Assume that after the FPN convolution operation, the output feature map size is H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map, and each channel corresponds to a feature map; the role of global average pooling is to reduce the H×W size feature map of each channel to a scalar value, which is the average value of all elements of the feature map of the channel.

[0052] The feature vector extracted at each time point is input into the LSTM layer to capture the temporal relationship between features at different time points, learn the pattern of feature changes during bone healing, and output the temporal feature h under the hidden state (B, h) of the last time step. T , used for classification, h = 256, h is the number of hidden layer units in the LSTM network; the feature vector h T represents the overall bone healing status of the beagle at the current time point, h T It is a fixed-length vector that contains the time-dependent information and spatial feature information captured by the LSTM layer. The present invention inputs the spatial features extracted at each time point into the long short-term memory network LSTM. LSTM can capture the temporal relationship between features at different time points and learn the pattern of feature changes during bone healing. The present invention uses the LSTM network to effectively process the time-dependent information in the fracture healing process, making up for the shortcomings of traditional single-time point analysis.

[0053] The time series features output by the LSTM layer are converted into the evaluation results of the bone healing state through the fully connected layer and the Softmax activation function, and the current bone healing state is output; assuming that the bone healing state is divided into three categories, the CNN-FPN-LSTM network model outputs a three-classification result, and the three-classification results are bone nonunion 0, bone healing 1, and bone nonunion 2. The predicted probability distribution is calculated based on the three-classification results. The formula of the Softmax activation function is:

[0054] y pred =Softmax(FC(LSTM(X)))

[0055] Where: y pred is the predicted probability distribution, for example, [0.1, 0.7, 0.2] indicates the highest probability of bone healing; X = [x1, x2, ..., x T ], LSTM(X), the result of LSTM aggregating the temporal features of the feature vector; FC is a fully connected layer, used for classification tasks;

[0056] The training is repeated multiple times until the classification accuracy of the verification data set of the CNN-FPN-LSTM network model converges, the classification performance of the CNN-FPN-LSTM network model is checked, and the trained CNN-FPN-LSTM network model is saved. During the model training process, the present invention uses multi-scale feature extraction, data enhancement and other technologies to optimize the model, and designs corresponding loss functions and hyperparameter tuning strategies to improve the performance of the model in practical applications.

[0057] The output prediction probability distribution of a given model and the true label y, the formula of the cross entropy loss function is:

[0058]

[0059] Where: C is the number of classification categories, there are 3 categories: nonunion, partial union, complete union; y i is the i-th category of the true label, represented by One-Hot encoding; is the model’s predicted probability for the i-th class.

[0060] Cross entropy loss measures the performance of a model in a classification task by calculating the difference between the predicted distribution and the true distribution; the smaller the loss, the closer the model's prediction is to the true label.

[0061] The present invention uses the Adam optimizer, which can show good convergence in most deep learning models; adopts a dynamic learning rate decay mechanism to adjust the learning rate according to the performance of the validation set; during the model training process, the loss value of the validation set is monitored, and if the validation set loss no longer decreases after several epochs, the training is stopped. Monitor the model performance on the validation set, and evaluate the performance of the classification task through indicators such as accuracy, ROC curve, and confusion matrix. The correlation between the predicted healing score and the actual healing situation is evaluated by the mean square error MSE and the Pearson correlation coefficient. The present invention adjusts the hyperparameter learning rate, the number of LSTM layers, the number of LSTM units, and the regularization parameter to improve the model performance. Use transfer learning technology to perform initial weight assignment with a pre-trained ResNet model to improve training efficiency.

[0062] (3) Obtain the original post-fracture CT image data to be evaluated, and perform image normalization, region of interest (ROI) cropping, and data enhancement preprocessing. Use medical image processing tools to convert the DICOM file of the original CT image data into a 3D or 2D array format, and normalize the grayscale value of the CT image or convert the HU value into a value within the standard CT range of [-1000, 4000]. Crop the CT slice containing the fracture site according to the fracture area, remove redundant background information, perform random rotation, translation, scaling, and noise addition operations, and obtain preprocessed CT image data. Use the CNN-FPN-LSTM network model trained in step S2 to evaluate the preprocessed CT image data and obtain the current state of bone healing.

[0063] Example 2

[0064] Based on the same inventive concept, the present invention discloses a bone healing timing evaluation system, comprising:

[0065] The data preprocessing unit is used to obtain the original CT image data of several different periods after fracture, and perform preprocessing to obtain a multi-time point CT image data set; annotate the bone healing degree label for each time point, where 0 represents nonunion, 1 represents bone healing, and 2 represents bone nonunion, to form a label file; divide the multi-time point CT image data set into a training data set and a validation data set, and combine them with the label file; the preprocessing in the data preprocessing unit refers to image normalization, region of interest ROI cropping and data enhancement preprocessing, that is, converting the DICOM file of the original CT image data into a 3D or 2D array format, and normalizing the grayscale value of the CT image or converting the HU value into a value within the standard CT range [-1000,4000], cropping the CT slice containing the fracture site according to the fracture area, removing redundant background information, and performing random rotation, translation, scaling and noise addition operations.

[0066] Model training unit, used to build a CNN-FPN-LSTM network, which includes a ResNet50 convolutional neural network, an FPN feature pyramid network, an average pooling layer, at least one LSTM layer, and a fully connected layer;

[0067] Import the acquired training data set into the CNN-FPN-LSTM network, stack the CT images at multiple time points into an input tensor of shape (B, T, H, W, C), and convert the CT images into 2D slices; where B is the batch size, B is initially set to 1, T is the number of time steps, representing the number of CT images at different time points, H is the height of the CT image, W is the width of the CT image, and C is the number of channels, C is the single channel corresponding to the grayscale;

[0068] The 2D slices are input into the ResNet50 convolutional neural network for feature extraction, and multiple feature maps of different scales are output. The multiple feature maps of different scales include the maximum resolution feature map P2, the high resolution feature map P3, the medium resolution feature map P4 and the low resolution feature map P5;

[0069] Input multiple feature maps of different scales into the FPN feature pyramid network for fusion to obtain the fused feature map;

[0070] The fused feature map is input into the average pooling layer, and the converted output is a fixed-length feature vector X=[x1,x2,...,x T ]; The feature vector extracted at each time point is input into the LSTM layer to capture the temporal relationship between features at different time points, and the temporal features in the hidden state of the last time step are output for classification;

[0071] The time series features output by the LSTM layer are converted into the evaluation results of the bone healing state through the fully connected layer and the Softmax activation function, and the current bone healing state is output; assuming that the bone healing state is divided into three categories, the CNN-FPN-LSTM network outputs a three-class classification, which are bone nonunion 0, bone healing 1 and bone nonunion 2, and the predicted probability distribution is calculated based on the three-classification results;

[0072] Optionally, the formula for the Softmax activation function in the model training unit is:

[0073] y pred =Softmax(FC(LSTM(X)))

[0074] Where: y pred is the predicted probability distribution, X = [x1, x2, ..., x T ], LSTM(X), the result of the LSTM layer aggregating the temporal features of the feature vector; FC is a fully connected layer, used for classification tasks;

[0075] Repeat the training cycle several times until the evaluation accuracy of the validation data set of the CNN-FPN-LSTM network converges, check the evaluation performance of the CNN-FPN-LSTM network, and save the trained CNN-FPN-LSTM network;

[0076] A bone healing assessment unit is used to obtain the original CT image data after the fracture to be assessed, perform preprocessing, obtain the preprocessed CT image data, and use the CNN-FPN-LSTM network trained in step S2 to evaluate the preprocessed CT image data to obtain the current state of bone healing; the bone healing assessment unit performs image normalization, region of interest (ROI) cropping and data enhancement preprocessing, converts the DICOM file of the original CT image data into a 3D or 2D array format, and normalizes the grayscale value of the CT image or converts the HU value into a value within the standard CT range of [-1000, 4000], and crops the CT slice containing the fracture site according to the fracture area, removes redundant background information, and performs random rotation, translation, scaling and noise addition operations.

[0077] Example 3

[0078] Another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a bone healing timing assessment method as described above.

[0079] The electronic device may include: a processor, a memory, a bus and a communication interface, wherein the processor, the communication interface and the memory are connected via a bus; the memory stores a computer program that can be run on the processor, and when the processor runs the computer program, it executes a bone healing timing evaluation method provided by any of the aforementioned embodiments of the present invention.

[0080] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk memory. The communication connection between the device network element and at least one other network element is realized through at least one communication interface (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0081] The bus may be an ISA bus, a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. Among them, the memory is used to store the program, and the processor executes the program after receiving the execution instruction. The bone healing timing evaluation method disclosed in any implementation of the above-mentioned embodiment of the present invention can be applied to the processor, or implemented by the processor.

[0082] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, which may include a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware.

[0083] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0084] Example 4

[0085] Another embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the method of any of the above-mentioned embodiments. The computer-readable storage medium is an optical disk on which a computer program (i.e., a program product) is stored, and when the computer program is executed by the processor, the method provided by any of the above-mentioned embodiments is executed.

[0086] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

Claims

1. A method for evaluating the timing of bone healing, characterized in that: The steps include: (1) Obtaining a number of original CT image data at different periods after fracture, and preprocessing them to obtain a multi-time point CT image dataset; labeling the degree of bone healing for each time point, where 0 represents nonunion, 1 represents bone healing, and 2 represents bone nonunion, to form a label file; dividing the multi-time point CT image dataset into a training dataset and a validation dataset, and combining them with the label file; (2) Construct a CNN-FPN-LSTM network, which includes a ResNet50 convolutional neural network, an FPN feature pyramid network, an average pooling layer, at least one LSTM layer, and a fully connected layer; Import the training data set obtained in step (1) into the CNN-FPN-LSTM network, stack the CT images at multiple time points into an input tensor of shape (B, T, H, W, C), and convert the CT images into 2D slices; where B is the batch size, B is initially set to 1, T is the number of time steps, representing the number of CT images at different time points, H is the height of the CT image, W is the width of the CT image, and C is the number of channels, C is the single channel corresponding to the grayscale; The 2D slices are input into the ResNet50 convolutional neural network for feature extraction, and multiple feature maps of different scales are output. The multiple feature maps of different scales include the maximum resolution feature map P2, the high resolution feature map P3, the medium resolution feature map P4 and the low resolution feature map P5; Input multiple feature maps of different scales into the FPN feature pyramid network for fusion to obtain the fused feature map; The fused feature map is input into the average pooling layer, and the converted output is a fixed-length feature vector X=[x1,x2,...,x T ]; The feature vector extracted at each time point is input into the LSTM layer to capture the temporal relationship between features at different time points, and the temporal features in the hidden state of the last time step are output for classification; The time series features output by the LSTM layer are converted into the evaluation results of the bone healing state through the fully connected layer and the Softmax activation function, and the current bone healing state is output; assuming that the bone healing state is divided into three categories, the CNN-FPN-LSTM network outputs a three-class classification, which are bone nonunion 0, bone healing 1 and bone nonunion 2, and the predicted probability distribution is calculated based on the three-classification results; Repeat the training cycle several times until the evaluation accuracy of the validation data set of the CNN-FPN-LSTM network converges, check the evaluation performance of the CNN-FPN-LSTM network, and save the trained CNN-FPN-LSTM network; (3) Obtain the original post-fracture CT image data to be evaluated, perform preprocessing, obtain preprocessed CT image data, and use the CNN-FPN-LSTM network trained in step S2 to evaluate the preprocessed CT image data to obtain the current state of bone healing.

2. A bone healing timing evaluation method according to claim 1, characterized in that: The preprocessing in step (1) refers to image normalization, region of interest (ROI) cropping and data enhancement preprocessing, that is, converting the DICOM file of the original CT image data into a 3D or 2D array format, normalizing the grayscale value of the CT image or converting the HU value into a value within the standard CT range of [-1000, 4000], cropping the CT slice containing the fracture site according to the fracture area, removing redundant background information, and performing random rotation, translation, scaling and noise addition operations.

3. A bone healing timing evaluation method according to claim 1, characterized in that: The formula of the Softmax activation function in step (2) is: y pred =Softmax(FC(LSTM(X))) Where: y pred is the predicted probability distribution, X = [x1, x2, ..., x T ], LSTM(X), is the result of the LSTM layer aggregating the temporal features of the feature vector; FC is a fully connected layer used for classification tasks.

4. A bone healing timing evaluation method according to claim 1, characterized in that: In the step (3), image normalization, region of interest (ROI) cropping and data enhancement preprocessing are performed, the DICOM file of the original CT image data is converted into a 3D or 2D array format, and the grayscale value of the CT image is normalized or the HU value is converted into a value within the standard CT range of [-1000, 4000], and the CT slice containing the fracture site is cropped according to the fracture area, the redundant background information is removed, and random rotation, translation, scaling and noise addition operations are performed.

5. A bone healing timing assessment system, characterized in that: include: A data preprocessing unit is used to obtain the original CT image data of several different periods after the fracture, and perform preprocessing to obtain a multi-time point CT image data set; annotate the bone healing degree label for each time point, where 0 represents bone nonunion, 1 represents bone healing, and 2 represents bone nonunion, to form a label file; divide the multi-time point CT image data set into a training data set and a validation data set, and combine them with the label file; Model training unit, used to build a CNN-FPN-LSTM network, which includes a ResNet50 convolutional neural network, an FPN feature pyramid network, an average pooling layer, at least one LSTM layer, and a fully connected layer; Import the acquired training data set into the CNN-FPN-LSTM network, stack the CT images at multiple time points into an input tensor of shape (B, T, H, W, C), and convert the CT images into 2D slices; where B is the batch size, B is initially set to 1, T is the number of time steps, representing the number of CT images at different time points, H is the height of the CT image, W is the width of the CT image, and C is the number of channels, C is the single channel corresponding to the grayscale; The 2D slices are input into the ResNet50 convolutional neural network for feature extraction, and multiple feature maps of different scales are output. The multiple feature maps of different scales include the maximum resolution feature map P2, the high resolution feature map P3, the medium resolution feature map P4 and the low resolution feature map P5; Input multiple feature maps of different scales into the FPN feature pyramid network for fusion to obtain the fused feature map; The fused feature map is input into the average pooling layer, and the converted output is a fixed-length feature vector X=[x1,x2,...,x T ]; The feature vector extracted at each time point is input into the LSTM layer to capture the temporal relationship between features at different time points, and the temporal features in the hidden state of the last time step are output for classification; The time series features output by the LSTM layer are converted into the evaluation results of the bone healing state through the fully connected layer and the Softmax activation function, and the current bone healing state is output; assuming that the bone healing state is divided into three categories, the CNN-FPN-LSTM network outputs a three-class classification, which are bone nonunion 0, bone healing 1 and bone nonunion 2, and the predicted probability distribution is calculated based on the three-classification results; Repeat the training cycle several times until the evaluation accuracy of the validation data set of the CNN-FPN-LSTM network converges, check the evaluation performance of the CNN-FPN-LSTM network, and save the trained CNN-FPN-LSTM network; A bone healing assessment unit is used to obtain the original CT image data after the fracture to be assessed, perform preprocessing, obtain the preprocessed CT image data, and use the CNN-FPN-LSTM network trained in step S2 to evaluate the preprocessed CT image data to obtain the current state of bone healing.

6. A bone healing timing evaluation system according to claim 5, characterized in that: The preprocessing in the data preprocessing unit refers to image normalization, region of interest ROI cropping and data enhancement preprocessing, that is, converting the DICOM file of the original CT image data into a 3D or 2D array format, and normalizing the grayscale value of the CT image or converting the HU value into a value within the standard CT range of [-1000,4000], cropping the CT slice containing the fracture site according to the fracture area, removing redundant background information, and performing random rotation, translation, scaling and noise addition operations.

7. A bone healing timing evaluation system according to claim 5, characterized in that: The formula of the Softmax activation function in the model training unit is: y pred =Softmax(FC(LSTM(X))) Where: y pred is the predicted probability distribution, X = [x1, x2, ..., x T ], LSTM(X), is the result of the LSTM layer aggregating the temporal features of the feature vector; FC is a fully connected layer used for classification tasks.

8. A bone healing timing evaluation system according to claim 5, characterized in that: The bone healing assessment unit performs image normalization, region of interest (ROI) cropping and data enhancement preprocessing, converts the DICOM file of the original CT image data into a 3D or 2D array format, normalizes the grayscale value of the CT image or converts the HU value into a value within the standard CT range of [-1000, 4000], crops the CT slice containing the fracture site according to the fracture area, removes redundant background information, and performs random rotation, translation, scaling and noise addition operations.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a bone healing timing evaluation method as claimed in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement a bone healing timing assessment method according to any one of claims 1 to 4.

Images