Method and system for detecting fine fractures based on target detection and graph attention mechanism
By combining object detection and graph attention mechanisms with temporal relationships between images, a small fracture detection model was optimized, solving the problem of inaccurate localization of small fracture regions in chest CT images and achieving higher recognition accuracy and faster detection speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TURING YIDAO MEDICAL DEVICE TECH (SHANGHAI) CO LTD
- Filing Date
- 2022-12-29
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies struggle to accurately identify small fracture areas in chest CT images, and human detection can lead to missed or misdiagnosed cases. Furthermore, they fail to effectively utilize the temporal relationships between images.
We employ a method based on object detection and graph attention mechanisms. We extract image features through convolutional neural networks and combine them with graph attention matrices. We also combine them with the YOLO object detection network to optimize the small fracture detection model. Finally, we use the z-direction-based NMS algorithm and object tracking algorithm for post-processing to improve recognition accuracy.
It improves the accuracy of identifying small fracture areas, especially for small targets smaller than 32×32 pixels, reduces detection time, shortens analysis time, and provides more accurate auxiliary diagnostic results.
Smart Images
Figure CN115908394B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, and in particular to a method and system for detecting small fractures based on target detection and graph attention mechanisms. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] When the chest is subjected to external pressure or violent impact, small fractures often occur at the site of impact. The fractured ends break inward, potentially damaging internal organs. Therefore, chest fractures have a high incidence rate and a certain risk of death. Among these, rib fractures are the most common type of small chest fracture and are a major focus of medical and forensic examinations.
[0004] Currently, computed tomography (CT) is the primary method for diagnosing small fractures of the chest. Physicians typically locate small fractures and their complications based on routine chest CT scans. However, the chest is a large area containing various bones such as ribs, clavicle, sternum, and vertebrae. Small fractures are often hidden and difficult to detect. Furthermore, chest CT scans generally consist of dozens to hundreds of images, making the location of the fracture in each image a mechanical and tedious task. If there are many patients, the workload for detecting small fractures in the chest becomes enormous. Additionally, considering the unavoidable visual fatigue associated with human detection, missed diagnoses and misdiagnoses are common, resulting in relatively low accuracy in detecting small fractures of the chest.
[0005] With the rapid development of deep learning technology in medical imaging, there has been considerable research on its application in detecting lesions such as pulmonary nodules, chest fractures, and mammograms, and it plays a significant role in practical applications. However, current methods for automatically identifying chest fracture sites involve detecting each chest CT image individually, ignoring the temporal relationship between images and neglecting the positional information of image features within the image. This positional information is particularly important for locating small targets, leading to low accuracy in locating fractures. Summary of the Invention
[0006] To address the shortcomings of the existing technologies, this invention provides a method and system for detecting small fractures based on target detection and graph attention mechanisms. Based on chest CT images, a target detection algorithm using graph attention mechanisms is employed to identify small fracture regions in chest CT images. Simultaneously, considering the temporal relationship between chest CT images, a target tracking algorithm is used to optimize the identification results, thereby improving the accuracy of small fracture region identification and solving the problem of inaccurate localization of small fracture regions in chest CT images in existing technologies.
[0007] In a first aspect, this disclosure provides a method for detecting small fractures based on target detection and graph attention mechanisms, including:
[0008] Multiple time-series chest CT images were acquired. After image preprocessing, chest CT images with annotated small fracture areas were used as training samples to construct a training sample set.
[0009] A small fracture detection model is trained using a training sample set. The training process is as follows: the training sample images are input into the small fracture detection model, image features are extracted using a convolutional neural network, a graph attention matrix is extracted using a graph convolutional neural network, the image features and the graph attention matrix are combined to obtain the final features, and based on the final features, combined with a YOLO object detection network, a predicted bounding box for the small fracture region is output. The small fracture detection model is trained based on the distance loss between the predicted bounding box and the target bounding box.
[0010] Input the time-series chest CT image to be tested into the trained small fracture detection model, and output the small fracture area and fracture type.
[0011] A further technical solution is that the preprocessing includes:
[0012] Multiple time-series chest CT images are stacked to create a single multi-channel image.
[0013] Normalization is achieved by resampling based on the pixel distance between chest CT images.
[0014] A further technical solution involves combining the image features and the graph attention matrix to obtain the final features, including:
[0015] The graph attention matrix extracted by the graph convolutional neural network is calculated using 1×1 convolution, and the final non-linear activation function is changed to sigmoid so that the value range of the calculation result is between 0 and 1.
[0016] The calculated graph attention matrix is multiplied by the feature map extracted by the convolutional neural network to obtain the final feature.
[0017] A further technical solution involves inputting time-series chest CT images into a small fracture detection model, sequentially outputting corresponding target detection prediction boxes, and inputting the output results into a small fracture detection optimization model. The model employs a z-direction-based NMS algorithm and a target tracking algorithm to post-process and enhance the prediction results of the entire image sequence.
[0018] Further technical solutions employ a z-axis-based NMS algorithm and target tracking algorithm, including:
[0019] The location of the highest confidence predicted bounding box in the previous chest CT image and the location of the highest confidence predicted bounding box in the next chest CT image are mapped to the current chest CT image. The location of the highest confidence predicted bounding box in the current chest CT image is used as the center point, and the location is compared with the mapped location.
[0020] If the z-coordinate of the mapped position is within the preset range and the IOU between the predicted bounding box of the mapped position and the predicted bounding box of the center point is greater than the set threshold, then the predicted bounding box of the mapped position is deleted, and the predicted bounding box with the highest confidence in the current chest CT image is output as the final predicted bounding box; otherwise, the predicted bounding box of the mapped position is output as the final predicted bounding box.
[0021] A further technical solution involves determining the range of z-coordinate values based on the pixel distance in the chest CT image.
[0022] Secondly, this disclosure provides a small fracture detection system based on target detection and graph attention mechanisms, including:
[0023] The training sample construction module is used to acquire multiple time-series chest CT images. After image preprocessing, the chest CT images with annotated small fracture areas are used as training samples to construct a training sample set.
[0024] The small fracture detection model training module is used to train the small fracture detection model using a training sample set. The training process is as follows: inputting training sample images into the small fracture detection model, extracting image features using a convolutional neural network, extracting a graph attention matrix using a graph convolutional neural network, combining the image features and the graph attention matrix to obtain the final features, and outputting a small fracture region prediction box based on the final features and a YOLO object detection network. The small fracture detection model is trained based on the distance loss between the prediction box and the target box.
[0025] The small fracture detection module is used to input the time-series chest CT images to be tested into the trained small fracture detection model, and output the small fracture area and fracture type.
[0026] Further technical solutions also include a small fracture detection optimization module, which is used to input the output results into the small fracture detection optimization model and use the z-direction-based NMS algorithm and target tracking algorithm to post-process and enhance the prediction results of the entire image sequence.
[0027] Thirdly, this disclosure also provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the method described in the first aspect.
[0028] Fourthly, this disclosure also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps of the method described in the first aspect.
[0029] The above one or more technical solutions have the following beneficial effects:
[0030] 1. This invention provides a method and system for detecting small fractures based on target detection and graph attention mechanism. Based on chest CT images, a target detection algorithm with graph attention mechanism is used to identify small fracture regions in chest CT images. At the same time, considering the temporal relationship between chest CT images, a target tracking algorithm is used to optimize the recognition results, improve the accuracy of small fracture region recognition, and solve the problem of inaccurate localization of small fracture regions in temporal chest CT images in the prior art.
[0031] 2. The present invention also employs a z-direction-based NMS algorithm to fuse the detection results of each CT image in the z-direction and filter the prediction results, thereby obtaining a more concise result for detecting small fractures.
[0032] 3. This invention effectively improves the problem of inaccurate localization of small fractures in the prediction results, especially for small targets with fracture areas smaller than 32×32 pixels. Moreover, compared with traditional image detection, it can effectively reduce the time required for predicting and detecting small fractures, shorten the analysis time, and provide auxiliary diagnostic results as early as possible. Attached Figure Description
[0033] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0034] Figure 1 This is a flowchart of the small fracture detection method based on target detection and graph attention mechanism in Embodiment 1 of the present invention;
[0035] Figure 2 This is a schematic diagram of the fusion of chest CT images in Embodiment 1 of the present invention;
[0036] Figure 3 This is a schematic diagram of the small fracture detection model in Embodiment 1 of the present invention;
[0037] Figure 4 This is a diagram showing the results of detecting small fracture areas in Embodiment 1 of the present invention. Detailed Implementation
[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0040] Example 1
[0041] To address the issue of low accuracy in identifying and detecting small fracture regions in chest CT images using existing technologies, this embodiment provides a method for detecting small fractures based on target detection and graph attention mechanisms, such as... Figure 1 As shown, it includes the following steps:
[0042] Step S1: Acquire multiple time-series chest CT images, preprocess the images, and construct a training sample set using chest CT images with annotated small fracture areas as training samples.
[0043] Step S2: Train the small fracture detection model using the training sample set. The training process is as follows: Input the training sample images into the small fracture detection model, extract image features using a convolutional neural network, extract a graph attention matrix using a graph convolutional neural network, combine the image features and the graph attention matrix to obtain the final features, and based on the final features, combine the YOLO object detection network to output the predicted small fracture region bounding box, and train the small fracture detection model according to the distance loss between the predicted bounding box and the target bounding box.
[0044] Step S3: Input the chest CT image to be tested into the trained small fracture detection model, and output the small fracture area and fracture type.
[0045] In step S1 above, multiple time-series chest CT images are acquired, and the images are preprocessed. This image preprocessing includes stacking the multiple time-series images to synthesize a single multi-channel image. In this embodiment, three adjacent images are stacked to synthesize a single 3-channel image, such as... Figure 2 As shown, after pre-training, chest CT images can have short-term features added to the original image data, thereby optimizing the detection results.
[0046] The above preprocessing also includes resampling based on the spacing of the chest CT image, i.e., the distance between pixels. This spacing refers to the actual distance represented by two adjacent pixels in the image. For example, 1 pixel represents 0.5 mm in the human chest. The chest CT image is resampled using the scipy.ndimage.zoom function to achieve normalization of image pixels, which is beneficial for subsequent unified image processing.
[0047] Meanwhile, in this embodiment, small fracture areas in chest CT images are labeled using target bounding boxes, and the fracture types of the small fracture areas are labeled, namely, rib labels, clavicle labels, sternal labels and vertebral labels. The chest CT images labeled with small fracture areas are used as training samples to construct a training sample set.
[0048] In step S2 above, a small fracture detection model is trained using a training sample set. For example... Figure 2 As shown, in this training process, the training sample images are first input into the small fracture detection model, the image features are extracted using a convolutional neural network, the graph attention matrix is extracted using a graph convolutional neural network, and the image features and the graph attention matrix are combined to obtain the final features.
[0049] Specifically, such as Figure 3 As shown, for each input chest CT image, a Convolutional Neural Network (CNN) and a Graph Convolutional Neural Network (Graph Convolutional Neural Network) are first used to extract image features and a graph attention matrix, respectively. The extracted image features are then processed using the graph attention matrix to simulate an attention mechanism and improve the accuracy of subsequent target detection. To facilitate the fusion of the graph attention matrix and image features, in this embodiment, a 1×1 convolution is also used to calculate the graph attention matrix (which is essentially equivalent to image features) extracted by the Graph Convolutional Neural Network. The final nonlinear activation function is changed to a sigmoid function, ensuring that the value range of the calculation result is between 0 and 1. This calculation method ensures that the output image features are consistent in dimension with the image features extracted by the CNN.
[0050] Subsequently, the calculated graph convolutional neural network features are multiplied with the feature maps extracted by the CNN as an attention mechanism to obtain the final features. That is, by performing a dot product operation between the graph and the feature maps extracted by the CNN, the attention mechanism is simulated, and the calculation result of the graph attention mechanism is added to the feature fusion layer of the original algorithm to enhance the small target position information lost due to downsampling in the subsequent YOLO object detection network, thereby achieving the purpose of optimizing detection.
[0051] Then, based on the final features, combined with the YOLO object detection network, a predicted bounding box for the small fracture region is output. The small fracture detection model is trained using the distance loss between the predicted bounding box and the target bounding box. The aforementioned YOLO object detection network is an existing object detection network used to identify and detect objects in images. This embodiment introduces the aforementioned convolutional neural network on top of this network, integrating an attention mechanism into the feature fusion layer of object detection to optimize object detection.
[0052] Finally, in step S3 above, the chest CT image to be tested is input into the trained small fracture detection model, and the small fracture area and fracture type are output.
[0053] Furthermore, in order to address the problem of not considering the temporal relationship between images as pointed out in the background art, this embodiment also presents an optimized model for small fracture detection based on the above-mentioned small fracture detection model. It adopts the NMS algorithm based on the z-direction and the target tracking algorithm to enhance the post-processing of the prediction results of the entire image sequence, optimize the prediction results of adjacent images, and improve the detection rate.
[0054] The aforementioned NMS algorithm is a non-maximum suppression algorithm, where the z-direction is the depth direction, i.e., the sequential direction of the input time-series chest CT images. It employs a z-direction-based NMS algorithm and a target tracking algorithm. Specifically, after the time-series chest CT images are input into the small fracture detection model, corresponding target detection prediction boxes are output sequentially. The target tracking algorithm maps the position of the prediction box with the highest confidence (i.e., the highest probability value of detecting a certain fracture type in the fracture region of the chest CT image by the above model) from the previous chest CT image and the position of the prediction box with the highest confidence in the next chest CT image to the current chest CT image. Then, based on the... The NMS algorithm in the z-direction filters the predicted bounding boxes. It uses the highest-confidence predicted bounding box in the current chest CT image as the center point and compares this position with the mapped position. If the z-coordinate of the mapped position is within a preset range (i.e., within the suppression range) and the IOU between the predicted bounding box at the mapped position and the predicted bounding box at the center point is greater than a set threshold, then the predicted bounding box at that mapped position is deleted, and the highest-confidence predicted bounding box in the current chest CT image is output as the final predicted bounding box. Otherwise, the predicted bounding box at the mapped position is output as the final predicted bounding box. This method ensures that the output target detection predicted bounding boxes are more accurate and cleaner. Furthermore, the z-coordinate range and the set threshold are preset values, with the z-coordinate range determined based on the spacing of the chest CT image.
[0055] Finally, step S3 is executed, inputting the time-series chest CT image to be tested into the trained small fracture detection model. After passing through the small fracture detection optimization model, the small fracture region and fracture type of the time-series chest CT image to be tested are output, such as... Figure 4As shown.
[0056] In summary, this embodiment maps images from Euclidean space to a non-Euclidean space through spatial mapping, enabling the use of a graph convolutional neural network to obtain a graph attention matrix. The features extracted by the CNN are combined with the graph attention matrix to obtain a weighted result for each pixel in the feature map, detecting small fracture regions in each chest CT image. Then, the output small fracture regions are used as input, and a target tracking algorithm is used to post-process the results, further improving recall. Finally, the detection results for each CT image are fused using Z-direction NMS, and the results are filtered to obtain a more concise small fracture detection result. The final detection result is saved as a JSON file, facilitating the provision of visualized auxiliary diagnostic results for doctors.
[0057] The above scheme effectively improves the problem of inaccurate localization of small fractures in the prediction results, especially for small targets with fracture areas smaller than 32×32 pixels. Moreover, compared with traditional image detection, it can effectively reduce the time required for predicting and detecting small fractures, shorten the analysis time, and provide auxiliary diagnostic results as early as possible.
[0058] Example 2
[0059] This embodiment provides a small fracture detection system based on target detection and graph attention mechanisms, including:
[0060] The training sample construction module is used to acquire multiple time-series chest CT images. After image preprocessing, the chest CT images with annotated small fracture areas are used as training samples to construct a training sample set.
[0061] The small fracture detection model training module is used to train the small fracture detection model using a training sample set. The training process is as follows: inputting training sample images into the small fracture detection model, extracting image features using a convolutional neural network, extracting a graph attention matrix using a graph convolutional neural network, combining the image features and the graph attention matrix to obtain the final features, and outputting a small fracture region prediction box based on the final features and a YOLO object detection network. The small fracture detection model is trained based on the distance loss between the prediction box and the target box.
[0062] The small fracture detection module is used to input the time-series chest CT images to be tested into the trained small fracture detection model, and output the small fracture area and fracture type.
[0063] It also includes a small fracture detection optimization module, which is used to input the output results into the small fracture detection optimization model, and uses the z-direction-based NMS algorithm and target tracking algorithm to post-process and enhance the prediction results of the entire image sequence.
[0064] Example 3
[0065] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps in the small fracture detection method based on target detection and graph attention mechanism as described above.
[0066] Example 4
[0067] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps in the small fracture detection method based on target detection and graph attention mechanism as described above.
[0068] The steps and methods involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.
[0069] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.
[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0071] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A method for detecting small fractures based on target detection and graph attention mechanisms, characterized in that, include: Multiple time-series chest CT images were acquired. After image preprocessing, chest CT images with annotated small fracture areas were used as training samples to construct a training sample set. A small fracture detection model is trained using a training sample set. The training process is as follows: the training sample images are input into the small fracture detection model, image features are extracted using a convolutional neural network, a graph attention matrix is extracted using a graph convolutional neural network, the image features and the graph attention matrix are combined to obtain the final features, and based on the final features, combined with a YOLO object detection network, a predicted bounding box for the small fracture region is output. The small fracture detection model is trained based on the distance loss between the predicted bounding box and the target bounding box. Input the time-series chest CT image to be tested into the trained small fracture detection model, and output the small fracture area and fracture type; After the time-series chest CT images are input into the small fracture detection model, the corresponding target detection prediction boxes are output sequentially. The output results are then input into the small fracture detection optimization model. The NMS algorithm based on the z-direction and the target tracking algorithm are used to post-process and enhance the prediction results of the entire image sequence. The z-axis-based NMS algorithm and target tracking algorithm include: The location of the highest confidence predicted bounding box in the previous chest CT image and the location of the highest confidence predicted bounding box in the next chest CT image are mapped to the current chest CT image. The location of the highest confidence predicted bounding box in the current chest CT image is used as the center point, and the location is compared with the mapped location. If the z-coordinate of the mapped position is within the preset range and the IOU between the predicted bounding box of the mapped position and the predicted bounding box of the center point is greater than the set threshold, then the predicted bounding box of the mapped position is deleted.
2. The method for detecting small fractures based on target detection and graph attention mechanism as described in claim 1, characterized in that, The preprocessing includes: Multiple time-series chest CT images are stacked to create a single multi-channel image. Normalization is achieved by resampling based on the pixel distance between chest CT images.
3. The method for detecting small fractures based on target detection and graph attention mechanism as described in claim 1, characterized in that, The image features and graph attention matrix are combined to obtain the final features, including: The graph attention matrix extracted by the graph convolutional neural network is calculated using 1×1 convolution, and the final non-linear activation function is changed to sigmoid so that the value range of the calculation result is between 0 and 1. The calculated graph attention matrix is multiplied by the feature map extracted by the convolutional neural network to obtain the final feature.
4. The method for detecting small fractures based on target detection and graph attention mechanism as described in claim 1, characterized in that, The range of z-coordinate values is determined based on the pixel distance in the chest CT image.
5. A small fracture detection system based on target detection and graph attention mechanism, characterized in that, include: The training sample construction module is used to acquire multiple time-series chest CT images. After image preprocessing, the chest CT images with annotated small fracture areas are used as training samples to construct a training sample set. The small fracture detection model training module is used to train the small fracture detection model using a training sample set. The training process is as follows: inputting training sample images into the small fracture detection model, extracting image features using a convolutional neural network, extracting a graph attention matrix using a graph convolutional neural network, combining the image features and the graph attention matrix to obtain the final features, and outputting a small fracture region prediction box based on the final features and a YOLO object detection network. The small fracture detection model is trained based on the distance loss between the prediction box and the target box. The small fracture detection module is used to input the time-series chest CT images to be tested into the trained small fracture detection model and output the small fracture area and fracture type. It also includes a small fracture detection optimization module, which is used to input the output results into the small fracture detection optimization model, and uses the z-direction-based NMS algorithm and target tracking algorithm to post-process and enhance the prediction results of the entire image sequence; The z-axis-based NMS algorithm and target tracking algorithm include: The location of the highest confidence predicted bounding box in the previous chest CT image and the location of the highest confidence predicted bounding box in the next chest CT image are mapped to the current chest CT image. The location of the highest confidence predicted bounding box in the current chest CT image is used as the center point, and the location is compared with the mapped location. If the z-coordinate of the mapped position is within the preset range and the IOU between the predicted bounding box of the mapped position and the predicted bounding box of the center point is greater than the set threshold, then the predicted bounding box of the mapped position is deleted.
6. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of a method for detecting small fractures based on target detection and graph attention mechanisms as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of a method for detecting small fractures based on target detection and graph attention mechanism as described in any one of claims 1-4.