Cross-modal enhanced fracture detection methods, systems, media and electronic devices

Abstract

This application provides a fracture detection method based on cross-modal enhancement. The method includes: acquiring a chest image and a chest scan projection; acquiring a chest anteroposterior image from the chest image; preprocessing the chest scan projection to obtain a chest fracture scan projection; performing dynamic feature enhancement on the fracture region of the chest fracture scan projection to obtain an enhanced chest fracture scan projection; generating a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the slightly distorted chest anteroposterior image; and processing the chest anteroposterior image and the simulated chest anteroposterior image using a cross-modal network model to obtain a chest fracture detection image, which includes a global chest fracture detection image and a local chest fracture detection image.

Description

Technical Field

[0001] This application belongs to the field of medical image processing technology and relates to a cross-modal enhanced fracture detection method, system, medium and electronic device. Background Technology

[0002] Rib fractures are a common traumatic condition in clinical practice, especially when the chest is injured. As we age, ribs become increasingly fragile and more susceptible to fracture from external forces. Rib fractures cause severe pain, impair respiratory function, and in severe cases, can lead to complications such as pleural effusion and hemopneumothorax, endangering life. Accurate rib detection and localization can provide a basis for timely intervention, reduce patient suffering, and improve prognosis. However, current methods for detecting rib fractures lack precision; therefore, developing a high-precision fracture detection method is a pressing issue. Summary of the Invention

[0003] The purpose of this application is to provide a cross-modal enhanced fracture detection method, system, medium, and electronic device to improve the accuracy of rib fracture detection.

[0004] In a first aspect, this application provides a fracture detection method based on cross-modal enhancement. The fracture detection method includes: acquiring a chest image and a chest scan projection; acquiring a chest anteroposterior image from the chest image; preprocessing the chest scan projection to obtain a chest fracture scan projection; performing dynamic feature enhancement on the fracture region of the chest fracture scan projection to obtain an enhanced chest fracture scan projection; generating a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the micro-distorted chest anteroposterior image; and processing the chest anteroposterior image and the simulated chest anteroposterior image using a cross-modal network model to obtain a chest fracture detection image, wherein the chest fracture detection image includes a global chest fracture detection image and a local chest fracture detection image; and the cross-modal network model includes a sample-level segmentation network and a region-level segmentation network.

[0005] The process of training a cross-modal network model to obtain chest fracture detection images includes: performing sample-level annotation on training chest anteroposterior images to obtain sample-level annotation results; performing region-level annotation on training simulated chest anteroposterior images to obtain region-level annotation results; training a sample-level segmentation network based on the training chest anteroposterior images and the sample-level annotation results; and training a region-level segmentation network using the training simulated chest anteroposterior images and the region-level annotation results.

[0006] In one implementation of the first aspect, the process of dynamically enhancing the fracture region features of the chest fracture scan projection to obtain an enhanced chest fracture scan projection includes: dynamically adjusting the image features of the fracture region based on the annotation of the chest fracture scan projection to obtain an enhanced chest fracture scan projection within the fracture region; optimizing the boundary features of the fracture region using a fuzzing function to obtain an enhanced chest fracture scan projection after boundary fusion; the enhanced chest fracture scan projection includes the enhanced chest fracture scan projection within the fracture region and the enhanced chest fracture scan projection after boundary fusion.

[0007] In one implementation of the first aspect, the process of generating a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the micro-distorted chest anteroposterior image includes: performing cross-modal data augmentation on the chest anteroposterior image based on the chest fracture scan projection to obtain a preliminary simulated chest anteroposterior image; performing homologous but dissimilar sample augmentation on the chest anteroposterior image to obtain the micro-distorted chest anteroposterior image; and obtaining the simulated chest anteroposterior image based on the preliminary simulated chest anteroposterior image and the micro-distorted chest anteroposterior image.

[0008] In one implementation of the first aspect, the training process of the cross-modal network model further includes: inputting the training chest anteroposterior image and the training simulated chest anteroposterior image into the cross-modal network model to obtain probability values ​​of fracture regions; comparing the probability values ​​with a positive judgment threshold to obtain positive regions, wherein the positive regions are regions where fractures exist; and adaptively adjusting the positive judgment threshold based on the positive regions to obtain a chest fracture detection image containing a single connected region, wherein the single connected region is a continuous positive region.

[0009] In one implementation of the first aspect, the cross-modal network model further includes a backbone network, which is used to extract image features from the chest anteroposterior image and the simulated chest anteroposterior image and input them into the sample-level segmentation network and the region-level segmentation network. The sample-level segmentation network and the region-level segmentation network are trained in parallel, and simultaneously output sample-level fracture detection results and region-level fracture detection results.

[0010] In one implementation of the first aspect, the training process of the cross-modal network model further includes: training based on the training chest anteroposterior image to obtain a preliminary cross-modal network model; and training the preliminary cross-modal network model based on the training simulated chest anteroposterior image.

[0011] In one implementation of the first aspect, the process of obtaining a chest anteroposterior image from a chest image further includes: identifying the chest image to obtain a chest image lacking metadata; and using a sample-level classification model to perform secondary identification on the chest image lacking metadata to obtain the chest anteroposterior image.

[0012] Secondly, this application provides a cross-modal enhanced fracture detection system, the fracture detection system comprising: a plain radiograph acquisition module for acquiring a chest image and a chest scan projection; a chest anteroposterior image acquisition module for acquiring a chest anteroposterior image from the chest image; a chest fracture scan projection acquisition module for preprocessing the chest scan projection to obtain a chest fracture scan projection; a dynamic feature enhancement module for enhancing the dynamic features of the fracture region of the chest fracture scan projection to obtain an enhanced chest fracture scan projection; a simulated image generation module for generating a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and a slightly distorted chest anteroposterior image; and cross-modal network model training. A module is used to process the chest anteroposterior image and the simulated chest anteroposterior image using a cross-modal network model to obtain chest fracture detection images, the chest fracture detection images including overall chest fracture detection images and local chest fracture detection images; wherein, the process of training the cross-modal network model to obtain chest fracture detection images includes: performing sample-level annotation on the training chest anteroposterior image to obtain sample-level annotation results; performing region-level annotation on the training simulated chest anteroposterior image to obtain region-level annotation results; training a sample-level segmentation network based on the training chest anteroposterior image and the sample-level annotation results; and training a region-level segmentation network using the training simulated chest anteroposterior image and the region-level annotation results.

[0013] Thirdly, this application provides an electronic device, the electronic device comprising: a memory storing a computer program thereon; and a processor communicatively connected to the memory for executing the computer program to implement the above-described cross-modal enhanced fracture detection method.

[0014] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-described cross-modal enhanced fracture detection method.

[0015] As described above, the cross-modal enhanced fracture detection method, system, medium, and electronic device described in this application have the following beneficial effects:

[0016] The fracture detection method based on cross-modal enhancement provided in this application combines chest images and chest scan projections for cross-modal enhancement, and dynamically adds features to the fracture region. A cross-modal network model is used to train the enhanced chest anteroposterior image and the chest fracture scan projection. During training, a sample-level segmentation network outputs a global chest fracture detection image, and a region-level segmentation network outputs a local chest fracture detection image, thereby obtaining a chest fracture detection image. The fracture detection method provided in this application can improve the accuracy and diagnostic efficiency of chest fracture detection. Attached Figure Description

[0017] Figure 1 The diagram shows an application scenario of the fracture detection method based on cross-modal enhancement described in this application.

[0018] Figure 2 The diagram shows a process schematic of the fracture detection method based on cross-modal enhancement as described in the embodiments of this application.

[0019] Figure 3 The diagram illustrates the process of obtaining a cross-modal augmentation dataset as described in an embodiment of this application.

[0020] Figure 4 This diagram illustrates the process of training a cross-modal network model as described in an embodiment of this application.

[0021] Figure 5 The diagram shown is a schematic representation of the process for obtaining enhanced thoracic fracture scan projection as described in an embodiment of this application.

[0022] Figure 6 The diagram shows the process of acquiring chest fracture detection images as described in the embodiments of this application.

[0023] Figure 7 The diagram shown is a structural schematic of the fracture detection system based on cross-modal enhancement as described in an embodiment of this application.

[0024] Figure 8 The diagram shown is a structural schematic of the electronic device described in an embodiment of this application.

[0025] Component designation explanation

[0026] 10 Human-Computer Interaction Systems

[0027] 101 DR Image Acquisition Equipment

[0028] 102 CT Scanning Equipment

[0029] 103 Cross-modal network models

[0030] 1. Fracture Detection System Based on Cross-Modal Enhancement

[0031] 11 Flat Slice Acquisition Module

[0032] 12. Chest anteroposterior image acquisition module

[0033] 121 Identification Unit

[0034] 122 Secondary Recognition Unit

[0035] 13. Chest fracture scan projection acquisition module

[0036] 14 Dynamic Feature Enhancement Module

[0037] 141 Dynamic Adjustment Unit

[0038] 142 Boundary Optimization Units

[0039] 15. Simulated Image Generation Module

[0040] 151 Data Augmentation Units

[0041] 152 Sample Augmentation Units

[0042] 153 Image Generation Units

[0043] 16 Cross-modal network model training module

[0044] 2 Electronic devices

[0045] 21. Memory

[0046] 22 processors

[0047] 23 Monitors

[0048] Steps S11 to S16

[0049] Steps S31 to S33

[0050] Steps S51 to S53 Detailed Implementation

[0051] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0052] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0053] Rib fractures are quite common when the chest is injured. With age, the elasticity of the ribs gradually decreases, making them more prone to fracture. Accurate rib detection and localization can provide a basis for timely intervention, reducing patient pain and improving prognosis. Digital radiography (DR) has been widely used to image internal structures or tissues of the human body, and can be further analyzed for medical diagnosis. Data augmentation has become an indispensable part of deep learning in recent years, ensuring full utilization of existing datasets and improving the accuracy of neural networks in various downstream tasks. However, most current implementations use affine-based augmentation preprocessing, focusing on neural network design rather than fully utilizing existing data. With improved medical conditions, CT scans have increasingly higher coverage, offering higher imaging resolution and enabling 3D reconstruction. However, current methods for diagnosing sternal fractures using DR chest X-rays fail to incorporate high-quality CT imaging data and employ controllable adaptive strategies for cross-modal hybrid augmentation, thus limiting the performance of downstream neural networks.

[0054] Traditional cross-modal dataset training typically employs conventional transfer learning and direct CT projection. However, conventional transfer learning often uses multiple identical head networks, pre-training on an auxiliary modality before training on the target modality. While this approach is versatile, incorporating modality transfer tasks into the adaptive learning of the neural network increases the training burden and limits efficiency. Direct CT projection, on the other hand, uses projection methods to convert images into DR modalities, then merges them with real DR data into a larger dataset. This dataset construction method has low utilization of CT sequence data and lacks accurate projection correction. With insufficient correction, directly using transfer learning limits the highest accuracy the model can achieve.

[0055] In response to at least the above-mentioned problems, the following embodiments of this application provide a fracture detection method, system, medium, and electronic device based on cross-modal enhancement.

[0056] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0057] Figure 1This diagram illustrates an application scenario of a fracture detection method based on cross-modal enhancement, as shown in one embodiment of this application. Figure 1 As shown, the human-computer interaction system 10 includes a DR image acquisition device 101, a CT scanning device 102, and a cross-modal network model 103. The DR image acquisition device uses digital X-ray imaging technology to acquire DR images of the patient, and the CT scanning device acquires CT sequences of the patient's CT projection. Based on the CT projection, the DR images are enhanced cross-modally to obtain a cross-modal enhanced dataset. The cross-modal enhanced dataset is input into the cross-modal network model for training, and finally, the location image of the patient's fracture is obtained.

[0058] Figure 2 This is a schematic diagram illustrating the process of a fracture detection method based on cross-modal enhancement in one embodiment of this application. Figure 2 As shown, the fracture detection method based on cross-modal enhancement includes:

[0059] S11, acquire a chest image and a chest scan projection. The chest image is a DR image identified by DR (digital radiography) scanning, and the chest scan projection is a CT sequence obtained by CT (computed tomography) scanning.

[0060] S12, Obtain a chest anteroposterior image from the chest image.

[0061] S13, preprocess the chest scan projection to obtain a chest fracture scan projection.

[0062] S14, enhance the dynamic features of the fracture area on the chest fracture scan projection to obtain an enhanced chest fracture scan projection.

[0063] S15, generate a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the micro-distorted chest anteroposterior image.

[0064] S16, the chest anteroposterior image and the simulated chest anteroposterior image are processed using a cross-modal network model to obtain chest fracture detection images. The cross-modal network model includes a sample-level segmentation network and a region-level segmentation network. The chest fracture detection images include overall chest fracture detection images and local chest fracture detection images. (See details...) Figure 3 .

[0065] Please refer to Figure 4 The process of training a cross-modal network model to obtain images for thoracic fracture detection includes:

[0066] S21, Perform sample-level annotation on the training chest frontal images to obtain sample-level annotation results.

[0067] S22, perform region-level annotation on the training simulation chest frontal image to obtain region-level annotation results.

[0068] S23, The sample-level segmentation network is trained based on the trained chest anteroposterior images and the sample-level annotation results.

[0069] S24, the region-level segmentation network is trained using the simulated chest frontal image and the region-level annotation results.

[0070] As described above, the fracture detection method based on cross-modal enhancement provided in this application can combine chest images and chest scan projections for cross-modal enhancement, and dynamically add features to the fracture region. A cross-modal network model is used to train the cross-modal enhanced chest anteroposterior image and the chest fracture scan projection. During training, a sample-level segmentation network outputs a global chest fracture detection image, and a region-level segmentation network outputs a local chest fracture detection image, thereby obtaining a chest fracture detection image. The fracture detection method provided in this application can improve the accuracy and diagnostic efficiency of chest fracture detection.

[0071] In one embodiment of this application, the process of obtaining a chest anteroposterior image from the chest image includes: acquiring DR (digital radiography) scan data and DR scan metadata; and determining the patient's posture relative to the detector based on the DR scan data and the DR scan metadata.

[0072] For example, the DR scan metadata includes: data precision and range, instrument and radiation parameters, patient disease and symptom description (anonymized), and patient scanning posture. The metadata is used to process DR scan images of the patient's chest (anteroposterior view, lateral sternal view, other areas, and discarded films) to obtain an anteroposterior chest image.

[0073] The process of obtaining a chest frontal image from the chest image further includes: recognizing the chest image to obtain a chest image with missing metadata; and using a sample-level classification model to perform secondary recognition on the chest image with missing metadata to obtain the chest frontal image.

[0074] For example, a sample-level classification model can be trained using a ResNet18 neural network.

[0075] In one embodiment of this application, the preprocessing of the chest scan projection includes: normalizing the chest scan projection (CT sequence) based on metadata from the standard DICOM specification, and correcting and normalizing non-standard CT scan sequences to ensure consistency in imaging value range and spatial definition. The normalization attributes include: effective HU value range and window, sequence reconstruction spacing, and sequence Z-axis order. Samples meeting any of the following conditions will be discarded: no rib fracture lesions detected, insufficient slice count (containing three or fewer ribs), or abnormal HU value distribution.

[0076] Figure 5 This is a schematic diagram illustrating the process of obtaining enhanced thoracic fracture scan projections in one embodiment of this application. For example... Figure 5 As shown, the process of enhancing the dynamic features of the fracture region in the chest fracture scan projection to obtain an enhanced chest fracture scan projection includes:

[0077] S31, dynamically adjust the image features of the fracture area based on the annotation of the chest fracture scan projection to obtain an enhanced chest fracture scan projection within the fracture area.

[0078] In some embodiments, the process of dynamically enhancing the fracture area includes: first, annotating the CT reconstruction sequence to obtain the Region of Interest (ROI), i.e., the area near the fracture tissue, which can be defined as... The corresponding area on the projection is Secondly, the ROI region is dynamically enhanced using dynamic adjustable intensity enhancement and transformation methods, including histogram equalization, nonlinear mapping, Gaussian filtering, and sharpening.

[0079] Specifically, histogram equalization can enhance local contrast in an image, making details within the RoI region more apparent. Non-linear mapping can adjust the non-linear relationship of pixel values, enhancing or weakening features in specific regions. Gaussian filtering and sharpening can enhance edge and texture features within the RoI region.

[0080] S32 uses a fuzzy function to optimize the boundary features of the fracture area in order to obtain an enhanced chest fracture scan projection after boundary fusion.

[0081] In some embodiments, the process of optimizing the boundary features of the fracture region using a fuzzy function includes: using a Poisson distribution to... The boundary pixels are post-processed to blur and minimize the gradient difference between the fracture area and the background DR image, so that it can be smoothly integrated into the original image with reasonable boundary features.

[0082] Adaptive enhancement of the fracture region and post-processing of the fracture boundary region can reduce the consistency between the fracture ROI and other regions, improving the consistency with the image and saving development and tuning costs. During training, dynamically adjusting the image features of the fracture region following a strong-then-weak approach ensures that the cross-modal network model can quickly locate a reasonable range of model parameters in the early stages of training, and then rapidly reduce the enhancement intensity to increase the difficulty of the samples. After the model has roughly fitted the data, and with visible precision observation data, anti-enhancement is used to remove certain features, further increasing the difficulty.

[0083] S33, the enhanced chest fracture scan projection includes the enhanced chest fracture scan projection within the fracture area and the enhanced chest fracture scan projection after boundary fusion.

[0084] In one embodiment of this application, the process of generating a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the micro-distorted chest anteroposterior image includes:

[0085] S41, perform cross-modal data augmentation on the chest anteroposterior image based on the chest fracture scan projection to obtain a preliminary simulated chest anteroposterior image.

[0086] In some embodiments, the process of generating DR images based on CT projection is as follows: X-rays are simulated by using a ray projection method to pass through a virtual X-ray source and onto the object under test (CT reconstruction sequence volume), projecting onto a detector plane perpendicular to the X-ray axis. During this process, the intersection points of the point source and each CT slice in the CT volume data can be calculated, thereby obtaining the corresponding CT value. The CT value obtained by accumulating the entire path is the DR image pixel value of the corresponding point on the detector. The specific description process can be expressed as follows:

[0087] Let the virtual ray source be The virtual detector plane is Perpendicular to the central axis of the ray, assume the object under test (CT volume data) is a three-dimensional volume. Each voxel has a CT value μ(x,y,z). For Each pixel on Define a line from Departure, Passing rays Its and The intersection point can be determined by the following equation ( It is the position vector of the ray source. It is the direction vector of the ray. It is a ray vector, a scalar. (This represents the distance from the radiation source to a point on the radiation beam.)

[0088]

[0089] The nearest neighbor method (Euclidean distance) is used to determine... and The relationship between them

[0090]

[0091] The virtual imaging pixel value is denoted as It can be passed along The integral is used to calculate:

[0092]

[0093] During CT reconstruction, the software included with the instrument already performs the alignment. The attenuation of rays is corrected, so the virtual image obtained by integration is also considered corrected, but a similar correction does not exist in DR imaging. To obtain better domain registration results, it is necessary to... To perform similar attenuation correction, this invention employs the following function ( (Incident ray intensity factor):

[0094]

[0095] S42, perform homologous but different-shape sample augmentation on the chest anteroposterior image to obtain the micro-distorted chest anteroposterior image.

[0096] For example, by adjusting the virtual ray source to Virtual detector plane Multiple chest image samples with the same origin but different shapes are obtained by spatially locating them, with the degree of distortion controlled within ±90°. Appropriately introducing samples with the same origin but different shapes helps the model more accurately grasp the required features and suppress overfitting.

[0097] S43, Based on the preliminary simulated chest frontal image and the slightly distorted chest frontal image, obtain the simulated chest frontal image.

[0098] Figure 6 This is a schematic diagram illustrating the process of acquiring images for chest fracture detection in one embodiment of this application. For example... Figure 6 As shown, the training process of the cross-modal network model further includes:

[0099] S51, input the trained chest anteroposterior image and the trained simulated chest anteroposterior image into the cross-modal network model to obtain the probability value of the fracture area.

[0100] S52, compare the probability value with the positive judgment threshold to obtain the positive area, which is the area where a fracture exists.

[0101] S53, the positive judgment threshold is adaptively adjusted based on the positive region to obtain a chest fracture detection image containing a single connected region, wherein the single connected region is a continuous positive region.

[0102] In some embodiments, chest X-ray images and simulated chest X-ray images are annotated at the sample level and input into a fully convolutional backbone network to generate feature maps. Each feature embedding in the feature map passes through a 1×1 convolutional layer, outputting a probability value for the fracture region and its boundary. The probability value is compared with a positive judgment threshold to identify positive regions. The positive judgment threshold is continuously increased until all positive regions are reduced to simply connected regions. By performing the above sample-based weak supervision operation on the images, no region anchor boxes or pixel-level annotations are required, significantly reducing annotation costs.

[0103] In one embodiment of this application, the cross-modal network model further includes a backbone network, which is used to extract image features from the chest anteroposterior image and the simulated chest anteroposterior image and input them into the sample-level segmentation network and the region-level segmentation network. The sample-level segmentation network and the region-level segmentation network are trained in parallel, and simultaneously output sample-level fracture detection results and region-level fracture detection results.

[0104] In some embodiments, the enhanced thoracic fracture scan projection is annotated and the annotations are projected onto a virtual detector plane. Region-level annotations are obtained, which have higher information entropy than sample-level annotations. A segmentation head sub-network, independent of the backbone network, outputs pixel-level fitting targets. As training progresses, this segmentation head sub-network is gradually discarded, and the frequency of using region-level annotations as training targets gradually decreases. This sub-network is not used during final inference. Through multi-objective warm-up continuous learning, the general direction of parameter optimization can be established in the early stages of training, thereby helping the backbone network reduce the risk of gradient instability caused by coarse-grained sample-level annotations. In cross-modal data, using a continuous learning strategy with multiple parallel segmentation head networks can increase the width of the neural network.

[0105] In one embodiment of this application, the training process of the cross-modal network model further includes:

[0106] S61, Training is performed based on the trained chest anteroposterior images to obtain a preliminary cross-modal network model.

[0107] S62, Based on the simulated chest frontal images, a preliminary cross-modal network model is trained.

[0108] Specifically, in the initial training phase, instead of using CT-simulated samples, real DR images are used directly. This ensures that the model parameters fit the characteristics of the DR modality in the early stages of training, avoiding the risk of reducing the efficiency of subsequent training due to the premature introduction of simulated samples. As training progresses, the usage rate of simulated samples will continuously increase, eventually reaching a proportion where CT samples account for the majority of all samples. Through this progressive augmentation strategy, the cross-modal network model can learn effectively.

[0109] In one embodiment of this application, the fracture detection method based on cross-modal enhancement further includes system integration and output. By integrating a cross-modal network model into an interactive human-machine system, the service flow includes: deployment and startup readiness, receiving DR scan metadata, performing neural network inference, visualizing prediction results, and interactive display. Deploying this interactive human-machine system in clinical practice enables responsive reception of patients' DR image data and returns interactive prediction results within seconds, improving diagnostic efficiency and accuracy for doctors.

[0110] The following will provide a detailed description of the fracture detection method based on cross-modal enhancement provided in this application through a specific example. It should be noted that the content of this example is only used to explain and illustrate the fracture detection method based on cross-modal enhancement provided in this application, and is not intended to limit the scope of protection of this application in any way. In specific applications, corresponding steps can be added or deleted based on this example according to actual needs. The process of the fracture detection method based on cross-modal enhancement in this example includes the following steps:

[0111] Step one: The patient undergoes digital radiography (DR) and CT scans to obtain chest DR images and chest CT sequences. The chest DR images are then analyzed and identified to obtain anteroposterior chest images.

[0112] Step 2: The fracture area in the CT sequence is labeled, the image features of the fracture area are enhanced by a dynamic adjustable intensity method, and the boundary of the fracture area is optimized by Poisson blurring technology to smoothly blend it with the background DR image.

[0113] Step 3 involves cross-modal data augmentation. Based on CT projection, ray transmission is used to simulate and generate DR images, i.e., preliminary simulated chest anteroposterior images. Sample augmentation is then performed by micro-distorting the simulated DR images to generate samples with the same origin but different shapes, thereby expanding the training dataset of the cross-modal network model.

[0114] Step four involves training the cross-modal network model. First, sample-level weak supervision is applied to both the chest X-ray image and a simulated chest X-ray image. After sample-level annotation, these images are input into the cross-modal network model. Parallel sample-level and region-level segmentation networks are used for continuous training, outputting sample-level and region-level fracture detection results respectively. Initially, real DR images are used for training, followed by a gradual increase in simulated DR images. This progressive augmentation strategy improves the model's generalization ability.

[0115] In summary, the fracture detection method based on cross-modal enhancement provided in this application combines CT and DR image data using cross-modal enhancement technology, significantly improving the accuracy and efficiency of rib fracture detection. Furthermore, through CT projection DR image generation technology enhanced with micro-distortion enhancement, the projection data of the CT modality is used to augment and enhance the image data of the DR modality, expanding the data sources for the neural network of DR analysis and obtaining richer and more diverse samples from a single subject, effectively improving the training accuracy of the cross-modal network model. Simultaneously, the sample-level annotation of chest anteroposterior images and simulated chest anteroposterior images, along with the parallel continuous learning strategy of sample-level and region-level networks, significantly reduces the cost of manual annotation and alleviates the problem of unstable data distribution in cross-modal training. In addition, the robustness and training efficiency of the cross-modal network model are improved through dynamic feature enhancement of the fracture region and the progressive augmentation strategy of gradually introducing simulated chest anteroposterior images. Finally, the integrated human-computer interaction system enables rapid deployment in clinical applications, providing doctors with interactive prediction results with sub-second returns, resulting in better diagnostic efficiency, accuracy, and patient quality of care.

[0116] The scope of protection of the fracture detection method based on cross-modal enhancement described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0117] This application also provides a fracture detection system based on cross-modal enhancement. The fracture detection system based on cross-modal enhancement can implement the fracture detection method based on cross-modal enhancement described in this application. However, the implementation device of the fracture detection method based on cross-modal enhancement described in this application includes, but is not limited to, the structure of the fracture detection system based on cross-modal enhancement listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0118] Figure 7 The diagram shown is a structural schematic of a fracture detection system based on cross-modal enhancement according to an embodiment of this application. Figure 7As shown, the fracture detection system 1 based on cross-modal enhancement includes: a plain radiograph acquisition module 11, a chest anteroposterior image acquisition module 12, a chest fracture scan projection acquisition module 13, a dynamic feature enhancement module 14, a simulated image generation module 15, and a cross-modal network model training module 16. Specifically, the plain radiograph acquisition module 11 acquires a chest image and a chest scan projection; the chest anteroposterior image acquisition module 12 identifies the chest image to obtain a chest anteroposterior image; the chest fracture scan projection acquisition module 13 cleans the chest scan projection to obtain a chest fracture scan projection; the dynamic feature enhancement module 14 enhances the fracture region dynamic features of the chest fracture scan projection to obtain an enhanced chest fracture scan projection; the simulated image generation module 15 generates a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the slightly distorted chest anteroposterior image; and the cross-modal network model training module 16 trains a simulated chest anteroposterior image based on the chest anteroposterior image. The simulated chest anteroposterior image and the simulated chest anteroposterior image are used to train a cross-modal network model to obtain chest fracture detection images, which include overall chest fracture detection images and local chest fracture detection images. The process of training the cross-modal network model to obtain chest fracture detection images includes: performing sample-level annotation on the chest anteroposterior image and the simulated chest anteroposterior image, and then inputting them into a sample-level segmentation network to output the overall chest fracture detection image; performing region-level annotation on the simulated chest anteroposterior image and then inputting it into a region-level segmentation network to output the local chest fracture detection image; the chest fracture detection images include both the overall chest fracture detection image and the local chest fracture detection image.

[0119] The chest anteroposterior image acquisition module 12 includes a recognition unit 121 and a secondary recognition unit 122. The recognition unit 121 is used to recognize the chest image to obtain a chest image lacking metadata. The secondary recognition unit 122 is used to perform secondary recognition on the chest image lacking metadata using a sample-level classification model to obtain the chest anteroposterior image.

[0120] The dynamic enhancement module 14 includes a dynamic adjustment unit 141 and a boundary optimization unit 142. The dynamic adjustment unit 141 dynamically adjusts the image features of the fracture region based on the annotations of the chest fracture scan projection to obtain an enhanced chest fracture scan projection within the fracture region. The boundary optimization unit 142 optimizes the boundary features of the fracture region using a fuzzing function to obtain a boundary-fused enhanced chest fracture scan projection; the enhanced chest fracture scan projection includes both the enhanced chest fracture scan projection within the fracture region and the boundary-fused enhanced chest fracture scan projection.

[0121] The simulated image generation module 15 includes a data augmentation unit 151, a sample augmentation unit 152, and an image generation unit 153. The data augmentation unit 151 performs cross-modal data augmentation on the chest anteroposterior image based on the chest fracture scan projection to obtain a preliminary simulated chest anteroposterior image. The sample augmentation unit 152 performs homologous but dissimilar sample augmentation on the chest anteroposterior image to obtain the slightly distorted chest anteroposterior image. The image generation unit 153 obtains the simulated chest anteroposterior image based on the preliminary simulated chest anteroposterior image and the slightly distorted chest anteroposterior image.

[0122] In one embodiment of this application, the training process of the cross-modal network model further includes:

[0123] The trained chest anteroposterior image and the trained simulated chest anteroposterior image are input into a cross-modal network model to obtain the probability value of the fracture region;

[0124] The probability value is compared with the positive judgment threshold to obtain the positive area, which is the area where a fracture exists;

[0125] The positive judgment threshold is adaptively adjusted based on the positive region to obtain a chest fracture detection image containing a single connected region, wherein the single connected region is a continuous positive region.

[0126] In one embodiment of this application, the cross-modal network model further includes a backbone network, which is used to extract image features from the chest anteroposterior image and the simulated chest anteroposterior image and input them into the sample-level segmentation network and the region-level segmentation network. The sample-level segmentation network and the region-level segmentation network are trained in parallel, and simultaneously output sample-level fracture detection results and region-level fracture detection results.

[0127] In one embodiment of this application, the training process of the cross-modal network model further includes:

[0128] Training is performed based on the aforementioned chest anteroposterior images to obtain a preliminary cross-modal network model;

[0129] Based on the simulated chest X-ray images, a preliminary cross-modal network model is trained.

[0130] It should be noted that, Figure 7 The modules in the fracture detection system 1 based on cross-modal enhancement shown are... Figure 2 The steps in the fracture detection method based on cross-modal enhancement correspond one by one, and will not be repeated here.

[0131] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0132] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0133] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the fracture detection method based on cross-modal enhancement provided in this application. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The above storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0135] This application embodiment may also provide an electronic device. Figure 8 The diagram shown is a structural schematic of an electronic device 2 according to an embodiment of this application. Figure 8 As shown, in this embodiment, the electronic device 2 includes a memory 21 and a processor 22.

[0136] The memory 21 is used to store computer programs. In some possible implementations, the memory 21 may include various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0137] In this embodiment, memory 21 may include a computer system readable medium in the form of volatile memory, such as RAM and / or cache memory. Electronic device 2 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 21 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0138] The processor 22 is connected to the memory 21 and is used to execute the computer program stored in the memory 21 so that the electronic device 2 performs a fracture detection method based on cross-modal enhancement.

[0139] For example, processor 22 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. In other embodiments, processor 22 may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0140] In some implementations, the electronic device 2 provided in this application embodiment may further include a display 23. The display 23 is communicatively connected to the memory 21 and the processor 22, and is used to display a graphical user interface (GUI) related to the fracture detection method based on cross-modal enhancement.

[0141] In this embodiment, the display 23 may include a display screen (display panel). In some implementations, the display panel may be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Alternatively, the display 23 may also be a touch panel (touchscreen, touch screen), which may include a display screen and a touch-sensitive surface. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor 22 to determine the type of touch event. Subsequently, the processor 22 provides corresponding visual output on the display device based on the type of touch event.

[0142] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0143] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A fracture detection method based on cross-modal enhancement, characterized in that, The fracture detection method includes: Acquire a chest image and a chest scan projection, wherein the chest image is a DR image identified by DR scan, and the chest scan projection is a CT sequence acquired by CT scan; Obtain an anteroposterior image of the chest from the chest image; The chest scan projection is preprocessed to obtain a chest fracture scan projection; Dynamic feature enhancement of the fracture region is performed on the chest fracture scan projection to obtain an enhanced chest fracture scan projection. A simulated chest anteroposterior image is generated based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the micro-distorted chest anteroposterior image. A cross-modal network model is used to process the chest anteroposterior image and the simulated chest anteroposterior image to obtain chest fracture detection images, which include overall chest fracture detection images and local chest fracture detection images; the cross-modal network model includes a sample-level segmentation network and a region-level segmentation network; The process of training a cross-modal network model to obtain images for thoracic fracture detection includes: Sample-level annotation was performed on the training chest frontal images to obtain sample-level annotation results; Region-level annotation was performed on the simulated chest frontal images during training to obtain the region-level annotation results; The sample-level segmentation network is trained based on the trained chest anteroposterior images and the sample-level annotation results; The region-level segmentation network is trained using the simulated chest frontal image and the region-level annotation results. The process of generating a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the micro-distorted chest anteroposterior image includes: Based on the chest fracture scan projection, cross-modal data augmentation is performed on the chest anteroposterior image to obtain a preliminary simulated chest anteroposterior image; The chest anteroposterior image is augmented with homologous but different-shaped samples to obtain the micro-distorted chest anteroposterior image; The simulated chest image is obtained based on the preliminary simulated chest anteroposterior image and the slightly distorted chest anteroposterior image; The process of generating DR images based on CT projection includes: using ray projection to simulate X-rays passing through a virtual X-ray source through the object to be tested, which is a CT reconstruction sequence volume, and projecting it onto a detector plane perpendicular to the X-ray axis. In this process, the intersection point of the point source and each CT slice in the CT volume data can be calculated to obtain the corresponding CT value. The CT value obtained by accumulating the entire path is the DR image pixel value of the corresponding point on the detector.

2. The fracture detection method according to claim 1, characterized in that, The process of enhancing the dynamic features of the fracture region in the chest fracture scan projection to obtain an enhanced chest fracture scan projection includes: The image features of the fracture region are dynamically adjusted based on the annotation of the chest fracture scan projection to obtain an enhanced chest fracture scan projection within the fracture region. The boundary features of the fracture region are optimized using a fuzzy function to obtain an enhanced chest fracture scan projection after boundary fusion. The enhanced chest fracture scan projection includes the enhanced chest fracture scan projection within the fracture area and the enhanced chest fracture scan projection after boundary fusion.

3. The fracture detection method according to claim 1, characterized in that, The training process of the cross-modal network model also includes: The trained chest anteroposterior image and the trained simulated chest anteroposterior image are input into a cross-modal network model to obtain the probability value of the fracture region; The probability value is compared with the positive judgment threshold to obtain the positive area, which is the area where a fracture exists; The positive judgment threshold is adaptively adjusted based on the positive region to obtain a chest fracture detection image containing a single connected region, wherein the single connected region is a continuous positive region.

4. The fracture detection method according to claim 1, characterized in that, The cross-modal network model also includes a backbone network, which is used to extract image features from the chest anteroposterior image and the simulated chest anteroposterior image and input them into the sample-level segmentation network and the region-level segmentation network. The sample-level segmentation network and the region-level segmentation network are trained in parallel and simultaneously output sample-level fracture detection results and region-level fracture detection results.

5. The fracture detection method according to claim 1, characterized in that, The training process of the cross-modal network model also includes: Training is performed based on the aforementioned chest anteroposterior images to obtain a preliminary cross-modal network model; Based on the simulated chest X-ray images, a preliminary cross-modal network model is trained.

6. The fracture detection method according to claim 1, characterized in that, The process of obtaining a chest anteroposterior image from the chest image also includes: Obtain DR scan data and DR scan metadata; The chest anteroposterior image is obtained based on the DR scan data and the DR scan metadata; The chest images are identified to obtain chest images that lack the DR scan metadata; A sample-level classification model is used to perform secondary identification on the chest image that lacks the DR scan metadata in order to obtain the chest anteroposterior image.

7. A cross-modal enhanced fracture detection system, characterized in that, The fracture detection system includes: The plain film acquisition module is used to acquire chest images and chest scan projections, wherein the chest images are DR images identified by DR scans, and the chest scan projections are CT sequences acquired by CT scans; A chest anteroposterior image acquisition module is used to acquire a chest anteroposterior image from the chest image; A chest fracture scan projection acquisition module is used to preprocess the chest scan projection to obtain a chest fracture scan projection. The dynamic feature enhancement module is used to enhance the dynamic features of the fracture area in the chest fracture scan projection to obtain an enhanced chest fracture scan projection. The simulated image generation module is used to generate a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the micro-distorted chest anteroposterior image. A cross-modal network model training module is used to process the chest anteroposterior image and the simulated chest anteroposterior image using a cross-modal network model to obtain chest fracture detection images, which include overall chest fracture detection images and local chest fracture detection images. The process of training a cross-modal network model to obtain images for thoracic fracture detection includes: Sample-level annotation was performed on the training chest frontal images to obtain sample-level annotation results; Region-level annotation was performed on the simulated chest frontal images during training to obtain the region-level annotation results; The sample-level segmentation network is trained based on the trained chest anteroposterior images and the sample-level annotation results; The region-level segmentation network is trained using the simulated chest frontal image and the region-level annotation results. The process of generating a simulated chest anteroposterior image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the micro-distorted chest anteroposterior image includes: Based on the chest fracture scan projection, cross-modal data augmentation is performed on the chest anteroposterior image to obtain a preliminary simulated chest anteroposterior image; The chest anteroposterior image is augmented with homologous but different-shaped samples to obtain the micro-distorted chest anteroposterior image; The simulated chest image is obtained based on the preliminary simulated chest anteroposterior image and the slightly distorted chest anteroposterior image; The process of generating DR images based on CT projection includes: using ray projection to simulate X-rays passing through a virtual X-ray source through the object to be tested, which is a CT reconstruction sequence volume, and projecting it onto a detector plane perpendicular to the X-ray axis. In this process, the intersection point of the point source and each CT slice in the CT volume data can be calculated to obtain the corresponding CT value. The CT value obtained by accumulating the entire path is the DR image pixel value of the corresponding point on the detector.

8. An electronic device, characterized in that, The electronic device includes: A memory on which computer programs are stored; A processor, communicatively connected to the memory, is used to execute the computer program to implement the cross-modal enhanced fracture detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the cross-modal enhanced fracture detection method as described in any one of claims 1 to 6.

Images