Cross-modal enhanced fracture detection method and system, medium and electronic equipment

Abstract

The invention provides a fracture detection method based on cross-modal enhancement. The fracture detection method based on cross-modal enhancement comprises the following steps: acquiring a chest image and a chest scanning projection; acquiring a chest normal position image from the chest image; preprocessing the chest scanning projection to obtain a chest fracture scanning projection; performing fracture area dynamic feature enhancement on the chest fracture scanning projection to obtain an enhanced chest fracture scanning projection; generating a simulated chest normal position image based on the chest fracture scanning projection, the enhanced chest fracture scanning projection and the micro-distorted chest normal position image; and processing the chest normal position image and the simulated chest normal position image by using a cross-modal network model to obtain chest fracture detection images, wherein the chest fracture detection images comprise an overall chest fracture detection image and a local chest fracture detection image.

Description

Technical Field

[0001] The present application belongs to the technical field of medical image processing, and relates to a cross-modal enhanced fracture detection method, system, medium and electronic equipment. Background Art

[0002] Rib fractures are common traumatic diseases in clinical practice, especially when the chest is injured. As people age, the ribs gradually become fragile and more susceptible to fractures caused by external forces. Rib fractures can cause severe pain and affect respiratory function. In severe cases, they may even lead to complications such as pleural effusion and hemothorax, which are life-threatening. Accurate rib detection and positioning can provide a basis for timely interventional treatment, reduce patient pain, and improve prognosis. However, the current rib fracture detection method has low detection accuracy. Therefore, how to provide a high-precision fracture detection method has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the present application is to provide a cross-modal enhanced fracture detection method, system, medium and electronic equipment to improve the accuracy of rib fracture detection.

[0004] In a first aspect, the present application provides a fracture detection method based on cross-modal enhancement, the fracture detection method comprising: acquiring a chest image and a chest scan projection; acquiring a chest AP image from the chest image; preprocessing the chest scan projection to acquire a chest fracture scan projection; performing fracture region dynamic feature enhancement on the chest fracture scan projection to acquire an enhanced chest fracture scan projection; generating a simulated chest AP image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and a slightly distorted chest AP image; processing the chest AP image and the simulated chest AP image using a cross-modal network model to acquire a chest fracture detection image, the chest fracture detection image comprising an overall chest fracture detection image and a local chest fracture detection image; the cross-modal network model comprises a sample-level segmentation network and a region-level segmentation network;

[0005] Among them, the process of using a cross-modal network model for training to obtain chest fracture detection images includes: performing sample-level annotation on the training chest AP images to obtain sample-level annotation results; performing region-level annotation on the training simulated chest AP images to obtain region-level annotation results; training a sample-level segmentation network based on the training chest AP images and the sample-level annotation results; and training a region-level segmentation network using the training simulated chest AP images and the region-level annotation results.

[0006] In an implementation of the first aspect, the process of dynamically enhancing the features of the fracture area of ​​the chest fracture scan projection to obtain an enhanced chest fracture scan projection includes: dynamically adjusting the image features of the fracture area based on the annotation of the chest fracture scan projection to obtain an enhanced chest fracture scan projection in the fracture area; optimizing the boundary features of the fracture area by using a fuzzy function to obtain an enhanced chest fracture scan projection after boundary fusion; the enhanced chest fracture scan projection includes the enhanced chest fracture scan projection in the fracture area and the enhanced chest fracture scan projection after boundary fusion.

[0007] In an implementation of the first aspect, the process of generating a simulated chest AP image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the slightly distorted chest AP image includes: performing cross-modal data augmentation on the chest AP image based on the chest fracture scan projection to obtain a preliminary simulated chest AP image; performing homologous but different-shape sample augmentation on the chest AP image to obtain the slightly distorted chest AP image; and obtaining the simulated chest AP image based on the preliminary simulated chest AP image and the slightly distorted chest AP image.

[0008] In an implementation of the first aspect, the training process of the cross-modal network model also includes: inputting the training chest AP image and the training simulated chest AP image into the cross-modal network model to obtain a probability value of a fracture area; comparing the probability value with a positive judgment threshold to obtain a positive area, where the positive area is an area where a fracture exists; and adaptively adjusting the positive judgment threshold based on the positive area to obtain a chest fracture detection image containing a simply connected area, where the simply connected area is a continuous positive area.

[0009] In an implementation of the first aspect, the cross-modal network model also includes a backbone network, which is used to extract image features of the chest AP image and the simulated chest AP image and input them into the sample-level segmentation network and the region-level segmentation network. The sample-level segmentation network and the region-level segmentation network are trained in parallel and output sample-level fracture detection results and region-level fracture detection results at the same time.

[0010] In an implementation of the first aspect, the process of training the cross-modal network model also includes: training based on the training chest AP image to obtain a preliminary cross-modal network model; and training the preliminary cross-modal network model based on the training simulated chest AP image.

[0011] In an implementation of the first aspect, the process of obtaining a chest AP image from a chest image further includes: identifying the chest image to obtain a chest image with missing metadata; and performing secondary identification on the chest image with missing metadata using a sample-level classification model to obtain the chest AP image.

[0012] In a second aspect, the present application provides a cross-modal enhanced fracture detection system, the fracture detection system comprising: a plain film acquisition module for acquiring a chest image and a chest scan projection; a chest AP image acquisition module for acquiring a chest AP image from the chest image; a chest fracture scan projection acquisition module for preprocessing the chest scan projection to obtain a chest fracture scan projection; a dynamic feature enhancement module for performing fracture area dynamic feature enhancement on the chest fracture scan projection to obtain an enhanced chest fracture scan projection; a simulated image generation module for generating a simulated chest AP image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and a slightly distorted chest AP image; and a cross-modal network model training module. A module is used to process the chest AP image and the simulated chest AP image using a cross-modal network model to obtain a chest fracture detection image, wherein the chest fracture detection image includes an overall chest fracture detection image and a local chest fracture detection image; wherein the process of using the cross-modal network model for training to obtain the chest fracture detection image includes: performing sample-level annotation on the training chest AP image to obtain a sample-level annotation result; performing region-level annotation on the training simulated chest AP image to obtain a region-level annotation result; training a sample-level segmentation network based on the training chest AP image and the sample-level annotation result; and training a region-level segmentation network using the training simulated chest AP image and the region-level annotation result.

[0013] In a third aspect, the present application provides an electronic device, comprising: a memory on which a computer program is stored; and a processor, communicatively connected to the memory, for executing the computer program to implement the above-mentioned cross-modal enhanced fracture detection method.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned cross-modal enhanced fracture detection method.

[0015] As described above, the cross-modal enhanced fracture detection method, system, medium and electronic device described in the present application have the following beneficial effects:

[0016] The fracture detection method based on cross-modal enhancement provided by the present application can combine the chest image and the chest scan projection for cross-modal enhancement, and add dynamic features to the fracture area. The cross-modal enhanced chest anteroposterior image and chest fracture scan projection are trained using a cross-modal network model. During the training process, the overall chest fracture detection image is output through a sample-level segmentation network, and the local chest fracture detection image is output through a region-level segmentation network, thereby obtaining a chest fracture detection image. The fracture detection method provided by the present application can improve the accuracy and diagnosis and treatment efficiency of chest fracture detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a schematic diagram of an application scenario of the fracture detection method based on cross-modal enhancement described in an embodiment of the present application.

[0018] Figure 2 Shown is a process schematic diagram of the fracture detection method based on cross-modal enhancement described in an embodiment of the present application.

[0019] Figure 3 Shown is a schematic diagram of the process of obtaining a cross-modal enhanced dataset as described in an embodiment of the present application.

[0020] Figure 4 Shown is a schematic diagram of the process of cross-modal network model training described in an embodiment of the present application.

[0021] Figure 5 Shown is a schematic diagram of the process of obtaining enhanced chest fracture scan projections described in an embodiment of the present application.

[0022] Figure 6 Shown is a schematic diagram of the process of obtaining a chest fracture detection image as described in an embodiment of the present application.

[0023] Figure 7 Shown is a schematic diagram of the structure of a fracture detection system based on cross-modal enhancement described in an embodiment of the present application.

[0024] Figure 8 Shown is a schematic diagram of the structure of an electronic device described in an embodiment of the present application.

[0025] Component number description

[0026] 10 Human-computer interaction system

[0027] 101 DR image acquisition equipment

[0028] 102 CT Scanning Equipment

[0029] 103 Cross-modal network model

[0030] 1 Fracture detection system based on cross-modal enhancement

[0031] 11. Flat Film Acquisition Module

[0032] 12 Chest AP image acquisition module

[0033] 121 Identification Unit

[0034] 122 Secondary recognition unit

[0035] 13. Chest fracture scan projection acquisition module

[0036] 14 Dynamic Feature Enhancement Module

[0037] 141 Dynamic Adjustment Unit

[0038] 142 Boundary Optimization Unit

[0039] 15. Simulation image generation module

[0040] 151 Data Augmentation Unit

[0041] 152 Sample augmentation units

[0042] 153 Image Generation Unit

[0043] 16 Cross-modal network model training module

[0044] 2 Electronic devices

[0045] 21 Memory

[0046] 22 Processor

[0047] 23 Display

[0048] Steps S11 to S16

[0049] Steps S31 to S33

[0050] Steps S51 to S53 DETAILED DESCRIPTION

[0051] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0052] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0053] When the chest is injured, rib fractures are quite common. With age, the elasticity of the ribs gradually decreases, making them more prone to fractures. Accurate rib detection and positioning can provide a basis for timely interventional treatment, reduce patient pain, and improve prognosis. Digital radiography (DR) has been widely used to image the internal structure or tissue of the human body, and can be further analyzed for medical diagnosis. Data augmentation is an indispensable part of deep learning in recent years. It can ensure the full utilization of existing data sets and improve the accuracy that neural networks can achieve in various downstream tasks. However, most current implementation methods use affine-based augmentation preprocessing, focusing on the design of neural networks, and insufficient utilization of existing data. With the improvement of medical conditions, the coverage of CT scans is getting higher and higher. It has higher imaging resolution and can perform 3D reconstruction. However, the current methods for diagnosing sternal fractures for DR chest radiographs fail to combine high-quality CT imaging data and use controllable adaptive strategies for cross-modal hybrid augmentation, thus limiting the performance of downstream neural networks.

[0054] Traditional cross-modal dataset training usually uses traditional transfer learning and direct CT projection. However, traditional cross-modal datasets in traditional transfer learning usually use multiple identical head networks, which are first pre-trained based on the auxiliary modality and then trained on the target modality. This method has strong versatility, but the training burden is increased and the training efficiency is limited because the modality transfer task is incorporated into the adaptive learning of the neural network. Direct CT projection uses the projection method to convert the image into the DR modality, and then merges it with the real DR data into a larger dataset. This dataset construction method has low utilization of CT sequence data and lacks precise projection correction. In the case of insufficient correction, the direct use of transfer learning will limit the maximum accuracy that the model can achieve.

[0055] At least to address the above-mentioned problems, the following embodiments of the present application provide a fracture detection method, system, medium and electronic device based on cross-modal enhancement.

[0056] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0057] Figure 1The following is a schematic diagram of an application scenario of a fracture detection method based on cross-modal enhancement in one embodiment of the present application. Figure 1 As shown, the human-computer interaction system 10 includes a DR image acquisition device 101, a CT scanning device 102, and a cross-modal network model 103. The DR image acquisition device uses digital X-ray photography technology to acquire DR images of patients, and the CT scanning device acquires a CT sequence of CT projections of patients, and performs cross-modal enhancement on the DR images based on the CT projections to acquire a cross-modal enhanced data set, and the cross-modal enhanced data set is input into the cross-modal network model for training, and finally an image of the position of the patient's fracture is obtained.

[0058] Figure 2 The process diagram of the fracture detection method based on cross-modal enhancement in one embodiment of the present application is shown. Figure 2 As shown, the fracture detection method based on cross-modal enhancement includes:

[0059] S11, obtaining a chest image and a chest scan projection. The chest image is a DR image identified by DR (digital radiography) scanning, and the chest scan projection is a CT sequence obtained by CT scanning (Computed Tomography).

[0060] S12, acquiring a chest frontal image from the chest image.

[0061] S13, preprocessing the chest scan projection to obtain a chest fracture scan projection.

[0062] S14, performing fracture region dynamic feature enhancement on the chest fracture scan projection to obtain an enhanced chest fracture scan projection.

[0063] S15, generating a simulated chest AP image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the slightly distorted chest AP image.

[0064] S16, using a cross-modal network model to process the chest anteroposterior image and the simulated chest anteroposterior image to obtain a chest fracture detection image, wherein the cross-modal network model includes a sample-level segmentation network and a region-level segmentation network; the chest fracture detection image includes an overall chest fracture detection image and a local chest fracture detection image, for details, please refer to Figure 3 .

[0065] Among them, see Figure 4 ,The process of training the cross-modal network model to obtain chest fracture detection images includes:

[0066] S21, performing sample-level annotation on the training chest frontal image to obtain a sample-level annotation result.

[0067] S22, performing region-level annotation on the training simulated chest frontal image to obtain a region-level annotation result.

[0068] S23, training a sample-level segmentation network based on the training chest AP image and the sample-level annotation result.

[0069] S24, training a region-level segmentation network using the training simulated chest AP image and the region-level annotation result.

[0070] According to the above description, the fracture detection method based on cross-modal enhancement provided by the present application can combine the chest image and the chest scan projection for cross-modal enhancement, and add dynamic features to the fracture area. The cross-modal enhanced chest anteroposterior image and chest fracture scan projection are trained using a cross-modal network model. During the training process, the overall chest fracture detection image is output through the sample-level segmentation network, and the local chest fracture detection image is output through the region-level segmentation network, so as to obtain the chest fracture detection image. The fracture detection method provided by the present application can improve the accuracy and diagnosis and treatment efficiency of chest fracture detection.

[0071] In one embodiment of the present application, the process of obtaining a chest AP image from the chest image includes: obtaining DR (digital radiography) scan data and DR scan metadata; and determining the patient's posture relative to the detector when being tested based on the DR scan data and the DR scan metadata.

[0072] Exemplarily, the DR scan metadata includes: data accuracy and range, instrument and radiation parameters, patient disease and symptom description (after anonymization), and patient scanning posture. The metadata is used to process the DR scan images of the patient's chest anteroposterior, sternal lateral, other parts, and waste films to obtain a chest anteroposterior image.

[0073] The process of acquiring a chest anteroposterior image from the chest image further includes: identifying the chest image to acquire a chest image with missing metadata; and performing secondary identification on the chest image with missing metadata using a sample-level classification model to acquire the chest anteroposterior image.

[0074] Exemplarily, the sample-level classification model can be obtained by training a ResNet18 neural network.

[0075] In one embodiment of the present application, the process of preprocessing the chest scan projection includes: normalizing the chest scan projection (CT sequence) based on the metadata of the standard Dicom specification, and correcting and normalizing the non-standard CT scan sequence to align the imaging range and spatial definition. Among them, the normalization attributes include: valid HU range and window, sequence reconstruction Spacing, sequence Z-axis order. Among them, samples that meet any of the following conditions will be discarded: chest rib fracture lesions are not scanned, the number of slices is too small (including three or less ribs), and the HU value distribution is abnormal.

[0076] Figure 5 The figure shows a schematic diagram of the process of obtaining enhanced chest fracture scan projection in one embodiment of the present application. Figure 5 As shown, the process of performing dynamic feature enhancement of the fracture area on the chest fracture scan projection to obtain enhanced chest fracture scan projection includes:

[0077] S31, dynamically adjusting the image features of the fracture area based on the annotation of the chest fracture scan projection to obtain an enhanced chest fracture scan projection in the fracture area.

[0078] In some embodiments, the process of dynamic feature enhancement of the fracture region includes: first, by annotating the CT reconstruction sequence to obtain a region of interest (ROI), that is, a region near the fracture tissue, the region can be defined as The corresponding area on the projection is Secondly, the ROI region is dynamically enhanced by using a dynamically adjustable intensity enhancement and transformation method, and the dynamically adjustable intensity method includes: histogram equalization, nonlinear mapping, Gaussian filtering and sharpening.

[0079] Specifically, histogram equalization can enhance the local contrast of the image, making the details in the RoI region more obvious. Nonlinear mapping can adjust the nonlinear relationship of pixel values ​​to enhance or weaken the features of a specific region. Gaussian filtering and sharpening can enhance the edge and texture features in the RoI region.

[0080] S32, optimizing the boundary features of the fracture area by using a fuzzy function to obtain an enhanced chest fracture scan projection after boundary fusion.

[0081] In some embodiments, the process of optimizing the boundary features of the fracture area using the fuzzy function includes: optimizing the boundary features of the fracture area using the Poisson distribution. The boundary pixels are post-processed to minimize the gradient difference between the fracture area and the background DR image, so that it can be smoothly merged into the original image with reasonable boundary features.

[0082] By adaptively enhancing the fracture area and post-processing the fracture boundary area, the consistency between the fracture ROI area and other areas can be weakened, the consistency with the image can be improved, and the cost of development and tuning can be saved. During the training process, the image features of the fracture area are dynamically adjusted in a strong-first-weak-later manner to ensure that the cross-modal network model can quickly locate a reasonable model parameter range in the early stage of training, and then quickly reduce the enhancement intensity to increase the difficulty of the sample. When the model is roughly fitted, the visual accuracy observation data can be used to erase certain features using anti-enhancement to further increase the difficulty.

[0083] S33, the enhanced chest fracture scan projection includes the enhanced chest fracture scan projection within the fracture area and the enhanced chest fracture scan projection after the boundary fusion.

[0084] In one embodiment of the present application, the process of generating a simulated chest AP image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the slightly distorted chest AP image includes:

[0085] S41, performing cross-modal data augmentation on the chest AP image based on the chest fracture scan projection to obtain a preliminary simulated chest AP image.

[0086] In some embodiments, the process of generating a DR image based on CT projection is: using the ray projection method to simulate X-rays from a virtual ray source through the object to be tested (CT reconstruction sequence body), and projecting to the detector plane perpendicular to the central axis of the ray. In this process, the intersection of the point source and each CT slice in the CT volume data can be calculated to obtain the corresponding CT value, and the CT value obtained by accumulating the entire path is the DR image pixel value of the corresponding point on the detector. The specific description process can be expressed as:

[0087] Assume that the virtual ray source is The virtual detector plane is Perpendicular to the central axis of the ray, the object to be measured (CT volume data) is a three-dimensional volume Each voxel has a CT value μ(x,y,z). Each pixel on Define a line from Departure, passing Rays Its The intersection point can be determined by the following equation ( is the position vector of the ray source, is the direction vector of the ray, is the ray vector, a scalar Represents the distance from the ray source to a point on the ray):

[0088]

[0089] Using the nearest neighbor method (Euclidean distance), determine and The relationship between

[0090]

[0091] The virtual imaging pixel value is recorded as It can be achieved by The integral of is calculated to get:

[0092]

[0093] During CT reconstruction, the software included with the instrument has The ray attenuation of is corrected, so the virtual image obtained by integration is also considered to be corrected, but there is no similar correction in DR imaging. In order to obtain a better domain registration effect, it is necessary to To perform similar attenuation correction, the present invention adopts the following function ( is the incident ray intensity factor):

[0094]

[0095] S42, performing homologous but different-shape sample augmentation on the chest AP image to obtain the slightly distorted chest AP image.

[0096] For example, by adjusting the virtual ray source to Virtual detector plane The spatial position of the image is used to obtain multiple chest image samples of the same source but different shapes, and the degree of distortion is controlled within the range of ±90°. By reasonably introducing samples of the same source but different shapes, the model can grasp the required features more accurately and suppress overfitting.

[0097] S43, acquiring the simulated chest AP image based on the preliminary simulated chest AP image and the slightly distorted chest AP image.

[0098] Figure 6 The figure shows a schematic diagram of the process of obtaining a chest fracture detection image in one embodiment of the present application. Figure 6 As shown, the training process of the cross-modal network model also includes:

[0099] S51, inputting the training chest AP image and the training simulated chest AP image into a cross-modal network model to obtain a probability value of a fracture area.

[0100] S52, comparing the probability value with a positive judgment threshold to obtain a positive area, where the positive area is an area where a fracture exists.

[0101] S53, adaptively adjusting the positive judgment threshold based on the positive area to obtain a chest fracture detection image containing a simply connected area, where the simply connected area is a continuous positive area.

[0102] In some embodiments, the chest anteroposterior image and the simulated chest anteroposterior image are annotated at the sample level and input into the fully convolutional backbone network to generate a feature map. Each feature embedding in the feature map passes through a 1×1 convolution layer to output the probability value of the fracture area and the fracture area boundary. The probability value is compared with the positive judgment threshold to obtain the positive area. The positive judgment threshold is continuously increased until all positive areas are reduced to a single connected area. By performing the above-mentioned sample weak supervision operation on the image, no region anchor box or pixel-level annotation is required, which greatly reduces the annotation cost.

[0103] In one embodiment of the present application, the cross-modal network model also includes a backbone network, which is used to extract image features of the chest AP image and the simulated chest AP image and input them into the sample-level segmentation network and the region-level segmentation network. The sample-level segmentation network and the region-level segmentation network are trained in parallel and output sample-level fracture detection results and region-level fracture detection results at the same time.

[0104] In some embodiments, the enhanced chest fracture scan projection is annotated and the annotation is projected onto the virtual detector plane. The region-level annotations are obtained from the network, which have higher information entropy than sample-level annotations. A segmentation head subnetwork independent of the backbone network is used to output pixel-level fitting targets. As the training progresses, the segmentation head subnetwork is gradually discarded, and the frequency of using region-level annotations as training targets is gradually reduced. This subnetwork will not be used in the final inference. Through multi-objective warm-up continuous learning, the general direction of parameter optimization can be established at the beginning of training, thereby helping the backbone network reduce the risk of gradient instability caused by sample-level coarse-grained annotations. In cross-modal data, the continuous learning strategy of using multiple parallel segmentation head networks can increase the width of the neural network.

[0105] In one embodiment of the present application, the training process of the cross-modal network model further includes:

[0106] S61, training is performed based on the training chest AP image to obtain a preliminary cross-modal network model.

[0107] S62, training a preliminary cross-modal network model based on the training simulated chest AP image.

[0108] Specifically, instead of using samples generated by CT simulation at the beginning of training, real DR images are used directly, which can ensure that the fitting direction of the model parameters at the beginning of training conforms to the characteristics of the DR modality, and avoid the risk of reducing the efficiency of subsequent training due to the premature introduction of simulated samples. As training progresses, the utilization rate of simulated samples will continue to increase, and eventually reach the proportion of CT samples in all samples. Through the above-mentioned progressive augmentation strategy, the cross-modal network model can be effectively learned.

[0109] In one embodiment of the present application, the fracture detection method based on cross-modal enhancement also includes system integration and output. By integrating the cross-modal network model into an interactive human-computer system, its service process includes: deployment and startup readiness, receiving DR scan metadata, performing neural network reasoning, visualizing prediction results, and interactive display. The above-mentioned interactive human-computer system is deployed and applied in clinical practice, which can responsively receive the patient's DR imaging data and return interactive prediction results in seconds, thereby improving the efficiency of diagnosis and treatment for doctors and assisting in improving the accuracy of diagnosis and treatment.

[0110] Next, the fracture detection method based on cross-modal enhancement provided by the embodiment of the present application will be introduced in detail through a specific example. It should be noted that the content in this example is only used to explain and illustrate the fracture detection method based on cross-modal enhancement provided by the embodiment of the present application, rather than to limit the scope of protection of the present application. In specific applications, the corresponding steps can be added or deleted based on the example according to actual needs. The process of the fracture detection method based on cross-modal enhancement in this example includes the following steps:

[0111] Step 1: The patient undergoes a digital radiography (DR) scan and a CT scan to obtain a chest DR image and a chest CT sequence. The chest DR image is analyzed and identified to obtain a chest anteroposterior image.

[0112] Step 2: Annotate the fracture area in the CT sequence, enhance the image features of the fracture area through a dynamically adjustable intensity method, and use Poisson fuzzification technology to optimize the boundary of the fracture area so that it can be smoothly integrated with the background DR image.

[0113] Step 3: Perform cross-modal data augmentation. Based on CT projection, use the ray transmission method to simulate and generate DR images, that is, preliminarily simulate chest anteroposterior images. Perform sample augmentation. By slightly distorting the simulated DR images, samples with the same source but different shapes are generated to expand the training data set of the cross-modal network model.

[0114] Step 4: Use the cross-modal network model for training. First, perform sample-level weak supervision on the chest anteroposterior image and the simulated chest anteroposterior image, and input them into the cross-modal network model after sample-level annotation. Continuous learning and training are performed through parallel sample-level segmentation network and region-level segmentation network to output sample-level fracture detection results and region-level fracture detection results respectively. In the early stage of training, real DR images are used for training, and then simulated DR images are gradually added, and the generalization ability of the model is improved through a progressive augmentation strategy.

[0115] In summary, the fracture detection method based on cross-modal enhancement provided by the present application combines CT and DR image data using cross-modal enhancement technology, significantly improving the accuracy and efficiency of rib fracture detection. And through the CT projection DR image generation technology enhanced by micro-distortion, the projection data of the CT modality is used to augment and enhance the image data of the DR modality, which expands the data source of the neural network for DR analysis, and can also obtain more abundant and diverse samples from a single subject, effectively improving the training accuracy of the cross-modal network model. At the same time, the sample-level annotation of the chest anteroposterior image and the simulated chest anteroposterior image, as well as the parallel continuous learning strategy of the sample-level network and the regional-level network, significantly reduce the cost of manual annotation and alleviate the problem of unstable data distribution in cross-modal training. In addition, by enhancing the dynamic features of the fracture area and gradually introducing the progressive augmentation strategy of the simulated chest anteroposterior image, the robustness and training efficiency of the cross-modal network model training are improved. Finally, the integrated human-computer interaction system can realize the rapid deployment of clinical applications, provide doctors with interactive prediction results returned in seconds, and provide better diagnosis and treatment efficiency, accuracy and better patient treatment quality.

[0116] The protection scope of the fracture detection method based on cross-modal enhancement described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0117] An embodiment of the present application also provides a fracture detection system based on cross-modal enhancement, which can implement the fracture detection method based on cross-modal enhancement described in the present application. However, the implementation device of the fracture detection method based on cross-modal enhancement described in the present application includes but is not limited to the structure of the fracture detection system based on cross-modal enhancement listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0118] Figure 7 The structure diagram of the fracture detection system based on cross-modal enhancement in one embodiment of the present application is shown. Figure 7As shown, the fracture detection system 1 based on cross-modal enhancement includes: a plain film acquisition module 11, a chest AP image acquisition module 12, a chest fracture scan projection acquisition module 13, a dynamic feature enhancement module 14, a simulated image generation module 15, and a cross-modal network model training module 16. Among them, the plain film acquisition module 11 is used to acquire a chest image and a chest scan projection; the chest AP image acquisition module 12 is used to identify the chest image to acquire a chest AP image; the chest fracture scan projection acquisition module 13 is used to clean the chest scan projection to acquire a chest fracture scan projection; the dynamic feature enhancement module 14 is used to perform dynamic feature enhancement of the fracture area on the chest fracture scan projection to acquire an enhanced chest fracture scan projection; the simulated image generation module 15 is used to generate a simulated chest AP image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the slightly distorted chest AP image; the cross-modal network model training module 16 is used to and the simulated chest anteroposterior image, and training with a cross-modal network model to obtain a chest fracture detection image, wherein the chest fracture detection image includes an overall chest fracture detection image and a local chest fracture detection image; wherein the process of training with a cross-modal network model to obtain a chest fracture detection image includes: performing sample-level annotation on the chest anteroposterior image and the simulated chest anteroposterior image, and then inputting them into a sample-level segmentation network to output the overall chest fracture detection image; performing region-level annotation on the simulated chest anteroposterior image, and then inputting them into a region-level segmentation network to output the local chest fracture detection image; the chest fracture detection image includes the overall chest fracture detection image and the local chest fracture detection image.

[0119] The chest anteroposterior image acquisition module 12 includes a recognition unit 121 and a secondary recognition unit 122. The recognition unit 121 is used to recognize the chest image to obtain the chest image with missing metadata. The secondary recognition unit 122 is used to perform secondary recognition on the chest image with missing metadata using a sample-level classification model to obtain the chest anteroposterior image.

[0120] The dynamic enhancement module 14 includes a dynamic adjustment unit 141 and a boundary optimization unit 142. The dynamic adjustment unit 141 is used to dynamically adjust the image features of the fracture area based on the annotation of the chest fracture scan projection to obtain an enhanced chest fracture scan projection in the fracture area. The boundary optimization unit 142 is used to optimize the boundary features of the fracture area using a fuzzy function to obtain an enhanced chest fracture scan projection after boundary fusion; the enhanced chest fracture scan projection includes the enhanced chest fracture scan projection in the fracture area and the enhanced chest fracture scan projection after boundary fusion.

[0121] The simulation image generation module 15 includes a data augmentation unit 151, a sample augmentation unit 152, and an image generation unit 153. The data augmentation unit 151 is used to perform cross-modal data augmentation on the chest AP image based on the chest fracture scan projection to obtain a preliminary simulated chest AP image. The sample augmentation unit 152 is used to perform homologous but different-shaped sample augmentation on the chest AP image to obtain the slightly distorted chest AP image. The image generation unit 153 is used to obtain the simulated chest AP image based on the preliminary simulated chest AP image and the slightly distorted chest AP image.

[0122] In one embodiment of the present application, the training process of the cross-modal network model further includes:

[0123] Inputting the training chest AP image and the training simulated chest AP image into a cross-modal network model to obtain a probability value of a fracture area;

[0124] Comparing the probability value with the positive judgment threshold to obtain a positive area, where the positive area is an area where a fracture exists;

[0125] The positive judgment threshold is adaptively adjusted based on the positive area to obtain a chest fracture detection image containing a simply connected area, where the simply connected area is a continuous positive area.

[0126] In one embodiment of the present application, the cross-modal network model also includes a backbone network, which is used to extract image features of the chest AP image and the simulated chest AP image and input them into the sample-level segmentation network and the region-level segmentation network. The sample-level segmentation network and the region-level segmentation network are trained in parallel and output sample-level fracture detection results and region-level fracture detection results at the same time.

[0127] In one embodiment of the present application, the process of training the cross-modal network model further includes:

[0128] Performing training based on the training chest anterior image to obtain a preliminary cross-modal network model;

[0129] Based on the training simulated chest AP images, a preliminary cross-modal network model is trained.

[0130] It should be noted that Figure 7 The modules in the fracture detection system 1 based on cross-modal enhancement are shown in FIG. Figure 2 The steps in the fracture detection method based on cross-modal enhancement correspond to each other and will not be described in detail here.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0132] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0133] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0134] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by the processor, the fracture detection method based on cross-modal enhancement provided by the present application embodiment is implemented. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state hard disk (SSD)), etc.

[0135] The embodiment of the present application may also provide an electronic device. Figure 8 The diagram shows a schematic diagram of the structure of an electronic device 2 in an embodiment of the present application. Figure 8 As shown, in this embodiment, the electronic device 2 includes a memory 21 and a processor 22 .

[0136] The memory 21 is used to store computer programs. In some possible implementations, the memory 21 may include: ROM, RAM, a disk, a USB flash drive, a memory card, or an optical disk, etc., various media that can store program codes.

[0137] In the embodiment of the present application, the memory 21 may include a computer system readable medium in the form of a volatile memory, such as a RAM and / or a cache memory. The electronic device 2 may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 21 may include at least one program product having a set (e.g., at least one) of program modules, which are configured to perform the functions of the various embodiments of the present application.

[0138] The processor 22 is connected to the memory 21 and is used to execute the computer program stored in the memory 21 so that the electronic device 2 executes the fracture detection method based on cross-modal enhancement.

[0139] Exemplarily, the processor 22 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. In other embodiments, the processor 22 may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0140] In some implementations, the electronic device 2 provided in the embodiment of the present application may further include a display 23. The display 23 is communicatively connected to the memory 21 and the processor 22, and is used to display a related graphical user interface (GUI) of the fracture detection method based on cross-modal enhancement.

[0141] In the embodiment of the present application, the display 23 may include a display screen (display panel). In some implementations, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. In addition, the display 23 may also be a touch panel (touch screen, touch screen), and the touch panel may include a display screen and a touch-sensitive surface. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 22 to determine the type of touch event, and then the processor 22 provides a corresponding visual output on the display device according to the type of touch event.

[0142] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0143] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A fracture detection method based on cross-modal enhancement, characterized in that: The fracture detection method comprises: Acquire chest images and chest scan projections; Acquire a chest frontal image from the chest image; Preprocessing the chest scan projection to obtain a chest fracture scan projection; Performing fracture area dynamic feature enhancement on the chest fracture scan projection to obtain an enhanced chest fracture scan projection; Generate a simulated chest AP image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the slightly distorted chest AP image; The chest anteroposterior image and the simulated chest anteroposterior image are processed by a cross-modal network model to obtain a chest fracture detection image, wherein the chest fracture detection image includes an overall chest fracture detection image and a local chest fracture detection image; the cross-modal network model includes a sample-level segmentation network and a region-level segmentation network; The process of using the cross-modal network model for training to obtain chest fracture detection images includes: Perform sample-level annotation on the training chest frontal images to obtain sample-level annotation results; Performing region-level annotation on the training simulated chest frontal image to obtain region-level annotation results; Training a sample-level segmentation network based on the training chest anteroposterior image and the sample-level annotation result; The region-level segmentation network is trained using the training simulated chest anteroposterior image and the region-level annotation result.

2. The fracture detection method according to claim 1, characterized in that: The process of performing dynamic feature enhancement of the fracture region on the chest fracture scan projection to obtain an enhanced chest fracture scan projection includes: Dynamically adjusting the image features of the fracture area based on the annotation of the chest fracture scan projection to obtain an enhanced chest fracture scan projection in the fracture area; The boundary features of the fracture area are optimized using fuzzy functions to obtain enhanced chest fracture scan projections after boundary fusion. The enhanced chest fracture scan projection includes the enhanced chest fracture scan projection within the fracture area and the enhanced chest fracture scan projection after the boundary fusion.

3. The fracture detection method according to claim 1, characterized in that: The process of generating a simulated chest AP image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the slightly distorted chest AP image comprises: Performing cross-modal data augmentation on the chest AP image based on the chest fracture scan projection to obtain a preliminary simulated chest AP image; Performing homologous but different-shape sample augmentation on the chest anteroposterior image to obtain the slightly distorted chest anteroposterior image; The simulated chest AP image is acquired based on the preliminary simulated chest AP image and the slightly distorted chest AP image.

4. The fracture detection method according to claim 1, characterized in that: The training process of the cross-modal network model also includes: Inputting the training chest AP image and the training simulated chest AP image into a cross-modal network model to obtain a probability value of a fracture area; Comparing the probability value with the positive judgment threshold to obtain a positive area, where the positive area is an area where a fracture exists; The positive judgment threshold is adaptively adjusted based on the positive area to obtain a chest fracture detection image containing a simply connected area, where the simply connected area is a continuous positive area.

5. The fracture detection method according to claim 1, characterized in that: The cross-modal network model also includes a backbone network, which is used to extract image features from the chest AP image and the simulated chest AP image and input them into the sample-level segmentation network and the region-level segmentation network. The sample-level segmentation network and the region-level segmentation network are trained in parallel and simultaneously output sample-level fracture detection results and region-level fracture detection results.

6. The fracture detection method according to claim 1, characterized in that: The training process of the cross-modal network model also includes: Performing training based on the training chest anterior image to obtain a preliminary cross-modal network model; Based on the training simulated chest AP images, a preliminary cross-modal network model is trained.

7. The fracture detection method according to claim 1, characterized in that: The process of acquiring a chest frontal image from the chest image further includes: identifying the chest image to obtain a chest image with missing metadata; A sample-level classification model is used to perform secondary recognition on the chest image with missing metadata to obtain the chest normal image.

8. A cross-modal enhanced fracture detection system, characterized in that: The fracture detection system comprises: A plain film acquisition module, used for acquiring chest images and chest scan projections; A chest anteroposterior image acquisition module, used for acquiring a chest anteroposterior image from the chest image; A chest fracture scan projection acquisition module, used for preprocessing the chest scan projection to acquire a chest fracture scan projection; A dynamic feature enhancement module, used for performing dynamic feature enhancement of the fracture area on the chest fracture scan projection to obtain an enhanced chest fracture scan projection; A simulated image generation module, used to generate a simulated chest AP image based on the chest fracture scan projection, the enhanced chest fracture scan projection, and the slightly distorted chest AP image; a cross-modal network model training module, used for processing the chest AP image and the simulated chest AP image using a cross-modal network model to obtain a chest fracture detection image, wherein the chest fracture detection image includes an overall chest fracture detection image and a local chest fracture detection image; The process of using the cross-modal network model for training to obtain chest fracture detection images includes: Perform sample-level annotation on the training chest frontal images to obtain sample-level annotation results; Performing region-level annotation on the training simulated chest frontal image to obtain region-level annotation results; Training a sample-level segmentation network based on the training chest anteroposterior image and the sample-level annotation result; The region-level segmentation network is trained using the training simulated chest anteroposterior image and the region-level annotation result.

9. An electronic device, characterized in that: The electronic device comprises: a memory having a computer program stored thereon; A processor is communicatively connected to the memory, and is used to execute the computer program to implement the cross-modal enhanced fracture detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the cross-modal enhanced fracture detection method described in any one of claims 1 to 7 is implemented.

Images