Children leukemia screening method based on multi-modal data and related device
By employing a multimodal data fusion method and utilizing a cross-attention mechanism to achieve deep semantic alignment and complementary feature fusion of image and tabular data, the accuracy and efficiency issues of existing leukemia screening methods are resolved, thereby improving the accuracy and automation level of childhood leukemia screening.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUYI UNIV
- Filing Date
- 2025-12-26
- Publication Date
- 2026-05-01
AI Technical Summary
Existing leukemia screening methods are mainly limited to a single signal space or a single data modality, ignoring the potential complementarity and synergistic effect between different data sources. This results in limited sensitivity and specificity in identifying primitive/immature cells, and the screening accuracy is difficult to meet high clinical standards. Furthermore, existing fusion strategies have failed to achieve deep semantic alignment and interaction of cross-modal features, leading to high false positive rates and heavy manual retesting workload.
A multimodal data-based screening method for childhood leukemia was adopted. Visual features of different medical images were extracted by a visual feature extractor, and attention values between tables, images, and between images were calculated using a cross-attention mechanism to achieve deep semantic alignment and complementary feature fusion. Finally, multi-level classification was performed by a fused feature classification head to obtain the screening results for childhood leukemia.
It achieves comprehensive complementarity of multimodal information, improves the sensitivity and specificity of identifying primitive cells, reduces the false positive rate, enhances the accuracy and efficiency of screening, and reduces the workload of manual re-examination.
Smart Images
Figure CN121962824A_ABST
Abstract
Description
A method and related device for screening childhood leukemia based on multimodal data Technical Field
[0001] This application relates to the field of image analysis, and more particularly to a method and related apparatus for screening childhood leukemia based on multimodal data. Background Technology
[0002] Leukemia screening primarily employs blood cell analysis, which utilizes a blood analyzer to output basic report parameters, research parameters, and white blood cell scatter plots. When the blood analyzer indicates abnormalities such as blast cells, manual microscopic examination is still required to ensure the reliability of the results. In actual clinical practice, the instrument alarm indicating blast cells has low specificity, a high re-examination rate, and the manual microscopic examination is extremely labor-intensive, limited by manpower and capabilities, and has potential drawbacks such as high rates of missed and false positives, making it difficult to improve work efficiency.
[0003] Furthermore, leukemia screening is primarily limited to a single signal space or a single data modality. It often focuses on a single data source, such as analyzing only scattering / fluorescence signals or processing only cell microscopic images, failing to achieve a deep fusion of visual features from scatter plots and structured blood routine parameters. This single-modality analysis approach ignores the potential complementarity and synergistic effects between different data sources, resulting in limited sensitivity and specificity for identifying primitive / immature cells, and screening accuracy that fails to meet high clinical standards.
[0004] Secondly, existing blood cell analysis systems based on traditional rules or thresholds have low specificity and high false positive rates in their screening results, leading to arduous and inefficient manual retesting.
[0005] Furthermore, even when attempting multimodal fusion methods, their fusion strategies are typically quite simple, such as feature concatenation or weighted summation, failing to achieve deep semantic alignment and interaction across modal features. This shallow fusion makes it difficult to effectively mine the complex nonlinear relationships between image features and tabular data, limiting the model's ability to learn discriminative features from multi-source heterogeneous data. Summary of the Invention
[0006] The following is an overview of the topics described in detail in this article.
[0007] The purpose of this application is to at least partially solve one of the technical problems existing in the related technologies. The embodiments of this application provide a method and related device for screening childhood leukemia based on multimodal data.
[0008] An embodiment of the first aspect of this application provides a method for screening childhood leukemia based on multimodal data, comprising: acquiring a first medical image, a second medical image, and routine blood test data; inputting the first medical image, the second medical image, and the routine blood test data into a childhood leukemia screening model, wherein the first medical image and the second medical image belong to different image modalities, and the routine blood test data belongs to a table modality; extracting visual features of the first medical image and visual features of the second medical image respectively through different branches of the visual feature extractor of the childhood leukemia screening model; and concatenating the visual features of the first medical image and the visual features of the second medical image. The visual features of the images are used to obtain joint visual features. The feature fusion module of the childhood leukemia screening model calculates the attention values from the table to the image, the image to the table, and the inter-image attention values based on the visual features of the first medical image, the visual features of the second medical image, the joint visual features, and the table modal features. The table-to-image, image-to-table, and inter-image attention values are then fused to obtain fused features. The global semantic features of the fused features are extracted by the fused feature classification head of the childhood leukemia screening model. The global semantic features are then classified at multiple levels to obtain the childhood leukemia screening results.
[0009] According to certain embodiments of the first aspect of this application, the visual feature extractor is provided with a first branch and a second branch in parallel, the first branch being used to extract visual features of the first medical image and the second branch being used to extract visual features of the second medical image.
[0010] According to certain embodiments of the first aspect of this application, the table-to-image attention value is calculated as follows: using table modal features as queries and joint visual features as keys and values, cross-attention calculation is performed to focus hematological parameters on relevant cell morphology evidence to generate enhanced table features; the enhanced table features are added to the table modal features through residual connections to obtain the table-to-image attention value.
[0011] According to certain embodiments of the first aspect of this application, the image-to-table attention value is calculated as follows: using joint visual features as the query and enhanced table features as the key and value, enhanced visual features are generated by deeply embedding quantification indicators into the visual semantic representation through cross-attention calculation; the enhanced visual features are added to the joint visual features through residual connection to obtain the image-to-table attention value.
[0012] According to certain embodiments of the first aspect of this application, the method further includes: separating the attention values of the image to the table along the sequence dimension into a first sub-attention value corresponding to the modality of the first medical image and a second sub-attention value corresponding to the modality of the second medical image.
[0013] According to certain embodiments of the first aspect of this application, the attention value between the images is calculated by the following method: using the visual features of the first medical image as a query and the visual features of the second medical image as a key and value, an enhanced cross feature corresponding to the first medical image is generated through attention calculation; the enhanced cross feature is added to the visual features of the first medical image through residual connection to obtain the attention value between the images.
[0014] According to certain embodiments of the first aspect of this application, the fusion of attention values from a table to an image, attention values from an image to a table, and attention values between images to obtain fusion features includes: normalizing the fusion weights into a probability distribution; multiplying the probability distribution of the fusion weights by the attention values from the image to the table, the first sub-attention value corresponding to the modality of the first medical image, the second sub-attention value corresponding to the modality of the second medical image, and the attention value between images, respectively, to obtain four multiplication results; and concatenating the four multiplication results along the sequence dimension to obtain the fusion features.
[0015] According to certain embodiments of the first aspect of this application, the step of extracting global semantic features of the fusion features through the fusion feature classification head of the childhood leukemia screening model, and sequentially performing multi-level classification on the global semantic features to obtain childhood leukemia screening results includes: sequentially performing a first linear transformation, a first-level normalization, a first activation function, a first random inactivation, and a second linear transformation on the fusion features through the fusion processing module of the fusion feature classification head to obtain high-level fusion features; and sequentially performing a third linear transformation, a second-level normalization, a second activation function, a second random inactivation, a fourth linear transformation, a third activation function, a third random inactivation, and a fifth linear transformation on the high-level fusion features through the classification processing module of the fusion feature classification head to obtain childhood leukemia screening results.
[0016] According to a second aspect of this application, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal data-based childhood leukemia screening method as described in the first aspect of this application.
[0017] According to a third aspect of this application, a computer storage medium stores computer-executable instructions for performing a multimodal data-based childhood leukemia screening method as described in an embodiment of the first aspect of this application.
[0018] The above scheme has at least the following beneficial effects: it constructs a multimodal intelligent screening framework, realizing more comprehensive information complementarity; it realizes a deep semantic alignment mechanism based on cross-attention, dynamically learns the correlation between modalities through three attention paths, and adaptively fuses multi-source features, thereby accurately capturing the complex correspondence between morphological anomalies and quantitative parameters. Attached Figure Description
[0019] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.
[0020] Figure 1 is a schematic diagram of the architecture of the childhood leukemia screening model; Figure 2 is a flowchart of the steps of the childhood leukemia screening method based on multimodal data; Figure 3 is a schematic diagram of the architecture of the feature fusion module; Figure 4 is a schematic diagram of the architecture of the fused feature classification head. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0022] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0023] Leukemia is a common malignant disease of the blood system. Acute myeloid leukemia is most common in adults, while acute lymphoblastic leukemia is the most prevalent in children. Early detection can significantly improve the five-year survival rate for children with acute lymphoblastic leukemia.
[0024] Peripheral blood blasts / immature cells refer to immature leukocyte precursor cells found in the peripheral (circulating) blood. They are extremely rare in healthy adults, but an abnormally high number is one of the important early morphological markers of hematologic malignancies such as acute leukemia, and a key target for leukemia screening. Therefore, accurately identifying the presence of blasts in peripheral blood is crucial for reducing leukemia mortality and improving leukemia diagnosis and treatment.
[0025] Blood cell analysis (complete blood count) is one of the most commonly used, rapid, and economical examination methods in clinical practice. It can output basic report parameters, research parameters, alarm information, and white blood cell scatter plots. With the development of blood cell classification technology, blood cell analysis methods have evolved from single impedance methods to multidimensional detection integrating impedance methods, flow cytometry, radiofrequency ablation, and cytochemical staining. For example, fully automated blood cell analyzers, based on laser scattering combined with fluorescence staining multidimensional analysis technology, can distinguish different types of white blood cells in a three-dimensional scattering space and indicate abnormalities such as immature granulocytes and blast cells. When the blood analyzer indicates abnormalities such as blast cells, manual microscopic examination is still required to ensure the reliability of the results. In actual clinical work, instrument alarms indicating blast cells have low specificity, a high re-examination rate, and manual microscopic examination is extremely labor-intensive, limited by manpower and ability, and has potential drawbacks such as high rates of missed and false detections, making it difficult to improve work efficiency.
[0026] Furthermore, leukemia screening is primarily limited to a single signal space or a single data modality. It often focuses on a single data source, such as analyzing only scattering / fluorescence signals or processing only cell microscopic images, failing to achieve a deep fusion of visual features from scatter plots and structured blood routine parameters. This single-modality analysis approach ignores the potential complementarity and synergistic effects between different data sources, resulting in limited sensitivity and specificity for identifying primitive / immature cells, and screening accuracy that fails to meet high clinical standards.
[0027] Secondly, existing blood cell analysis systems based on traditional rules or thresholds have low specificity and high false positive rates in their screening results, leading to arduous and inefficient manual retesting.
[0028] Furthermore, even when attempting multimodal fusion methods, their fusion strategies are typically quite simple (such as feature concatenation or weighted addition), failing to achieve deep semantic alignment and interaction across modal features. This shallow fusion makes it difficult to effectively mine the complex nonlinear relationships between image features and tabular data, limiting the model's ability to learn discriminative features from multi-source heterogeneous data.
[0029] Embodiments of this application provide a method for screening childhood leukemia based on multimodal data.
[0030] Understandably, multimodal data-based childhood leukemia screening methods are primarily used for intelligent screening of acute childhood leukemia. Intelligent screening for acute childhood leukemia refers to an automated diagnostic process that uses computer algorithms to automatically analyze peripheral blood samples from children to identify the presence of primitive or immature cells, thereby assisting clinical assessment of the risk of childhood leukemia. This process typically relies on a comprehensive assessment of multi-source data (such as images and numerical parameters) generated by a blood cell analyzer.
[0031] The embodiments of this application will be further described below with reference to the accompanying drawings.
[0032] Referring to Figures 1 and 2, a method for screening childhood leukemia based on multimodal data includes the following steps: Step S100, acquiring a first medical image, a second medical image, and blood routine test data, and inputting the first medical image, the second medical image, and the blood routine test data into a childhood leukemia screening model; Step S200, extracting visual features of the first medical image and the second medical image respectively through different branches of the visual feature extractor of the childhood leukemia screening model, and splicing the visual features of the first medical image and the second medical image to obtain joint visual features; Step S300, ... The feature fusion module of the childhood leukemia screening model calculates the attention value from table to image, the attention value from image to table, and the attention value between images based on the visual features of the first medical image, the visual features of the second medical image, the joint visual features, and the table modal features, respectively. The attention values from table to image, image to table, and between images are fused to obtain the fusion feature. In step S400, the global semantic features of the fusion feature are extracted through the fusion feature classification head of the childhood leukemia screening model. The global semantic features are then classified at multiple levels to obtain the childhood leukemia screening result.
[0033] In step S100, the first medical image, the second medical image, and the blood routine test data are acquired. The first medical image is a DIFF scatter plot, and the second medical image is a WNB scatter plot. Both the DIFF and WNB scatter plots are two-dimensional distribution images generated by the fully automated blood cell analyzer during the testing process. The DIFF (white blood cell differential) scatter plot mainly uses laser scattering technology to distinguish the major white blood cell subgroups (neutrophils, lymphocytes, monocytes, etc.); the WNB (white blood cell / basophil) scatter plot combines fluorescence staining and scattering signals, and is particularly helpful in identifying basophils and indicating abnormal groups such as primitive and immature cells. Both provide a direct and visual basis for the instrument to classify white blood cells and indicate abnormalities.
[0034] A complete blood count (CBC) report is a structured numerical report output by a blood cell analyzer after the test is completed. It typically includes 42 quantitative indicators such as white blood cell count (WBC), hemoglobin concentration (HGB), and platelet count (PLT), which objectively reflect the quantity and morphological characteristics of various cells in a patient's blood and serve as the basic quantitative basis for disease screening.
[0035] DIFF scatter plots and WNB scatter plots belong to different image modalities, while blood routine test table data belongs to the table modality.
[0036] Input the DIFF scatter plot, WNB scatter plot, and blood routine test data into the childhood leukemia screening model.
[0037] For step S200, visual features of the first medical image and visual features of the second medical image are extracted by different branches of the visual feature extractor of the childhood leukemia screening model. Specifically, the visual feature extractor is set with a first branch and a second branch in parallel. The first branch is used to extract the visual features of the first medical image, and the second branch is used to extract the visual features of the second medical image.
[0038] The visual feature extractor is based on the ConvNeXt architecture, a convolutional neural network (CNN) architecture. It incorporates Transformer design principles (such as larger convolutional kernels and layer normalization) into the traditional CNN architecture, enhancing the model's ability to capture long-range dependencies and complex patterns in images while maintaining the efficiency of convolutional computation. It is commonly used for feature extraction from medical images.
[0039] The first and second branches of the visual feature extractor use ConvNeXt-Base as the backbone network to extract high-level visual features and obtain a 1024-dimensional feature vector through global average pooling (GAP).
[0040] The ConvNeXt-Base architecture integrates the Transformer design philosophy with modern convolutional optimization techniques (such as large convolutional kernels and layer normalization), resulting in superior feature extraction capabilities and computational efficiency. Compared to different backbone networks (such as Swintransformer and EfficientNet), the ConvNeXt architecture demonstrates the best performance in medical imaging tasks, enabling better extraction of fine-grained visual features and improving classification accuracy.
[0041] Then, the visual features of the first medical image and the visual features of the second medical image are stitched together in the channel dimension to obtain 2048-dimensional joint visual features, which are then classified using a multilayer perceptron (MLP) classification head.
[0042] The dual-branch visual feature extractor effectively preserves the complementary information of the DIFF and WNB image modalities, and further achieves efficient multimodal visual feature fusion through the MLP classification head. This design strategy not only enhances the model's multimodal learning ability but also improves the accuracy and robustness of intelligent screening for childhood leukemia.
[0043] Of course, in other embodiments, the visual feature extractor is not limited to the ConvNeXt architecture, and may also employ other convolutional neural networks or visual feature extraction models, such as ResNet, EfficientNet, DenseNet, Swin Transformer or their improved structures, to extract features from scatter plot images.
[0044] Referring to Figure 3, in step S300, cross-attention is applied through the feature fusion module to achieve deep semantic alignment and complementary feature fusion between the image modality (DIFF and WNB scatter plots) and the tabular modality (structured blood parameters).
[0045] It is understandable that cross-attention is an attention mechanism derived from the Transformer architecture. It allows features from one modality (such as tabular data) to act as "queries" to actively retrieve and focus on relevant information in features from another modality (such as images), thereby achieving deep alignment and interactive fusion of features from different modalities at the semantic level.
[0046] In addition to using multilayer perceptron (MLP) for feature encoding, blood routine data can also be represented by one-dimensional convolutional networks, attention mechanism networks, or tree-based embedding feature extraction methods to represent structured blood parameters, as long as they can output numerical feature vectors that can be used for subsequent multimodal fusion.
[0047] The feature fusion module achieves cross-modal semantic interaction through three parallel attention paths.
[0048] The first path is the table-to-image attention value. Using the table modality feature (Table_feature) as the query and the joint visual feature (image_feature) as the key and value, cross-attention calculation is used to focus hematological parameters on relevant cell morphology evidence, generating an enhanced table feature (enhanced_table). This enhanced table feature is then added to the table modality feature via a residual connection and subjected to layer normalization to obtain the table-to-image attention value. Layer normalization ensures stable training.
[0049] The second path involves the image-to-table attention value. Using the joint visual feature (image_feature) as the query and the enhanced table feature (enhanced_table) as the key and value, cross-attention calculation is used to deeply embed the quantified indicators into the visual semantic representation, generating the enhanced visual feature (enhanced_image). This enhanced visual feature is then added to the joint visual feature via a residual connection and stabilized by layer normalization to obtain the image-to-table attention value. Subsequently, the image-to-table attention value is separated along the sequence dimension into a first sub-attention value corresponding to the modality of the first medical image and a second sub-attention value corresponding to the modality of the second medical image. The first sub-attention value is the DIFF branch enhanced feature (enhanced_Diff), and the second sub-attention value is the WNB branch enhanced feature (enhanced_Wnb), corresponding to the enhanced representations of the DIFF and WNB modalities, respectively.
[0050] The third path involves the attention value between images. Using the visual features of the first medical image as the query and the visual features of the second medical image as the key and value, enhanced cross features corresponding to the first medical image are generated through attention calculation. These enhanced cross features are then added to the visual features of the first medical image via residual connections and normalized to obtain the attention value between the images. One-way querying and interaction are performed between the two scatter plot features to achieve internal complementarity and consistency verification of morphological information.
[0051] The three paths are designed in a cascaded manner, with subsequent steps using the enhanced features from the previous step as input. Finally, the four enhanced features (enhanced_Diff, enhanced_Wnb, enhanced_table, and Diff_cross_Wnb) enter a learnable weighted fusion stage.
[0052] The fusion feature is obtained by fusing the attention values from the table to the image, the attention values from the image to the table, and the attention values between images. The steps include: normalizing the fusion weights into a probability distribution; multiplying the probability distribution of the fusion weights by the attention values from the image to the table, the first sub-attention value corresponding to the modality of the first medical image, the second sub-attention value corresponding to the modality of the second medical image, and the attention value between images, respectively, to obtain four multiplication results; and concatenating the four multiplication results in the sequence dimension to obtain the fusion feature.
[0053] Specifically, a set of trainable fusion weights (with a dimension of 4) is first normalized to a probability distribution using the Softmax function, and then multiplied by the corresponding four enhancement features to achieve adaptive importance weighting. The four weighted features are then concatenated along the sequence dimension to form the final output of this module, which is the fusion feature with dimensions [B, 4, 256], and can be directly input into the subsequent classification head for further processing.
[0054] It should be noted that the cross-attention layer used is based on a multi-head self-attention mechanism and is specifically adapted for cross-modal interaction scenarios. This layer maps the input features into query, key, and value vectors respectively through learnable linear projection, and uses scaled dot product attention to calculate the correlation weights between cross-modal features.
[0055] Cross attention is represented as: In the formula, Q represents the query, K represents the key, and V represents the value.
[0056] Referring to Figure 4, for step S400, the global semantic features of the fusion features are extracted through the fusion feature classification head of the childhood leukemia screening model. The global semantic features are then classified at multiple levels to obtain the childhood leukemia screening results. This includes the following steps: the fusion processing module of the fusion feature classification head sequentially performs a first linear transformation, a first-level normalization, a first activation function, a first random inactivation, and a second linear transformation on the fusion features to obtain high-level fusion features; the classification processing module of the fusion feature classification head sequentially performs a third linear transformation, a second-level normalization, a second activation function, a second random inactivation, a fourth linear transformation, a third activation function, a third random inactivation, and a fifth linear transformation on the high-level fusion features to obtain the childhood leukemia screening results.
[0057] Specifically, features from three modalities (DIFF image modality, WNB image modality, and table modality) are modeled by a cross-attention module, outputting a fused feature (dimension B×3×d, where d=256) containing contextual information from the three modalities. This fused feature is flattened into a one-dimensional vector and input to a fusion processing module, block_A, for further extraction of a high-order global joint semantic representation. The fusion processing module sequentially performs linear transformations on the fused feature. Layer normalization processing Activation function processing Random inactivation treatment (0.5) and linear transformation processing This yields advanced fusion features. The processing procedure of the fusion module is represented by the following formula: .in, This indicates the fusion characteristics after flattening. This is an advanced fusion feature.
[0058] Global advanced fusion features The data is input into the three-level joint feature classification block_B. The classification processing module performs linear transformations on the high-level fusion features sequentially. Layer normalization (BN) and activation function processing Random inactivation treatment (0.5) Linear transformation processing Activation function processing Random inactivation treatment (0.3) and linear transformation processing The processing procedure of the classification module is represented by the following formula: ; ; .
[0059] This joint structure not only fully preserves the global information after multimodal fusion, but also achieves efficient mapping from bottom-level fusion to final semantic discrimination through hierarchical feature abstraction and nonlinear transformation.
[0060] The following validation is performed using a dataset.
[0061] Blood routine tests and white blood cell differentials were performed on multiple outpatient and inpatient blood samples using a blood analyzer.
[0062] All included samples underwent routine blood tests using a blood analyzer, followed by automated slide preparation and staining using a slide staining machine. A fully automated cell morphology analyzer was then used to count and classify 200 nucleated cells, with the classification results manually reviewed and confirmed. Samples were divided into four groups according to international rules for positive smear microscopy. Grouping data are shown in Table 1.
[0063] Table 1 Grouped Data Table
[0064] To evaluate the performance of TCM-Net in the four-category screening task for childhood leukemia, six key metrics were used: accuracy (ACC), precision, recall, specificity, and F1 score. The model employs a combination of metrics, including accuracy (the overall classification correctness), precision and recall (representing accuracy and recall respectively), specificity (assessing the ability to identify negative samples), F1 score (balancing precision and recall), and AUC (evaluating the model's overall discriminative performance). These metrics collectively validate the model's reliability in medical screening tasks.
[0065] The MMPB-ALL multimodal dataset, constructed from multiple samples collected from real clinical settings, was trained using an NVIDIA RTX2080 Ti GPU and the PyTorch 2.1.0 framework. During training, the batch size was set to 32, and the AdamW optimizer (initial learning rate 0.0001, weight decay 0.01) was used in conjunction with ReduceLROnPlateau to dynamically adjust the learning rate for 100 epochs. The image branch loaded pre-trained ConvNeXt-Base weights and froze the parameters, while the table branch used pre-trained MLP weights. Only the cross-attention fusion module and the classification head participated in training. During the evaluation phase, in addition to comparisons with unimodal and common backbone networks, ablation experiments were conducted to verify the contribution of each module, ensuring the reliability of the results.
[0066] The comparison results of the algorithms for different backbone networks were verified, as shown in Table 2.
[0067] Table 2. Comparison of performance metrics for different backbone network algorithms.
[0068] Ablation experiments were conducted using the TCM-Net model, and the results are shown in Table 3.
[0069] Table 3 Ablation Experiment Results of the TCM-Net Model
[0070] Ablation experiments were conducted on the feature fusion module of the TCM-Net model, and the experimental results are shown in Table 4.
[0071] Table 4 Ablation Experiment Table of Feature Fusion Module of TCM-Net Model
[0072] Experimental results show that TCM Net achieved an accuracy of 86.87% and an AUC of 0.95, significantly outperforming the baseline.
[0073] Embodiments of this application provide an electronic device. The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the childhood leukemia screening method based on multimodal data as described above.
[0074] This electronic device can be any smart terminal, including computers.
[0075] In general, for the hardware structure of electronic devices, the processor can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, to execute relevant programs and implement the technical solutions provided in the embodiments of this application.
[0076] The memory can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and is called and executed by the processor.
[0077] Input / output interfaces are used to implement information input and output.
[0078] The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0079] The bus transmits information between various components of a device, such as the processor, memory, input / output interfaces, and communication interfaces. The processor, memory, input / output interfaces, and communication interfaces communicate with each other within the device via the bus.
[0080] Embodiments of this application provide a computer storage medium. The computer storage medium stores computer-executable instructions for performing the multimodal data-based childhood leukemia screening method described above.
[0081] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0082] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0083] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0086] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms. Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.
[0087] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.
Claims
1. A method for screening childhood leukemia based on multimodal data, characterized in that, include: A first medical image, a second medical image, and routine blood test data are acquired and input into a childhood leukemia screening model. The first and second medical images belong to different image modalities, while the routine blood test data belongs to a table modality. Visual features of the first and second medical images are extracted from different branches of the visual feature extractor of the childhood leukemia screening model. The visual features of the first and second medical images are then combined to obtain joint visual features. The feature fusion module of the childhood leukemia screening model calculates the table-to-image attention value, the image-to-table attention value, and the inter-image attention value based on the visual features of the first and second medical images, the joint visual features, and the table modal features extracted from the routine blood test data. The table-to-image attention value, the image-to-table attention value, and the inter-image attention value are then fused to obtain a fused feature. The global semantic features of the fusion features are extracted by the fusion feature classification head of the childhood leukemia screening model. The global semantic features are then classified at multiple levels to obtain the childhood leukemia screening results.
2. The method for screening childhood leukemia based on multimodal data according to claim 1, characterized in that, The visual feature extractor is provided with a first branch and a second branch in parallel. The first branch is used to extract the visual features of the first medical image, and the second branch is used to extract the visual features of the second medical image.
3. The method for screening childhood leukemia based on multimodal data according to claim 1, characterized in that, The attention value from the table to the image is calculated as follows: using table modal features as queries and joint visual features as keys and values, cross-attention calculation is used to focus hematological parameters on relevant cell morphology evidence to generate enhanced table features; The enhanced table features are added to the table modal features via residual connections to obtain the attention value from the table to the image.
4. The method for screening childhood leukemia based on multimodal data according to claim 3, characterized in that, The attention value from the image to the table is calculated as follows: using joint visual features as the query and enhanced table features as the key and value, cross-attention calculation is used to deeply embed the quantitative indicators into the visual semantic representation to generate enhanced visual features. The enhanced visual features are added to the joint visual features via residual connections to obtain the attention value from the image to the table.
5. The method for screening childhood leukemia based on multimodal data according to claim 4, characterized in that, The method further includes: separating the attention value from the image to the table along the sequence dimension into a first sub-attention value corresponding to the modality of the first medical image and a second sub-attention value corresponding to the modality of the second medical image.
6. The method for screening childhood leukemia based on multimodal data according to claim 5, characterized in that, The attention value between the images is calculated as follows: using the visual features of the first medical image as the query and the visual features of the second medical image as the key and value, the enhanced cross features corresponding to the first medical image are generated through attention calculation. The enhanced cross features are added to the visual features of the first medical image via residual connections to obtain the attention value between the images.
7. The method for screening childhood leukemia based on multimodal data according to claim 6, characterized in that, The fusion feature is obtained by fusing the attention values from the table to the image, the attention values from the image to the table, and the attention values between images. This includes: normalizing the fusion weights into a probability distribution; multiplying the probability distribution of the fusion weights by the attention values from the image to the table, the first sub-attention value corresponding to the modality of the first medical image, the second sub-attention value corresponding to the modality of the second medical image, and the attention values between images, respectively, to obtain four multiplication results; and concatenating the four multiplication results in the sequence dimension to obtain the fusion feature.
8. The method for screening childhood leukemia based on multimodal data according to claim 1, characterized in that, The process involves extracting global semantic features from the fusion feature classification head of the childhood leukemia screening model, and then performing multi-level classification on these global semantic features to obtain the childhood leukemia screening results. This includes: sequentially performing a first linear transformation, a first-level normalization, a first activation function, a first random inactivation, and a second linear transformation on the fusion feature classification head's fusion processing module to obtain high-level fusion features; and sequentially performing a third linear transformation, a second-level normalization, a second activation function, a second random inactivation, a fourth linear transformation, a third activation function, a third random inactivation, and a fifth linear transformation on the high-level fusion features through the fusion feature classification head's classification processing module to obtain the childhood leukemia screening results.
9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method for screening childhood leukemia based on multimodal data as described in any one of claims 1 to 8.
10. A computer storage medium, characterized in that, The device stores computer-executable instructions for performing the childhood leukemia screening method based on multimodal data as described in any one of claims 1 to 8.