A bone tumor fine-grained classification model training, classification method and device

Abstract

The application provides a bone tumor fine-grained classification model training, classification method and device, constructs a multi-modal positive sample containing global / local positive lateral X-ray image, lesion attribute and patient information, combines text semantic similarity to remove false negative part to construct high-quality negative sample; global / local image features are extracted through a double-image encoder, the lesion attribute keywords are converted into "entity-translation-existence" triplets relying on a medical knowledge base, and global / local semantic features of patient basic information and lesion attributes are extracted by using a text encoder; based on contrast learning, the global image and global semantic are constructed InfoNCE contrast loss, the model is trained based on the local mutual information loss calculated based on the DV variation formula and the classification loss, the global text feature alignment and the local text feature alignment are realized, the medical terminology semantics are deeply combined, the training stability is improved, the bone tumor subtype classification accuracy and robustness are significantly improved, and reliable support is provided for clinical accurate diagnosis.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a fine-grained classification model training, classification method and apparatus for bone tumors. Background Technology

[0002] Precise diagnosis and treatment of bone tumors heavily rely on the identification of their specific histological subtypes. Different subtypes exhibit significant differences in treatment plans and prognoses, making fine-grained classification based on X-ray images a crucial requirement for clinical diagnostic support. However, due to the similarity in imaging manifestations among bone tumor subtypes and the diversity of morphology within subtypes, coupled with the fact that X-ray films are two-dimensional projections of three-dimensional structures, this task poses a challenge even to experienced radiologists. With the development of deep learning technology, researchers have gradually explored its application in this field. Early methods primarily relied on convolutional neural networks to automatically learn and classify features from single X-ray images, but these efforts largely stopped at a binary judgment of benign or malignant, failing to meet the clinical need for distinguishing specific subtypes. Subsequently, to improve model performance, methods emerged that integrated structured clinical data such as patient age and location with imaging features. However, these methods failed to effectively utilize the unstructured professional textual information in imaging reports that describes lesion details—the core basis for physicians' diagnostic reasoning.

[0003] In recent years, the rise of medical vision-language pre-trained models has offered new insights. By enabling models to learn the association between medical images and their corresponding report texts, it is hoped that AI can achieve semantic understanding capabilities closer to that of doctors. The closest existing techniques typically employ a global contrastive learning paradigm, such as having the model learn to align an entire image with an entire report summary. However, this coarse-grained alignment method struggles to capture minute lesion regions crucial for differential diagnosis. Further improvements attempt to introduce local alignment mechanisms, capturing details by matching image sub-regions with text words. However, these methods often extract features from intermediate layers of the network, and limited resolution may lead to the loss of fine-grained information. More importantly, contrastive learning-based methods generally rely on the construction of a large number of negative samples. In medical scenarios, similar diseases in different patients can generate semantically similar text descriptions. Randomly combined negative sample pairs are essentially "false negative" noise, which severely interferes with the model's learning of discriminative features, limiting the accuracy and robustness of its final classification. Furthermore, how to effectively fuse multi-view images, such as anteroposterior and lateral views, to construct more comprehensive spatial information remains a challenge that current solutions have not yet adequately addressed. Therefore, existing technologies have limitations in terms of deep semantic utilization of medical texts, accurate alignment of local details, suppression of training noise, and fusion of multi-perspective information. There is an urgent need for a fine-grained classification method that can collaboratively optimize global and local information and has stronger robustness. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a training, classification method and apparatus for fine-grained classification models of bone tumors, which solves the problems of low accuracy and robustness of X-ray bone tumor subtype classification caused by insufficient use of medical text, inaccurate alignment of local features and false negative interference during training.

[0005] One aspect of the present invention provides a method for training a fine-grained classification model for bone tumors, the method comprising the following steps: Image and text data of multiple bone tumor cases were acquired. The image data included global X-ray images and local lesion X-ray images, both of which included anteroposterior and lateral views. The text data included lesion attributes and basic patient information, and the bone tumor type was labeled as a positive sample. Negative samples are constructed by randomly replacing the text data in the positive samples with the text data from different bone tumor cases, and the negative samples with a similarity of text data before and after replacement exceeding a set threshold are removed. For each positive or negative sample, the global X-ray image is processed by a first image encoder to obtain global anterolateral fusion features, and the local lesion X-ray image is processed by a second image encoder to obtain local anterolateral fusion features. The feature keywords of the lesion attributes are queried from a preset medical knowledge base to obtain a natural language description. The feature keywords, the natural language description, and the existence marker are used as a triple, and a first semantic feature is extracted by a text encoder. The patient's basic information is used to extract a second semantic feature by the text encoder. The global anterolateral fusion features of the positive sample are further processed by a fully connected layer to output a bone tumor subtype classification prediction value. The local lateral fusion features of positive and negative samples and the first semantic features are input into the mutual information discriminator to output local mutual information. The lower bound of mutual information is calculated based on the DV variational formula, and the mutual information loss is constructed by maximizing the lower bound of mutual information. The InfoNCE contrast loss is constructed based on the global lateral fusion features of positive and negative samples and the second semantic features. The classification loss is constructed by minimizing the deviation between the predicted value and the label. The parameters of the first image encoder, the second image encoder, the text encoder, the fully connected layer, and the mutual information discriminator are updated by combining the mutual information loss, the contrast loss, and the classification loss. The first image encoder and the fully connected layer are combined into a fine-grained classification model for bone tumors.

[0006] In some embodiments, the lesion attributes include: lesion type, lesion location, presence of soft tissue mass, and tumor enhancement; the patient basic information includes patient gender, age, lesion site, and pathological examination classification results; The method further includes: removing noise, artifacts, missing or duplicate parts of key components from the image data; and performing structured processing on the text data.

[0007] In some embodiments, removing negative samples whose text data similarity before and after replacement is higher than a set threshold includes: The Clinical BERT model, fine-tuned for the medical field, is used to process the text data before and after replacement in the negative samples to obtain embedded vectors, and cosine similarity is calculated. Negative samples with cosine similarity higher than a set threshold are removed.

[0008] In some embodiments, the first image encoder and the second image encoder employ a ResNet-50 / 101 model, a DenseNet model, a ViT model, or a Swing Transformer network. The Swing Transformer network consists of multiple Transformer blocks, and each Transformer block includes a multi-head attention module and a multi-layer perceptron module. The text encoder employs a BioBERT model, a PubMedBERT model, a BERT model, a RoBERTa model, or a Clinical BERT model fine-tuned for the medical field.

[0009] In some embodiments, the local lateral fusion features and the first semantic features are input into a mutual information discriminator to output local mutual information. A lower bound for the mutual information is calculated based on the DV variational formula, and the mutual information loss is constructed by maximizing this lower bound. This includes: The optimization objective is defined based on the aforementioned DV variational formula: ; The formula for calculating the mutual information loss is: ; in, f The mutual information discriminator is used to determine the local positive-lateral fusion features. and the first semantic feature Originating from the positive sample or the negative sample; This represents the expected score of the mutual information discriminator for the positive sample. This represents the expected logarithm of the score given by the mutual information discriminator to the negative sample. This represents the mutual information loss; The InfoNCE contrast loss is constructed based on the global positive and negative lateral fusion features of the positive and negative samples and the second semantic features, including: ; in, This represents the contrast loss. The global positive lateral fusion feature representing the positive sample at the anchor point. The second semantic feature represents the positive sample of the anchor point. The second semantic feature of the negative sample is represented by sim; cosine similarity is calculated, and N is the batch size. The temperature parameter is used; the formula for calculating the classification loss is: ; in, This represents the classification loss. This represents the label of the i-th positive sample. Let C represent the predicted value of the bone tumor subtype classification of the i-th positive sample, where C is the number of categories and N is the total number of positive samples. The formula for calculating the joint loss is: ; in, These are the weighting coefficients. The joint loss is mentioned above.

[0010] In some embodiments, the preset medical knowledge base uses UMLS, SNOMED CT, or MeSH medical terminology sets.

[0011] On the other hand, the present invention also provides a method for fine-grained classification of bone tumors, the method comprising the following steps: Obtain global X-ray images and local lesion X-ray images of the case to be analyzed, input the bone tumor fine-grained classification model in the above bone tumor fine-grained classification model training method, and output the bone tumor classification results.

[0012] On the other hand, the present invention also provides a fine-grained bone tumor classification device, including a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor is used to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the above method.

[0013] On the other hand, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0014] On the other hand, the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. The bone tumor fine-grained classification model training, classification method, and apparatus described in this invention constructs multimodal positive samples containing global and local anteroposterior and lateral X-ray images, lesion attributes, and basic patient information. Highly similar samples are filtered and eliminated based on text semantic similarity to construct high-quality negative samples. A dual image encoder extracts global and local anteroposterior and lateral fusion features, while a medical knowledge base is used to convert lesion attribute keywords into "entity-translation-existence" triples, which are then combined with a text encoder to extract two levels of semantic features. Finally, the model is trained end-to-end using InfoNCE contrast loss, local mutual information loss based on the DV variational formula, and classification loss to obtain a bone tumor fine-grained classification model. This invention achieves precise alignment between global macroscopic semantics and local fine-grained lesion features, effectively overcomes the semantic gap in medical terminology through knowledge augmentation, and improves training stability through a negative sample selection strategy that resists false negatives. Ultimately, it significantly improves the accuracy and robustness of bone tumor subtype classification, providing reliable auxiliary decision-making basis for precise clinical diagnosis.

[0015] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0016] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a schematic flowchart of a bone tumor fine-grained classification model training method according to an embodiment of the present invention.

[0018] Figure 2 This is a logical schematic diagram of a fine-grained diagnostic method for bone tumors based on global contrastive learning and local mutual information, as described in another embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0020] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0021] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0022] Currently, research on fine-grained classification of bone tumors has evolved from single-image analysis to multimodal methods that integrate imaging and clinical data, and further explores visual-language pre-trained models based on image-report text alignment. However, existing technologies still face several key bottlenecks: First, most methods fail to fully utilize the rich details and semantic content of professional texts in imaging reports, leading to gaps in the model's understanding of complex medical concepts; second, while mainstream global alignment paradigms can capture overall correlations, they struggle to focus on minute lesion areas crucial for diagnosis, and the introduced local alignment methods often suffer from insufficient feature resolution or downsampling operations, resulting in the loss of fine-grained information; third, and most notably, contrastive learning-based training mechanisms rely on randomly constructed negative samples, which can easily misclassify different cases describing similar lesions as negative examples in medical datasets, introducing a large amount of "false negative" noise and severely interfering with the model's learning of discrimination boundaries; furthermore, how to effectively integrate multi-view information such as anteroposterior and lateral views to overcome the overlap and occlusion of three-dimensional anatomical structures in two-dimensional projection is also a problem that existing solutions have not adequately addressed. These factors collectively limit the accuracy, robustness, and clinical applicability of existing technologies in the task of classifying bone tumor subtypes.

[0023] In view of this, the present invention provides a method for training a fine-grained classification model for bone tumors, such as... Figure 1 As shown, the method includes the following steps S101~S104: Step S101: Obtain imaging data and text data for multiple bone tumor cases. The imaging data includes global X-ray images and local lesion X-ray images. Both global X-ray images and local lesion X-ray images include anteroposterior and lateral views. The text data includes lesion attributes and basic patient information. The bone tumor type is labeled as a tag to construct positive samples.

[0024] Step S102: Use text data from different bone tumor cases to randomly replace the text data in the positive samples to construct negative samples, and remove negative samples whose similarity between the text data before and after replacement is higher than a set threshold.

[0025] Step S103: For each positive or negative sample, the global X-ray image is processed by the first image encoder to obtain global anterolateral fusion features, and the local lesion X-ray image is processed by the second image encoder to obtain local anterolateral fusion features; the feature keywords of the lesion attributes are queried from a preset medical knowledge base to obtain natural language descriptions, and the feature keywords, natural language descriptions and existence markers are used as triples to extract the first semantic features through a text encoder; the patient's basic information is used to extract the second semantic features through a text encoder; among them, the global anterolateral fusion features of the positive sample are also output as bone tumor subtype classification prediction values ​​through a fully connected layer.

[0026] Step S104: Input the local lateral fusion features of positive and negative samples and the first semantic features into the mutual information discriminator to output local mutual information. Calculate the lower bound of mutual information based on the DV variational formula, and maximize the lower bound of mutual information to construct the mutual information loss. Construct the InfoNCE contrast loss based on the global lateral fusion features of positive and negative samples and the second semantic features. Minimize the deviation between the predicted value and the label to construct the classification loss. Combine the mutual information loss, contrast loss, and classification loss to update the parameters of the first image encoder, the second image encoder, the text encoder, the fully connected layer, and the mutual information discriminator. Combine the first image encoder and the fully connected layer into a fine-grained classification model for bone tumors.

[0027] In step S101, a multimodal, high-quality positive sample dataset of bone tumors is constructed to provide basic data support for subsequent model training. Its implementation method must strictly follow the standardized data collection, cleaning and annotation process.

[0028] During the data acquisition phase, imaging data needs to be collected for both global X-ray images and local lesion X-ray images of bone tumor cases. Both types of images need to include both anteroposterior and lateral views. Global X-ray images are used to present the overall anatomical structure of the bone and the background of the lesion, while local lesion X-ray images are images of the tumor area obtained through manual segmentation or automatic detection. They can focus on fine-grained features such as the texture and edges of the tumor. For example, for a case of osteosarcoma of the distal femur, it is necessary to simultaneously collect anteroposterior and lateral global X-ray images of the area, as well as anteroposterior and lateral local images of the osteosarcoma lesion area marked by the radiologist.

[0029] The text data must cover lesion attributes and basic patient information. Lesion attributes include: lesion type, lesion location, presence of soft tissue mass, and tumor enhancement, etc. Destruction type, such as osteolytic type, osteoblastic type, etc., lesion location, such as central, eccentric, etc. Basic patient information includes patient gender, age, lesion location, and pathological examination classification results, such as "The patient is a 15-year-old male, the lesion is located in the distal femur, the lesion shows osteolytic destruction, accompanied by soft tissue swelling."

[0030] In some embodiments, the method further includes: removing noise, artifacts, missing or duplicated key areas from the image data; and performing structured processing on the text data. During the data preprocessing stage, quality control of the image data is required to remove images with severe noise, obvious artifacts, or missing key areas. Simultaneously, at least two experienced radiologists must annotate the local lesion areas, and a third expert must review the annotations to ensure accuracy. The text data needs to be structured to extract key information.

[0031] Finally, the imaging data and text data of each case were matched one-to-one with their corresponding pathological diagnosis results to construct positive samples. The pathological diagnosis results served as the bone tumor subtype labels, such as osteosarcoma and Ewing's sarcoma. Each positive sample contained complete multimodal information including "global anteroposterior and lateral images - local anteroposterior and lateral images - lesion attributes - patient basic information - tumor subtype label".

[0032] Step S102 aims to construct a negative sample set free of false negative noise, addressing the model training bias caused by the semantic similarity of negative samples in traditional contrastive learning. Its implementation requires combining text semantic similarity calculation to achieve negative sample selection. First, the initial construction of negative samples employs a text replacement method, randomly selecting text data from different bone tumor cases and replacing the original text data in the positive samples. For example, replacing the text data of osteosarcoma positive samples with text data from chondrosarcoma cases creates an initial negative sample pair of "osteosarcoma image - chondrosarcoma text".

[0033] Subsequently, the initially constructed negative samples need to be screened. The core task is to calculate the semantic similarity between the text data before and after the replacement. Specifically, a pre-trained Clinical BERT model can be used to convert the two sets of text data before and after the replacement into high-dimensional semantic vectors, and then the semantic similarity between the vectors is evaluated. A similarity threshold is set, such as α=0.7. If the similarity between the texts before and after the replacement is higher than this threshold, it means that the tumor features corresponding to the two texts are highly similar semantically. For example, if two texts both describe "osteolytic destruction and eccentric distribution," even if the corresponding tumor subtypes are different, the initial negative sample should be removed to avoid the model misclassifying semantically similar samples as negative samples. If the similarity is lower than the threshold, the sample is retained as the final negative sample. The negative sample set constructed in this way can ensure that the negative samples are significantly different from the positive samples semantically, thus providing reliable training data for subsequent comparative learning.

[0034] Step S103 is the feature extraction stage of model training. Its core is to achieve feature decoupling and semantic enhancement of multimodal data, so as to provide high-quality feature representation for subsequent loss calculation. Its implementation relies on dual image encoders and text encoders.

[0035] For image feature extraction, two independent image encoders need to be deployed: a first image encoder and a second image encoder. Both encoders can use the Swing Transformer network as the backbone network. The first image encoder takes global anteroposterior and lateral X-ray images as input, extracts global skeletal structure features through a hierarchical self-attention mechanism, and then uses a cross-attention mechanism to fuse the features of the anteroposterior and lateral images to generate global anteroposterior and lateral fused features. These features can reflect macroscopic information such as the overall location and extent of the lesion. The second image encoder takes local lesion anteroposterior and lateral X-ray images as input, and uses the same network structure and fusion strategy as the first image encoder to extract fine-grained features such as texture and edges of the local lesion to generate local anteroposterior and lateral fused features.

[0036] For text feature extraction, semantic enhancement and feature transformation are achieved in two steps. The first step is to construct an "entity-translation-existence" triple, where the entity is a feature keyword in the lesion attribute, such as "osteolysis." A true multiple entity query is performed on a preset medical knowledge base to obtain the translation of the keyword, i.e., a natural language description, such as "osteolysis is a form of localized bone destruction, usually occurring in malignant tumors." In some embodiments, the preset medical knowledge base uses UMLS, SNOMED CT, or MeSH medical terminology sets. Simultaneously, an existence marker, such as "present / absent," is added based on the actual presence of the feature in the image. The three are then concatenated to form structured text data. The second step is feature encoding. The constructed triple data is input into a text encoder, such as Clinical BERT, to extract the first semantic feature, which contains professional semantic information about the lesion attribute. The patient's basic information is input into the same text encoder to extract the second semantic feature, which reflects the patient's demographic characteristics.

[0037] Finally, for positive samples, their global positive and lateral fusion features need to be input into a fully connected layer. Through multi-layer nonlinear mapping, the classification prediction value of bone tumor subtype is output, providing a basis for subsequent classification loss calculation.

[0038] In some embodiments, the first image encoder and the second image encoder employ a ResNet-50 / 101 model, a DenseNet model, a ViT model, or a Swing Transformer network. The Swing Transformer network consists of multiple Transformer blocks, each of which contains a multi-head attention module and a multi-layer perceptron module. The text encoder employs the BioBERT model, PubMedBERT model, BERT model, RoBERTa model, or the Clinical BERT model fine-tuned for the medical field.

[0039] Step S104 is the joint training and optimization stage of the model. The core is to update the model parameters through the collaborative optimization of multiple loss functions, and finally obtain a high-performance fine-grained classification model for bone tumors. Its implementation relies on the weighted combination of three loss functions.

[0040] The first type of loss is mutual information loss, which is used to achieve fine-grained matching between local lesion image features and lesion attribute text features. During calculation, the local frontal and lateral fusion features and the first semantic features are input into the mutual information discriminator. The lower bound of local mutual information is calculated based on the DV variational formula. Maximizing this lower bound is used as the optimization objective to construct the mutual information loss function. This loss can uncover the potential statistical correlation between local image features and text features, and improve the model's ability to identify small lesions.

[0041] In some embodiments, the local frontal and lateral fusion features and the first semantic features are input into the mutual information discriminator to output local mutual information. The lower bound of the mutual information is calculated based on the DV variational formula, and the mutual information loss is constructed by maximizing the lower bound of the mutual information, including: The optimization objective is defined based on the DV variational formula: ; The formula for calculating mutual information loss is: ; in, f This represents a mutual information discriminator used to determine local frontal and lateral fusion features. and first semantic features Originating from positive or negative samples; This represents the expected score of the mutual information discriminator for positive samples. This represents the expected logarithm of the score given by the mutual information discriminator for negative samples. This indicates a loss of mutual information.

[0042] The second type of loss is the InfoNCE contrastive loss, which is used to achieve cross-modal alignment between global image features and global text features of patient basic information. During calculation, the global positive and lateral fusion features and the second semantic features of positive and negative samples are respectively input into the contrastive learning module. By calculating the cosine similarity between the global image features and global text features of the same positive sample, and the feature similarity of the negative sample, the InfoNCE loss function is constructed. The optimization goal of this loss is to narrow the feature distance between the global image features and global text features corresponding to the positive sample, and widen the feature distance between the global image features and global text features corresponding to the negative sample, so that the model learns the global-level image-text semantic relationship.

[0043] In some embodiments, the InfoNCE contrastive loss is constructed based on the global positive-side fusion features of positive and negative samples and the second semantic features, including: ; in, Indicates comparative loss, This represents the global positive lateral fusion feature of the anchor point positive sample. The second semantic feature representing the positive sample at the anchor point. The second semantic feature of the negative sample is represented by sim; cosine similarity is calculated, and N is the batch size. This refers to the temperature parameter.

[0044] The third type of loss is the classification loss, which is used to constrain the accuracy of the model's classification results. The cross-entropy loss function is used to calculate the deviation between the predicted value of the bone tumor subtype classification of the positive sample and the true label. By minimizing this deviation, the model learns the discriminative features of the bone tumor subtype.

[0045] In some embodiments, the classification loss is calculated as follows: ; in, Represents classification loss. This represents the label of the i-th positive sample. Let C represent the predicted value of the bone tumor subtype classification for the i-th positive sample, where C is the number of categories and N is the total number of positive samples.

[0046] Finally, the three loss functions are weighted and combined, and weight coefficients are set. By controlling the contributions of contrast loss, mutual information loss, and classification loss to the total loss respectively, a joint loss function is constructed, and the calculation formula is: ; in, These are the weighting coefficients. This is a joint loss.

[0047] Based on the backpropagation algorithm, the parameters of the first image encoder, the second image encoder, the text encoder, the fully connected layer, and the mutual information discriminator are updated using the total loss function. After the model training converges, the first image encoder and the fully connected layer are combined to obtain the final fine-grained classification model of bone tumors. This model can directly input the global anteroposterior and lateral X-ray images of new cases and output the classification results of bone tumor subtypes.

[0048] On the other hand, the present invention also provides a fine-grained classification method for bone tumors, the method comprising: acquiring global X-ray images and local lesion X-ray images of the case to be analyzed, inputting the fine-grained classification model of bone tumors in the bone tumor fine-grained classification model training method described in steps S101 to S104 above, and outputting the bone tumor classification result.

[0049] On the other hand, the present invention also provides a fine-grained bone tumor classification device, including a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor is used to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the above method.

[0050] The present invention will now be described with reference to a specific embodiment: This embodiment provides a fine-grained diagnostic method for bone tumors based on global contrastive learning and local mutual information (BT-GLMI). By deeply fusing image and text features and enhancing semantic understanding, it aims to fully explore the potential information in imaging reports. A dataset containing image, report, and basic clinical information is constructed, and knowledge-enhanced pre-training is performed using medical knowledge, such as introducing professional interpretations of image feature descriptions. This compensates for the model's shortcomings in understanding medical terminology and complex semantics, thereby improving its fine-grained classification ability for bone tumors. By combining global information alignment based on global contrastive learning and fine-grained matching based on local mutual information, it aims to address the problems of single global methods ignoring small lesions and single local methods lacking global consistency, balancing the precision of semantic expression with global consistency. Finally, by designing a negative sample screening mechanism resistant to false negatives and multi-view cross-attention fusion, the accuracy and robustness of fine-grained diagnosis of bone tumors are improved.

[0051] This embodiment proposes a global contrastive learning-local mutual information framework to achieve more accurate bone tumor classification. The framework first models the entire image using global features, capturing the overall anatomical information and macroscopic semantics of the bone tumor. Building upon this, it further combines local mutual information, optimizing the mutual information between local region features of the image and local descriptions in the text, to accurately align descriptions related to specific lesions with corresponding local regions in the image. The BT-GLMI algorithm framework includes an input module, an image feature extractor pre-training module, a global contrastive learning module, and a local mutual information optimization module.

[0052] Reference Figure 2 The specific implementation method is as follows: 1. Dataset Construction: A multimodal dataset was constructed, encompassing X-ray image data, lesion attribute information, and patients' clinical records. The dataset construction strictly followed standardized procedures to ensure data diversity, quality, and accurate annotation. Furthermore, a method specifically designed to avoid false negatives was implemented during the positive and negative sample construction process to improve the model's learning performance.

[0053] 1.1 Data Acquisition: The data comes from the bone tumor imaging and clinical record database of medical institutions. The collected data includes: (1) X-ray imaging data: The acquired images cover different genders, age groups and multiple sites, and the image quality is ensured through standardized acquisition protocols. (2) Lesion attribute information: Lesion features extracted from the images, including lesion type (such as osteolytic type, osteoblastic type), lesion location (such as eccentric or central distribution), presence of soft tissue mass and tumor enhancement. (3) Basic clinical information: including the patient's gender, age, lesion site and classification results obtained by pathological examination.

[0054] 1.2 Data Cleaning and Labeling: To ensure high data quality and consistency, the data cleaning and annotation process includes the following steps: (1) Image quality control: Images with severe noise, obvious artifacts, or missing key parts are removed, and duplicate images are deduplicated. (2) Lesion area annotation: Two experienced radiologists segment and annotate the lesion areas in the images, and a third expert reviews the annotations to generate an annotated image of the local lesion area. (3) Text annotation: Lesion attributes are extracted from the electronic medical records and structured.

[0055] 1.3 Construction of positive and negative samples To address the training requirements of the global contrastive learning and local mutual information optimization modules, this embodiment designs a positive and negative sample construction method, with the key focus on avoiding false negatives in negative samples.

[0056] 1.3.1 Positive Sample Construction Each case's X-ray image is aligned with its corresponding clinical record (including basic patient information and lesion attribute description), forming a positive sample pair. At the local feature level, the labeled lesion area features and the corresponding fine-grained text description (such as "osteolytic lesion with soft tissue mass") form a local positive sample pair.

[0057] 1.3.2 Negative Sample Construction Negative sample pairs are constructed by randomly combining images and text from different cases. To avoid false negatives, this embodiment filters negative samples based on the similarity of the text descriptions: A pre-trained Clinical BERT model is used to embed the text into a vector space, and the cosine similarity of the text pairs is calculated. If the similarity between two texts is higher than a set threshold, If the images and text are highly similar, they are considered semantically similar, and the corresponding images and text are not treated as negative sample pairs. This approach avoids the model learning incorrect image-text mismatch relationships due to being misled by similar text.

[0058] 2. Model Building 2.1 Input Module The input module fully integrates multimodal information, including X-ray images, lesion attributes based on X-ray images, and patient clinical data, to comprehensively improve the model's diagnostic capabilities.

[0059] First, X-ray images consist of two parts: global images and local lesion images. Global images provide overall structural and background information of the lesion area, which helps to model a more comprehensive contextual semantics; local lesion images focus on fine-grained features of key areas, which can enhance the model's ability to capture local features of lesions, thereby enhancing the accurate representation of the target area.

[0060] Secondly, the lesion characteristics based on X-ray images encompass the following aspects: 1) Intraosseous location: The lesion is located in the center or periphery of the bone. 2) Destructive characteristics: Osteoporosis or cortical bone destruction. Destruction type: Including osteolytic, proliferative, or mixed types. 3) Soft tissue swelling: Present or absent. 4) Tumor enhancement: Present or absent.

[0061] In addition, this part requires knowledge enhancement. This embodiment proposes a structured feature extraction method based on triples {Entity, Translation, Exist} to better capture lesion descriptions and semantic information.

[0062] The input consists of three parts: Entity: Represents the core features of the lesion, extracted directly from the medical report. Examples include keywords describing lesion characteristics such as "osteolysis" and "eccentricity." Translation: By querying the Medical Knowledge Base (UMLS), entities are transformed into detailed natural language text descriptions. Further decomposition of specialized medical entities into basic attributes helps the model capture richer semantic information. For example, the entity "osteolysis" is translated as: "Osteolysis is a form of bone destruction, commonly seen in malignant tumors," and the entity "eccentricity" is translated as: "Eccentricity indicates that the lesion is located at the edge of the bone." Existence: Used to indicate whether the feature exists in the current image. Values ​​are "True" or "False." These three values ​​are then concatenated into a natural language input, formatted as follows: "[Entity]: Osteolysis; [Translation]: Osteolysis is a form of localized bone destruction, usually occurring in malignant tumors; [Exist]: Existence."

[0063] Finally, the patient's clinical data was also input in natural language, mainly including the patient's gender, age, lesion location, and lesion classification. This information further enriched the diversity of the input data, helping the model to perform more comprehensive and accurate diagnostic analysis.

[0064] 2.2 Image Feature Extractor Module This embodiment pre-trains the image feature extractor. The module design includes two parts: global image feature extraction and local image feature extraction, both implemented based on the Swin Transformer network. The Swin Transformer extracts multi-scale features stepwise through a hierarchical self-attention mechanism, and its backbone network consists of multiple Transformer blocks. Each block contains a multi-head self-attention module and a multilayer perceptron (MLP) module, and stabilizes gradient propagation through residual connection structures, improving the stability and performance of model training.

[0065] 2.2.1 Global Image Feature Extraction The global image feature extraction module takes the global image as input and outputs the image classification result. By training the model, the weights of the best-performing model are saved, denoted as SwinT_Global_Model. This module's global feature extraction capability provides high-level semantic information for subsequent tasks.

[0066] 2.2.2 Local Image Feature Extraction The local image feature extraction module takes a local image as input and outputs the classification result of that local image. After model training is complete, the weights of the best-performing model are saved, denoted as SwinT_Local_Model. The purpose of local feature extraction is to capture fine-grained local information in the image to compensate for the inadequacy of global features.

[0067] 2.2.3 Feature Fusion and Classification To effectively capture complementary information between X-ray images of different views (such as anteroposterior and lateral views), this embodiment extracts features from anteroposterior and lateral views based on the SwinT_Global_Model. The extracted global features are fused using a cross-attention mechanism, then input into a fully connected layer to complete the classification task, and the model weights with the best performance are saved and named CA_SwinT_Global_Model.

[0068] Similarly, the local image feature extraction module also adopts a similar method, further improving classification performance by fusing features from local images and saving the best-performing model weights, named CA_SwinT_Local_Model.

[0069] 2.3. Global Comparison Learning Module Global contrastive learning aims to align the overall semantic features of X-ray images and patient clinical information. By bringing the feature distance between positive sample pairs (the global X-ray image corresponding to the case and the text of the global patient basic information) closer together, while pushing away negative sample pairs (the global X-ray image and the text of the global patient basic information that do not correspond to the case), it achieves efficient alignment of cross-modal features.

[0070] 2.3.1 Image and Text Feature Extraction Global image features are extracted using the CA_SwinT_Global_Model network to generate fused features of the anteroposterior and lateral global images, denoted as F_GI. This feature integrates complementary information from the anteroposterior and lateral images, effectively capturing the global semantic features of the patient images.

[0071] Global text features are obtained by processing the text using the Clinical BERT model based on basic patient information, such as: "The patient is a 45-year-old male with a malignant bone tumor located in the distal femur." Clinical BERT captures the contextual relationships between words in the text through its multi-layered Transformer structure, generating a high-dimensional semantic feature representation, denoted as F_GT. This model effectively models the semantic information of clinical text, thus providing accurate text features for cross-modal alignment.

[0072] 2.3.2 Contrastive Learning and Feature Alignment Image features F_GI and text features F_GT are input into the contrastive learning module. This embodiment uses the InfoNCE loss function, which is defined as follows: ; Where sim represents cosine similarity, used to measure the similarity of image and text features. τ The temperature parameter is used to adjust the smoothness of the contrast distribution, and N is the batch size.

[0073] 2.4 Local Mutual Information Optimization Module Since CLIP focuses more on global semantic consistency, this global feature alignment method may ignore local details. To learn more fine-grained information, this embodiment does not use CLIP in local feature learning, but proposes a method to maximize local mutual information by combining the lesion area and lesion attributes of the X-ray film.

[0074] 2.4.1 Image and Text Feature Extraction Local lesion image features are extracted based on the CA_SwinT_Local_Model network to generate fused features of the anteroposterior and lateral local images, denoted as F_LI. This feature fuses complementary information from the anteroposterior and lateral images, effectively capturing the global semantic features of the patient images.

[0075] Local text features are based on X-ray lesion information enhanced with patient knowledge, and the text is processed using the Clinical BERT model. Clinical BERT captures the contextual relationships between words in the text through its multi-layer Transformer structure, generating high-dimensional semantic feature representations, denoted as F_LT.

[0076] 2.4.2 Maximizing Local Mutual Information The local image features F_LI and local text features F_LT are input into a local mutual information discriminator, which outputs local mutual information values. The goal is to maximize the local mutual information. Local mutual information is defined as follows: .

[0077] in, This represents the features of the i-th local region in the image. Let represent the features of the j-th sentence in the text. To optimize mutual information, the part with the largest mutual information between each pair of local image features and text features is selected for optimization: .

[0078] In this way, each local feature of the image is ensured to match its most relevant text feature, thus achieving more accurate feature alignment.

[0079] 2.4.3 Local Mutual Information Estimation The estimation of local mutual information depends on a discriminator. Its function is to distinguish input feature pairs It comes from the joint distribution or the product of marginal distributions Based on the Donsker-Varadhan (DV) variational formula, the optimization objective of mutual information is defined as: .

[0080] Among them, the expected value of positive sample pairs (from the joint distribution) Indicates from the joint distribution Medium-sampled positive sample pairs Then, the average score of these positive sample pairs is used by the discriminator f. This part aims to improve the relevance of positive sample pairs and encourage the model to judge these samples as having high mutual information; the log-expected value of negative sample pairs (from the product of marginal distributions) is also used. Indicates distribution from the margins Randomly combined negative sample pairs Then, the discriminator f calculates the log-expected value of the exponential scores for these negative sample pairs. This part aims to reduce the correlation of negative sample pairs and prevent the model from incorrectly classifying irrelevant feature pairs as positively correlated. This is achieved by optimizing the discriminator parameters. To maximize the lower bound of mutual information, the final loss function for this part is: .

[0081] 3. Classification Head Module and Overall Loss Function The goal of the classification head module is to receive global image feature representations and complete the bone tumor classification task. According to the aforementioned framework, the classification head receives the feature F_GI, which is a concatenation of global multi-view features, and then outputs the predicted probability of the class through multi-layer nonlinear mapping. The loss function for this part is: in, It is the actual category of the i-th sample. Let be the probability of predicting the class of the i-th sample, and c be the number of classes. The loss function of the entire algorithm aims to jointly optimize global contrastive learning, maximizing local mutual information, and the performance of the classification task. The overall goal of the loss function is to comprehensively optimize the following three parts through multi-task learning: ; in, The weighting coefficient controls the importance of each task in the overall optimization.

[0082] The final trained model can be used to analyze global X-ray images of cases and identify bone tumor types.

[0083] Corresponding to the above method, the present invention also provides an apparatus / system including a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus / system performs the steps of the method as described above.

[0084] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0085] In summary, the bone tumor fine-grained classification model training, classification method, and apparatus described in this invention construct multimodal positive samples containing global and local anteroposterior and lateral X-ray images, lesion attributes, and basic patient information. Highly similar samples are filtered and eliminated based on text semantic similarity to construct high-quality negative samples. A dual image encoder is used to extract global and local anteroposterior and lateral fusion features, while a medical knowledge base is combined to transform lesion attribute keywords into "entity-translation-existence" triples, which are then used in conjunction with a text encoder to extract two levels of semantic features. Finally, the model is trained end-to-end using InfoNCE contrast loss, local mutual information loss based on the DV variational formula, and classification loss to obtain a bone tumor fine-grained classification model. This invention achieves precise alignment between global macroscopic semantics and local fine-grained lesion features, effectively overcomes the semantic gap in medical terminology through knowledge augmentation, and improves training stability through a negative sample selection strategy that resists false negatives. Ultimately, it significantly improves the accuracy and robustness of bone tumor subtype classification, providing reliable auxiliary decision-making basis for precise clinical diagnosis.

[0086] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0087] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0088] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0089] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0090] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated and with authorization from the owner of the relevant device.

Claims

1. A method for training a fine-grained classification model for bone tumors, characterized in that, The method includes the following steps: Image and text data from multiple bone tumor cases were acquired. The image data included global X-ray images and local lesion X-ray images, both of which included anteroposterior and lateral views. The text data included lesion attributes and basic patient information, with bone tumor type labeled as a positive sample. The lesion attributes included lesion type, lesion location, presence of soft tissue mass, and tumor enhancement. The basic patient information included patient gender, age, lesion location, and pathological examination classification results. Negative samples are constructed by randomly replacing the text data in the positive samples with the text data from different bone tumor cases, and the negative samples with a similarity of text data before and after replacement exceeding a set threshold are removed. For each positive or negative sample, the global X-ray image is processed by a first image encoder to obtain global anterolateral fusion features, and the local lesion X-ray image is processed by a second image encoder to obtain local anterolateral fusion features. The feature keywords of the lesion attributes are queried from a preset medical knowledge base to obtain a natural language description. The feature keywords, the natural language description, and the existence marker are used as a triple, and a first semantic feature is extracted by a text encoder. The patient's basic information is used to extract a second semantic feature by the text encoder. The global anterolateral fusion features of the positive sample are further processed by a fully connected layer to output a bone tumor subtype classification prediction value. The local lateral fusion features of positive and negative samples and the first semantic features are input into the mutual information discriminator to output local mutual information. The lower bound of mutual information is calculated based on the DV variational formula, and the mutual information loss is constructed by maximizing the lower bound of mutual information. The InfoNCE contrast loss is constructed based on the global lateral fusion features of positive and negative samples and the second semantic features. The classification loss is constructed by minimizing the deviation between the predicted value and the label. The parameters of the first image encoder, the second image encoder, the text encoder, the fully connected layer, and the mutual information discriminator are updated by combining the mutual information loss, the contrast loss, and the classification loss. The first image encoder and the fully connected layer are combined into a fine-grained classification model for bone tumors.

2. The method for training a fine-grained classification model for bone tumors according to claim 1, characterized in that, The method further includes: removing noise, artifacts, missing or duplicate parts of key components from the image data; and performing structured processing on the text data.

3. The method for training a fine-grained classification model for bone tumors according to claim 1, characterized in that, The negative samples whose text data similarity before and after replacement exceeds a set threshold are removed, including: The Clinical BERT model, fine-tuned for the medical field, is used to process the text data before and after replacement in the negative samples to obtain embedded vectors, and cosine similarity is calculated. Negative samples with cosine similarity higher than a set threshold are removed.

4. The method for training a fine-grained classification model for bone tumors according to claim 1, characterized in that, The first image encoder and the second image encoder adopt a ResNet-50 / 101 model, a DenseNet model, a ViT model, or a SwinTransformer network. The SwinTransformer network is composed of multiple Transformer blocks, and each Transformer block includes a multi-head attention module and a multi-layer perceptron module. The text encoder employs a BioBERT model, a PubMedBERT model, a BERT model, a RoBERTa model, or a Clinical BERT model fine-tuned for the medical field.

5. The method for training a fine-grained classification model for bone tumors according to claim 1, characterized in that, The local lateral fusion features and the first semantic features are input into the mutual information discriminator to output local mutual information. A lower bound for the mutual information is calculated based on the DV variational formula. The mutual information loss is constructed by maximizing this lower bound, including: The optimization objective is defined based on the aforementioned DV variational formula: ； The formula for calculating the mutual information loss is: ； in, f The mutual information discriminator is used to determine the local positive-lateral fusion features. and the first semantic feature Originating from the positive sample or the negative sample; This represents the expected score of the mutual information discriminator for the positive sample. This represents the expected logarithm of the score given by the mutual information discriminator to the negative sample. This represents the mutual information loss; The InfoNCE contrast loss is constructed based on the global positive and negative lateral fusion features of the positive and negative samples and the second semantic features, including: ； in, This represents the contrast loss. The global positive lateral fusion feature representing the positive sample at the anchor point. The second semantic feature represents the positive sample of the anchor point. The second semantic feature of the negative sample is represented by sim; cosine similarity is calculated, and N is the batch size. The temperature parameter is used; the formula for calculating the classification loss is: ； in, This represents the classification loss. This represents the label of the i-th positive sample. Let C represent the predicted value of the bone tumor subtype classification of the i-th positive sample, where C is the number of categories and N is the total number of positive samples. The formula for calculating the joint loss is: ； in, These are the weighting coefficients. The joint loss is mentioned above.

6. The method for training a fine-grained classification model for bone tumors according to claim 1, characterized in that, The preset medical knowledge base uses UMLS, SNOMED CT, or MeSH medical terminology sets.

7. A method for fine-grained classification of bone tumors, characterized in that, The method includes the following steps: Obtain global X-ray images and local lesion X-ray images of the case to be analyzed, input the fine-grained bone tumor classification model in the bone tumor fine-grained classification model training method as described in any one of claims 1 to 6, and output the bone tumor classification results.

8. A fine-grained bone tumor classification device, comprising a processor, a memory, and a computer program or instructions stored in the memory, characterized in that, The processor is configured to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images