Image genomics prediction model construction and image annotation method

Abstract

The invention relates to the technical field of computer vision, in particular to an image genomics prediction model construction and image annotation method, which comprises the following steps of: respectively acquiring batch gene sequencing sample data and disease sample image data with a pairing relationship; sequencing the proportion of each first cell type in the batch gene sequencing sample data; determining a first proportion group and a second proportion group corresponding to each second cell type in the gene sequencing sample data, wherein the second cell type represents a cell type conforming to a preset difference in the first cell type; and iteratively constructing an image genomics prediction model corresponding to each cell type by taking the first proportion group and the second proportion group corresponding to each second cell type as tags and combining the disease sample image data and the gene sequencing sample data. According to the scheme, the tumor microenvironment is fused into image marking, the image marker related to the tumor microenvironment is obtained through an unsupervised method, and the method has important significance on early screening of tumors and cancer detection.

Description

Technical Field

[0001] This specification relates to the field of computer vision technology, and in particular to a method for constructing an imaging genomics prediction model and an image annotation method. Background Art

[0002] Cancer is one of the leading causes of death in humans, and current cancer detection relies primarily on medical imaging. However, current tumor annotation methods for medical images cannot directly identify molecular subtypes and the tumor microenvironment (TME), hindering accurate cancer diagnosis and treatment. In current clinical practice, tumor annotation in medical images relies heavily on manual delineation by radiologists, which is time-consuming and labor-intensive, subject to significant variability, and limited scalability.

[0003] In recent years, AI-based automatic segmentation methods such as U-Net and Transformer have significantly improved efficiency, but they still have certain limitations. AI-based image annotation methods are categorized as supervised, semi-supervised, and unsupervised. Current supervised and semi-supervised segmentation methods are annotation-dependent, requiring trained models with labeled data. However, medical image annotation is extremely costly and has strong technical barriers, making data scarcity particularly problematic for rare cancers or complex cases.

[0004] Unsupervised methods typically rely on iterative training with pseudo-labels, but the accuracy of pseudo-labels is limited by the performance of the initial model. Furthermore, many current segmentation methods focus on tumor segmentation, such as lesion boundary identification, while ignoring the structural and functional heterogeneity of the TME. Studies have shown that the spatial distribution of the TME is closely related to tumor metastasis, drug resistance, and immunotherapy response. Therefore, it is necessary to propose an unsupervised image annotation method that can annotate TME-related distributions on images without pre-setting tumor annotations for training. Summary of the Invention

[0005] To solve the problems in the prior art, this specification provides a method for constructing an imaging genomics prediction model and annotating an image, the method comprising: separately collecting batch gene sequencing sample data and disease sample image data having a paired relationship; determining, based on the proportion of each first cell type in the batch gene sequencing sample data, a first proportion group and a second proportion group corresponding to the proportion of each second cell type in the batch gene sequencing sample data, wherein the first cell type represents a cell type associated with the disease, and the second cell type represents a cell type in the first cell type that meets a preset difference; using the first proportion group and the second proportion group corresponding to the proportion of each second cell type as labels, and combining the disease sample image data and the batch gene sequencing sample data, iteratively constructing an imaging genomics prediction model corresponding to each cell type.

[0006] According to one aspect of the embodiments of this specification, the batch gene sequencing sample data and disease sample image data are collected from a variety of different disease types, and the second cell type is determined in the following manner: using the t-test and the Wilcox test respectively, the proportion of each first cell type in the target disease type and the non-target disease type in the multiple different disease types is determined, and the first probability value of the t-test and the second probability value of the Wilcox test are determined; the cell type whose first probability value and the second probability value are both lower than the preset threshold is determined to be a cell type that meets the preset difference.

[0007] According to one aspect of an embodiment of the present specification, the first cell type is determined by: determining the proportion of multiple cell types and the expression of each cell type in the batch gene sequencing data based on a deconvolution algorithm; and screening out the first cell type that meets preset requirements from the multiple cell types.

[0008] According to one aspect of the embodiments of this specification, iteratively constructing an imaging genomics prediction model corresponding to each cell type further includes: inputting the disease sample image data and batch gene sequencing sample data into an initial model to obtain an initial prediction result; classifying the initial prediction result according to a preset threshold to obtain a category corresponding to each initial prediction result; calculating the loss value of the classified initial prediction result and the first proportion group or the second proportion group corresponding to the category, iteratively updating the initial model according to the loss value until the preset conditions are met, and constructing an imaging genomics prediction model.

[0009] According to one aspect of an embodiment of this specification, the initial prediction result is determined by: extracting image features from the disease sample image data, and extracting expression features of target disease type cells from the batch gene sequencing sample data; fusing the image features and the expression features of the target disease type cells to obtain a combined feature; and inputting the combined feature into a multi-layer perceptron to obtain an initial prediction result.

[0010] According to one aspect of an embodiment of the present specification, the first proportion group and the second proportion group are determined in the following manner: the proportion of each second cell type is sorted in a preset order to obtain a cell type proportion data set; the cell type proportion data set is divided according to the median, and the group whose cell type proportion exceeds the median is used as the first proportion group, and the group whose cell type proportion does not exceed the median is used as the second proportion group.

[0011] An embodiment of the present specification also provides an image annotation method, which applies the imaging genomics prediction model construction method, including: determining the prediction score of the prediction category corresponding to the disease image in the external validation set based on the attention mechanism, wherein the prediction category is the prediction result obtained by inputting the disease image in the external validation set into the imaging genomics prediction model; determining the weight of the feature map channel based on the prediction score and the feature layer in the imaging genomics prediction model; multiplying the weight by the feature map to obtain a weighted feature map for each feature map channel, and generating a two-dimensional heat map; converting the area in the two-dimensional heat map where the pixel score value exceeds a preset ratio into an image annotation, wherein the image annotation is an image annotation related to the microenvironment.

[0012] This specification also provides an image annotation method, including: obtaining a target disease image; inputting the target disease image into the imaging genomics prediction model to obtain an annotated area of ​​the disease image, wherein the imaging genomics prediction system is composed of the imaging genomics prediction model and an attention mechanism.

[0013] This specification provides an imaging genomics prediction model construction device, which includes: a data acquisition unit, which is used to respectively collect batch gene sequencing sample data and disease sample image data with a paired relationship; a determination unit, which is used to determine a first proportion group and a second proportion group corresponding to each second cell type in the batch gene sequencing sample data based on the proportion of each first cell type in the gene sequencing sample data, wherein the first cell type represents a cell type associated with the disease, and the second cell type represents a cell type in the first cell type that meets a preset difference; a model construction unit, which is used to use the first proportion group and the second proportion group corresponding to each second cell type as labels, and combine the disease sample image data and the batch gene sequencing sample data to iteratively construct an imaging genomics prediction model corresponding to each cell type.

[0014] This specification provides an image annotation device, which includes: an image acquisition unit for acquiring a target disease image; and a labeling region determination unit for inputting the target disease image into the imaging genomics prediction model to obtain a labeling region of the disease image.

[0015] An embodiment of the present specification provides an imaging genomics prediction model construction device, which includes: a data acquisition unit, which is used to respectively collect batch gene sequencing sample data and disease sample image data with a pairing relationship; a determination unit, which is used to determine a first proportion group and a second proportion group corresponding to each second cell type in the gene sequencing sample data based on the proportion of each first cell type and the expression of each first cell type in the gene sequencing sample data, wherein the first cell type represents a cell type associated with the disease, and the second cell type represents a cell type in the first cell type that meets a preset difference; a model construction unit, which is used to use the first proportion group and the second proportion group corresponding to each second cell type as labels, and combine the disease sample image data and the gene sequencing sample data to iteratively construct an imaging genomics prediction model corresponding to each cell type.

[0016] An embodiment of the present specification also provides an image annotation device, which includes: a prediction score determination unit, which is used to determine the prediction score of the prediction category corresponding to the disease image in the external validation set based on the attention mechanism, and the prediction category is the prediction result obtained by inputting the disease image in the external validation set into the imaging genomics prediction model; a weight determination unit, which is used to determine the weight of the feature map channel based on the prediction score and the feature layer in the imaging genomics prediction model; a heat map determination unit, which is used to multiply the weight by the feature map to obtain a weighted feature map for each feature map channel and generate a two-dimensional heat map; and an annotation unit, which is used to convert the area in the two-dimensional heat map where the pixel score value exceeds a preset ratio into an image annotation, and the image annotation is an image annotation related to the microenvironment.

[0017] An embodiment of this specification also provides an image annotation device, which includes: an image acquisition unit for acquiring target disease image data; and an annotation area determination unit for inputting the target disease image into a genomics prediction system to obtain an annotation area of ​​the disease image.

[0018] An embodiment of this specification also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the imaging genomics prediction model construction and image annotation method are implemented.

[0019] The embodiments of this specification also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the imaging genomics prediction model construction and image annotation method.

[0020] This solution integrates the tumor microenvironment into image annotation and obtains image markers related to the tumor microenvironment through unsupervised methods, solving the problem that microenvironment annotations cannot be obtained manually. It is of great significance for early tumor screening and cancer detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 FIG2 is a schematic diagram showing the structure of an imaging genomics prediction model construction and image annotation method system according to an embodiment of the present invention;

[0023] Figure 2 The figure shows a flow chart of a method for constructing an imaging genomics prediction model according to an embodiment of this specification;

[0024] Figure 3 Shown is a flow chart of a method for determining a first cell type according to an embodiment of this specification;

[0025] Figure 4 Shown is a flow chart of a method for iteratively updating an imaging genomics prediction model according to an embodiment of this specification;

[0026] Figure 5 The figure shows a flow chart of a method for determining an initial prediction result according to an embodiment of the present specification;

[0027] Figure 6 Shown is a flow chart of a method for determining a second cell type according to an embodiment of this specification;

[0028] Figure 7 The figure shows a flow chart of a method for determining the first ratio group and the second ratio group corresponding to each cell type according to an embodiment of this specification;

[0029] Figure 8 The figure shows a flow chart of an image annotation method according to an embodiment of the present specification;

[0030] Figure 9 The figure shows a schematic diagram of the structure of an imaging genomics prediction model construction device according to an embodiment of this specification;

[0031] Figure 10 The figure shows a schematic diagram of the structure of an image annotation device according to an embodiment of the present specification;

[0032] Figure 11The figure shows a schematic diagram of the structure of an image annotation device based on an imaging genomics prediction model according to an embodiment of this specification;

[0033] Figure 12 Shown is a schematic diagram of a UNet module according to an embodiment of this specification;

[0034] Figure 13 FIG2 is a schematic diagram of an SMLP module according to an embodiment of the present invention;

[0035] Figure 14 The figure shows a schematic structural diagram of an initial model according to an embodiment of the present specification;

[0036] Figure 15 Shown is a flowchart of an image annotation method based on an imaging genomics prediction model according to an embodiment of this specification;

[0037] Figure 16a Schematic diagram showing the image annotation results of an image-CAF model on an external validation set according to an embodiment of the present invention;

[0038] Figure 16b Schematic diagram showing image annotation results of another image-CAF model on an external validation set according to this illustrative embodiment;

[0039] Figure 17 The figure shows a schematic diagram of the structure of a computer device according to an embodiment of the present specification.

[0040] Description of the accompanying symbols:

[0041] 101. Terminal;

[0042] 102. Server;

[0043] 901, data acquisition unit;

[0044] 902. Determine the unit;

[0045] 903, model building unit;

[0046] 1001. Prediction score determination unit;

[0047] 1002. Weight determination unit;

[0048] 1003. Heat map determination unit;

[0049] 1004, marking unit;

[0050] 1101, image acquisition unit;

[0051] 1102. Marking area determination unit;

[0052] 1702. Computer equipment;

[0053] 1704, processor;

[0054] 1706, memory;

[0055] 1708, driving mechanism;

[0056] 1710, input / output module;

[0057] 1712. Input devices;

[0058] 1714. Output devices;

[0059] 1716. Presentation equipment;

[0060] 1718. Graphical User Interface;

[0061] 1720, network interface;

[0062] 1722, communication link;

[0063] 1724. Communication bus. DETAILED DESCRIPTION

[0064] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative work are within the scope of protection of this specification.

[0065] It should be noted that the terms "first," "second," and the like in the description and claims of this specification and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0066] This specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many orderings and does not represent the only execution order. When a system or device product is actually executed, the method can be executed in the order shown in the embodiments or the drawings or in parallel.

[0067] In this specification, the tumor microenvironment is the environment surrounding the tumor, including surrounding blood vessels, immune cells, fibroblasts, signaling molecules, and extracellular matrix. Within the tumor microenvironment, the cells and fibers in the tumor's surroundings can influence tumor development. Therefore, this application requires accurate identification of the tumor's microenvironment.

[0068] It should be noted that the imaging genomics prediction model and image annotation method of this specification can be used in the field of computer vision and in the field of artificial intelligence technology. This specification does not limit the application fields of the imaging genomics prediction model and image annotation method.

[0069] Figure 1 The diagram shows a schematic diagram of a system for constructing an imaging genomics prediction model and an image annotation method according to an embodiment of the present invention. The system may include a terminal 101 and a server 102. The terminal 101 and the server 102 communicate with each other via a network. The network may include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, and is connected to a website, a user device (e.g., a computing device), and a back-end system. The terminal 101 may collect batch gene sequencing sample data and disease sample image data with a pairing relationship and send them to the server 102. After receiving the batch gene sequencing sample data and disease sample image data, the server 102 determines the first ratio group and the second ratio group corresponding to each second cell type in the gene sequencing sample data based on the proportion of each first cell type in the batch gene sequencing sample data. The first ratio group and the second ratio group corresponding to each second cell type are used as labels. The imaging genomics prediction model corresponding to each cell type is iteratively constructed in combination with the disease sample image data and the gene sequencing sample data. The imaging genomics prediction model is then sent to the terminal 101.

[0070] In the embodiments of this specification, the server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms.

[0071] In an optional embodiment, the terminal 101 may include, but is not limited to, electronic devices such as self-service terminals, desktop computers, tablet computers, laptop computers, and smart wearable devices. Optionally, the operating system running on the electronic device may include, but is not limited to, Android, iOS, Linux, and Windows. Of course, the terminal 101 is not limited to the aforementioned electronic devices having a certain physical form; it may also be software running on the aforementioned electronic devices.

[0072] In addition, it should be noted that Figure 1 What is shown is only an application environment provided by the present disclosure. In actual application, multiple terminals 101 may be included, and this specification does not limit this.

[0073] Figure 2 The flowchart of a method for constructing an imaging genomics prediction model according to an embodiment of this specification is shown. This method is used to train and construct an imaging genomics prediction model that can predict the microenvironment in which a tumor resides. Specifically, it includes steps 201 to 203:

[0074] Step 201 : Collect batch gene sequencing sample data and disease sample image data that have a paired relationship.

[0075] During this step, when acquiring sample data for model training during the model training phase, the collected batches of gene sequencing sample data and disease sample image data are paired. Specifically, gene sequencing sample data and disease sample image data are collected from the same patient, and further, gene sequencing sample data and disease sample image data are collected from multiple patients. Only after these paired data are collected can the tumor microenvironment be analyzed. In other words, if only disease sample image data is collected without gene sequencing sample data, the tumor microenvironment cannot be analyzed.

[0076] In some embodiments of the present specification, breast cancer is used as the target disease for research, so this step requires the collection of gene sequencing sample data and disease sample image data of patients with triple-negative breast cancer (TNBC) and non-triple-negative breast cancer (non-TNBC). Wherein, triple-negative breast cancer (TNBC) and non-triple-negative breast cancer (non-TNBC) are both breast cancer. Based on this, image data and genome data with a pairing relationship are collected from the Cancer Genome Atlas and Cancer Image Archives. Specifically, the batch gene sequencing sample transcriptome data of TCGA-BRCA is downloaded from the Cancer Genome Atlas (TCGA) data set; the TCGA-BRCA image data is downloaded from the Cancer Imaging Archive (TCIA) database. In the embodiments of the present specification, the TCGA atlas includes tumor samples from a large number of breast cancer patients, that is, the patient's genome, transcriptome, epigenome and clinical data.

[0077] In this document, paired invasive breast cancer TCGA-BRCA image-genomic paired data, totaling 117 samples, were obtained from TCGA and TCIA. The data were divided into a training set (88 samples) and an internal validation set (29 samples). Images and associated tumor annotations were obtained from ISPY1 and BRCA-NACT-Pilot, respectively, with 81 and 29 samples.

[0078] Step 202: Determine a first proportion group and a second proportion group corresponding to the proportion of each second cell type in the batch gene sequencing sample data based on the proportion of each first cell type in the batch gene sequencing sample data, wherein the first cell type represents a cell type associated with the disease, and the second cell type represents a cell type in the first cell type that meets a preset difference.

[0079] In this step, the first cell type represents a cell type associated with the disease, which can be further understood as a cell type with a greater correlation with the disease. The disease here can represent one or more diseases, and is not limited to the breast cancer disease described in the above embodiment.

[0080] Specifically, based on the deconvolution algorithm, the proportion of multiple preset cell types in the gene sequencing data and the expression of each preset cell type can be determined, and the multiple preset cell types belong to the first cell type. After determining the proportion of the first cell type, a second cell type that meets the preset difference is further screened out from the first cell type. Among them, there are obvious differences in the second cell type among patients with different types of breast cancer, so the second cell type is extracted from the first cell type for further study. Specifically, in one embodiment of the present specification, by analyzing the cell composition in the tumor microenvironment of patients with triple-negative breast cancer (TNBC) and patients with non-triple-negative breast cancer (non-TNBC), a cell type with significant differences is extracted, which is the second cell type. For details about the process of determining the second cell type, see Figure 3 and Figure 6 describe.

[0081] In step 203 , the first proportion group and the second proportion group corresponding to the proportion of each second cell type are used as labels, and the image data of the disease sample and the batch gene sequencing sample data are combined to iteratively construct an imaging genomics prediction model corresponding to each cell type.

[0082] In this step, the cell type ratio reflects the composition of various types of cells in a certain environment (such as blood, tumor tissue, inflammatory sites, etc.). The ratio of each second cell type can be divided into a first ratio group and a second ratio group according to the median. The first ratio group and the second ratio group are used as labels for model prediction and compared with the initial prediction results of the model. Among them, the initial prediction result of the model is: the result of the first ratio group or the second ratio group of a specific cell type in the second cell type for the input data. For a detailed description of the first ratio group and the second ratio group corresponding to the second cell type ratio in this step, see Figure 7 .

[0083] This manual integrates the tumor microenvironment into image annotation and obtains image markers related to the tumor microenvironment through unsupervised methods. This solves the problems of being unable to obtain microenvironment annotations manually and traditional segmentation methods requiring pre-training with labeled images. It is of great significance for early tumor screening and cancer detection.

[0084] Figure 3 The flowchart of a method for determining a first cell type according to an embodiment of the present specification is shown, specifically including steps 301 to 302:

[0085] Step 301: Determine the proportion of multiple cell types and the expression of each cell type in the batch gene sequencing data based on a deconvolution algorithm.

[0086] In the examples of this specification, each sample data in the batch gene sequencing sample data cannot be accurately measured at the cellular level. Therefore, it is necessary to use deconvolution to refine the batch gene sequencing sample data to determine which cell types are included in the batch gene sequencing data and what the proportion of the included cell types is. Specifically, by constructing a deconvolution model, all cell types, the proportion of all cell types, and the expression of all cell types in the batch gene sequencing sample data can be subdivided.

[0087] Specifically, some known single-cell sequencing data are used as a reference matrix, and a template is constructed based on the Bayesian algorithm and single-cell sequencing data. The template is input into the deconvolution model as prior knowledge to obtain the proportion and cell expression of each cell type in the batch gene sequencing sample data.

[0088] Step 302 : Screening out a first cell type that meets preset requirements from a plurality of cell types.

[0089] In some embodiments of this specification, breast cancer is the target disease for research, and therefore the applicant focuses on cell types associated with breast cancer in bulk gene sequencing data. The deconvolution model in step 301 analyzes the bulk gene sequencing data to determine that there are multiple cell types, from which a first cell type associated with breast cancer is further screened. In this embodiment of the specification, B cells, CD4+ T cells, CD8+ T cells, cancer-associated fibroblasts (CAFs), macrophages, endothelial cells, fibroblasts, and breast cancer cells are screened as the first cell type.

[0090] Among the first cell types, B cells are the core of the adaptive immune system, developing and maturing in the bone marrow and recognizing antigens through the BCR (B cell receptor). Their role in breast cancer is to: fight tumors, enhance immune responses through antibody-mediated ADCC (antibody-dependent cellular cytotoxicity) or form tertiary lymphoid structures (TLS), and improve prognosis.

[0091] CD4+T cells (also known as helper T cells), Th1: secrete IFN-γ, which can activate macrophages and CD8+T cells and enhance anti-tumor immunity. Th2: secrete IL-4 and IL-5, promote humoral immunity, but may inhibit Th1 response. Th17: secrete IL-17, participate in chronic inflammation, and may promote tumor progression. Treg (regulatory T cells): express FoxP3, suppress immune response by secreting IL-10 and TGF-β, and help tumors escape. The role in breast cancer is: Treg infiltration is usually associated with poor prognosis, while Th1 cells may enhance anti-tumor immunity, but the overall effect depends on the balance of subtypes.

[0092] CD8+ T cells (cytotoxic T cells) are designed to directly recognize and kill cancer cells that present antigens with MHC class I molecules, releasing perforins and granzymes to induce apoptosis. High levels of CD8+ T cells in breast cancer generally predict a better prognosis, but they may be "exhausted" due to activation of checkpoints such as PD-1 / PD-L1. Immune checkpoint inhibitors (such as anti-PD-1) can restore their activity.

[0093] Cancer-associated fibroblasts (CAFs) are derived from normal fibroblasts or mesenchymal stem cells in the tumor microenvironment. CAFs are closely associated with chemotherapy resistance, immunosuppression, and metastasis, and are potential therapeutic targets.

[0094] There are two types of macrophages: M1 and M2. M1 is pro-inflammatory, secreting ROS and NO to kill tumors. M2 (tumor-associated macrophages, TAMs) are anti-inflammatory, secreting VEGF (which promotes angiogenesis), TGF-β (which inhibits T cells), and EGF (which promotes tumor growth).

[0095] Fibroblasts maintain the structure of connective tissue and secrete ECM components (such as collagen and fibronectin). Normal fibroblasts can suppress tumors, but they are transformed into cancer-promoting CAFs in the tumor microenvironment.

[0096] Further, in the subsequent processing, the eight first cell types are tested and analyzed to determine the second cell type. Figure 4 describe.

[0097] Figure 4 The flowchart of a method for iteratively updating an imaging genomics prediction model according to an embodiment of the present specification is shown, specifically including steps 401 to 403:

[0098] Step 401: Input the disease sample image data and batch gene sequencing sample data into the initial model to obtain an initial prediction result.

[0099] The initial model is Figure 14 As shown in Figure 1, the initial model specifically includes a UNet module, an SMLP module, and an MLP module. The UNet module is used to extract image features from disease sample image data.

[0100] The SMLP module is used to extract gene features related to target disease types in gene sequencing samples. Figure 13As shown, the input layer of SMLP is M-dimensional data, and the output layer is N-dimensional data. SMLP compresses M-dimensional data into N-dimensional data, and is used to reduce the dimension and extract features from a large amount of gene sequencing expression data, and screen out gene expressions related to breast cancer. The SMLP module introduces a one-to-one connection layer with a ReLU activation function before the original multi-layer perceptron (MLP) to determine the features that play an important role in the prediction results. In this specification, a patient's batch gene sequencing data may contain about 60,000 gene expression data. After normalization and other preprocessing, the gene expression data input to SMLP is more than 20,000. After SMLP feature extraction, the gene expression data of breast cancer cells may be only more than 10,000. During the initial model update process, the parameters in the UNet module and the SMLP module are continuously updated. During the UNet module update process, the areas with higher parameters focus on images that are more relevant to the microenvironment. During the SMLP module update process, the areas with higher parameters focus on genes that are more relevant to the microenvironment.

[0101] In this step, the process of determining the initial prediction results is detailed in Figure 5 describe.

[0102] In step 402, the initial prediction results are classified according to a preset threshold to obtain a category corresponding to each initial prediction result. In step 301, the initial prediction result can be a prediction score or a prediction probability value. The initial prediction result is compared with the preset threshold to classify the initial prediction result into a specific category. For example, if the initial prediction result is 0.6 and the preset threshold is 0.5, the initial prediction result is greater than the preset threshold, and the initial prediction result is classified as a high-value category. For another example, if the initial prediction result is 0.3, which is lower than the preset threshold, the initial prediction result is classified as a low-value category.

[0103] Step 403 : Calculate the loss value between the classified initial prediction result and the first ratio group or the second ratio group, iteratively update the initial model according to the loss value until a preset condition is met, and construct a radiogenomics prediction model.

[0104] In this specification, the first ratio group and the second ratio group may each represent at least one of a high ratio group and a low ratio group. For example, the first ratio group represents a high ratio group, and the second ratio group represents a low ratio group; or alternatively, the first ratio group represents a low ratio group, and the second ratio group represents a high ratio group. This application does not limit whether the first ratio group and the second ratio group represent a high ratio group or a low ratio group.

[0105] Assuming that the first proportion group represents the high proportion group and the second proportion group represents the low proportion group, based on the example in step 302, if the initial prediction result after classification is classified as the high-value class, the loss value is calculated by combining the initial prediction result with the labels composed of the first proportion group, and the parameters in the initial model are further iteratively trained based on the loss value. If the initial prediction result after classification is classified as the low-score class, the loss value is calculated by combining the initial prediction result with the labels composed of the second proportion group, and the initial model is further iteratively trained.

[0106] Specifically, the initial model is updated by iteratively updating the parameters of the one-to-one connection layer. When the initial model is iteratively trained a preset number of times, or the loss function converges to a preset threshold, the iterative training of the initial model is complete, and the imaging genomics prediction model is constructed. This imaging genomics prediction model links images with the microenvironment.

[0107] During model training, this specification uses high- and low-proportion groups of significantly different cell types as label data, eliminating the need for any image annotation data to obtain tumor environmental annotations, reducing training costs and improving model training efficiency. Based on the different label types, image-CAF prediction models, image-CD4 prediction models, image-macrophage prediction models, and image-epithelial cell prediction models are generated.

[0108] After constructing the imaging genomics prediction model, we further evaluated it using a five-fold cross-validation method. The dataset was divided into five parts, with four parts used as training data and one part used as validation data for each trial. Each trial yielded a corresponding accuracy rate, and the average of the five results was used as an estimate of the algorithm's accuracy. The accuracy and AUC for all four image-microenvironment prediction models were 1.

[0109] Figure 5 The flowchart of a method for determining an initial prediction result according to an embodiment of the present specification is shown, which specifically includes steps 501 to 503:

[0110] Step 501 : extracting image features from the disease sample image data, and extracting expression features of target type cells from the batch gene sequencing sample data.

[0111] In this step, the UNet module is used to extract the image features in the disease sample image data, and the SMLP module is used to extract the genomic features in the gene sequencing sample data. Figure 12 As shown in the figure, the green part represents the extracted image features, and the yellow part represents the extracted genomic features. Figure 12As shown in Figure 1, the UNet module consists of four downsampling modules (encoder modules), four upsampling modules (decoder modules), and a bottom connection layer. The bottom connection layer is used to link the upsampling modules and downsampling modules. Each module contains two convolutional layers.

[0112] according to Figure 4 As described in [1], SMLP is used to screen for genes associated with breast cancer from a large amount of gene sequencing expression data. Therefore, the target cell type in this step is breast cancer cells. SMLP ultimately screens and identifies the genomic signatures of breast cancer cells.

[0113] Step 502: Fusing the image features and the expression features of the target type cells to obtain a combined feature.

[0114] In the embodiments of this specification, the image features can be processed into the form of image feature vectors, the genomic features can be processed into vector form, and the image feature vectors and the genomic feature vectors of breast cancer cells can be spliced ​​into a joint feature vector. Among them, a cross-modal comparative learning method (such as CLIP) can be used to align the image features and genomic features to obtain a combined feature. Alternatively, a direct splicing method can be used to merge the image features with the expression features of the target type of cells.

[0115] Step 503: input the combined features into a multi-layer perceptron to obtain an initial prediction result.

[0116] The merged features are input into the MLP, and the initial prediction result output by the MLP is the classification result corresponding to the input data, that is, the prediction of which category the input data belongs to. In this specification, labels are obtained based on four different types, and the same input data is input into four models respectively to obtain four initial prediction results. Specifically, the initial prediction results are: the input data belongs to the first proportion group or the second proportion group of CAFs, the input data belongs to the first proportion group or the second proportion group of CD4+T, the input data belongs to the first proportion group or the second proportion group of macrophages, and the input data belongs to the first proportion group or the second proportion group of epithelial cells. Epithelial cells are high.

[0117] Figure 6 The flowchart of a method for determining a second cell type according to an embodiment of the present specification is shown, specifically including steps 601 to 602:

[0118] In step 601 , a t-test and a Wilcox test are used to determine the proportion of each first cell type in target disease types and non-target disease types among multiple different disease types, and a first probability value corresponding to the t-test and a second probability value corresponding to the Wilcox test are determined.

[0119] In the examples of this specification focusing on breast cancer, patients with breast cancer specifically include those with triple-negative breast cancer (TNBC) and those with non-triple-negative breast cancer (non-TNBC). TNBC and non-TNBC have significant differences in the cellular composition of the tumor microenvironment, which directly impacts their biological behavior, therapeutic response, and clinical prognosis.

[0120] Therefore, it is necessary to analyze cell types with significant differences among the different disease types suffered by the patients corresponding to the data collected in step 601 .

[0121] Therefore, in the examples of this specification, a t-test and Wilcox test were used to analyze the proportions of each cell type in triple-negative breast cancer (TNBC) and non-triple-negative breast cancer (non-TNBC). The specific process is as follows: the differences between the cell proportions are analyzed to obtain cell types with predetermined differences.

[0122] Specifically, t-test types include the single-sample t-test, independent-sample t-test, and paired-sample t-test. Based on the t-distribution and degrees of freedom, a first probability p-value is determined using statistical software or a table. If p < 0.05, the null hypothesis is rejected, and the two group means are considered significantly different. If normality or homogeneity of variance is not met (e.g., p < 0.05), the Wilcoxon test or data transformation is used instead.

[0123] For paired samples, the Wilcoxon Signed-Rank Test was used; for independent samples, the Mann-Whitney U Test was used. The U or W statistic was calculated based on the rank order, and the secondary probability p-value was obtained using software or a table. If p < 0.05, the null hypothesis was rejected, and the distribution positions (medians) of the two groups were considered to be significantly different.

[0124] Step 602 : Determine the cell type for which both the first probability value and the second probability value are lower than a preset threshold value, as a cell type that meets a preset difference.

[0125] In this application, a t-test and a Wilcoxon test were used in combination. When the p-values ​​of the two test methods were both less than 0.05, it was considered that there was a significant difference in the cell type currently being tested, and such cell type was classified as the second cell type.

[0126] The above analysis confirmed that the cell types with significant differences between TNBC and non-TNBC are CAFs, CD4+ T cells, macrophages, and epithelial cells. TNBC typically exhibits a high infiltration of immune cells, particularly CD8+ T cells (cytotoxic T cells), which can account for 20-40%. Macrophages account for a high proportion, approximately 30-50%, and have immunosuppressive functions, secreting factors such as IL-10 and TGF-β to promote tumor growth and metastasis.

[0127] Non-TNBCs, on the other hand, have low immune cell infiltration, particularly in the hormone receptor-positive (HR+) subtype, where CD8+ T cells typically account for less than 10%. Low immune infiltration, driven by hormone / growth factor signaling, relies on endocrine therapy or HER2-targeted therapy. CAFs account for a relatively low proportion (approximately 20-40%): in the HR+ subtype, CAFs may promote tumor growth through paracrine estrogen signaling.

[0128] Therefore, through the above steps, it can be determined that there are significant differences in the proportions of the following types of cells between triple-negative breast cancer (TNBC) and non-triple-negative breast cancer (non-TNBC). Such differences reflect the unique microenvironmental characteristics of TNBC and non-TNBC: high immune infiltration (CD8+ T cells), high CAFs, and M2 macrophage dominance, which are suitable for combined immune targeted therapy.

[0129] Four cell types with significant differences, namely CAFs, CD4+ T cells, macrophages, and epithelial cells, were identified as the second cell type. In subsequent processing, corresponding labels were generated based on the second cell type for model training.

[0130] Figure 7 The flowchart of the method for determining the first ratio group and the second ratio group corresponding to each cell type according to an embodiment of this specification is shown, specifically including steps 701 to 702:

[0131] Step 701 : sort the proportion of each second cell type according to a preset order to obtain a cell type proportion data set.

[0132] In this step, the proportion of cell types that meet the preset differences is sorted in order from large to small, or from small to large, to form a proportion data set corresponding to each cell type. The proportion data is usually in the form of a percentage or decimal. Specifically, the proportion data corresponding to each of the four cell types, CAFs, CD4+T cells, macrophages, and epithelial cells, in the batch gene sequencing sample data collected from a large number of patients are sorted in order from large to small, or from small to large, to obtain a proportion data set corresponding to the second cell type. For example, the proportion of CAFs cells in sample 1 is 18%, the proportion of CAFs cells in sample 2 is 29%, the proportion of CAFs cells in sample 3 is 16%, ... the proportion of CAFs cells in sample n is 52%, etc. The proportion data sets of CD4+T cells, macrophages, and epithelial cells are similar.

[0133] Step 702 : Divide the cell type proportion dataset according to the median, and divide the group whose cell type proportion exceeds the median into the first proportion group; and divide the group whose cell type proportion does not exceed the median into the second proportion group.

[0134] In this step, the median divides the numerical set of cell type proportions into two equal parts. Specifically, the first proportion group may be referred to as the high proportion group, and the second proportion group may be referred to as the low proportion group. Alternatively, the first proportion group may be referred to as the low proportion group, and the second proportion group may be referred to as the high proportion group.

[0135] In one embodiment of the present specification, assuming that the median of the CAFs cell ratio in 117 samples is 26%, this is used as the dividing point. In the embodiment of step 701, the CAFs cell ratio of sample 1 is lower than the median, and sample 1 is divided into the second proportion group; the CAFs cell ratio of sample 2 is higher than the median, and sample 2 is divided into the first proportion group; the CAFs cell ratio of sample 3 is lower than the median, and sample 3 is divided into the second proportion group; the CAFs cell ratio of sample n is higher than the median, and sample n is divided into the first proportion group.

[0136] This application does not limit the method of dividing the first proportion group and the second proportion group, and this application does not limit the method of dividing the proportion of cell types. Take four cell types as research objects, determine the first proportion group and the second proportion group corresponding to these four cell types, and construct imaging genomics prediction models corresponding to these four cell types respectively. In this specification, the image attention area is extracted from the imaging genomics prediction model through the attention mechanism, so as to obtain image annotations related to the microenvironment.

[0137] Figure 8 The flowchart of an image annotation method according to an embodiment of the present specification is shown, which specifically includes steps 801 to 804:

[0138] Step 801: Determine the prediction score of the prediction category corresponding to the disease image in the external validation set based on the attention mechanism, where the prediction category is the prediction result obtained by inputting the disease image in the external validation set into the imaging genomics prediction model.

[0139] In this step, we first obtain the MRI data used in the external validation set of this application from the cancer image archive and input it into the imaging genomics prediction model constructed according to the above description to obtain the predicted category. We then use the CNN module to obtain the predicted score of the predicted category. Assuming that the category corresponding to the initial prediction result is c, the prediction score is expressed as: c .

[0140] Step 802 : Determine the weight of the feature map channel based on the prediction score and the feature layer in the imaging genomics prediction model.

[0141] The feature layer in this manual is the last layer in the imaging genomics prediction model, which can also be understood as the feature layer to be visualized, denoted as feature layer A. The gradient of category c is further calculated: Where k represents the kth channel of the image and A represents the last layer in the imaging genomics prediction model.

[0142] This step further includes: performing adaptive average pooling on the gradient to obtain the weight of each feature map channel, that is, in Represents global average pooling, Z is the product of width i and height j, y c represents the prediction score, and A represents the last layer in the radiogenomics prediction model.

[0143] Step 803: multiply the weight by the feature map to obtain a weighted feature map for each feature map channel, and generate a two-dimensional heat map.

[0144] In this step, the feature map and the weight are multiplied channel by channel to obtain a weighted feature map for each channel. These weighted feature maps are then added channel by channel to obtain a two-dimensional heat map, which represents the importance of each pixel to the target category. The two-dimensional heat map is processed by ReLU to suppress the unimportant parts of the weight. The above process can be expressed as follows:

[0145] During the initial model training process in this manual, the model extracts features from the input data and iteratively updates parameters based on the differences between the features and the labels, thereby optimizing the model. During model optimization and parameter updates, the model updates focus on areas with higher parameters, while areas with lower parameters are ignored. Areas with higher parameters are more closely related to the model's predicted labels. Areas with higher parameters are areas related to the microenvironment.

[0146] The attention mechanism extracts this area, which is the area related to the model's predicted label. Therefore, the attention mechanism in this step is equivalent to visualizing the size of the parameters during the training process. This determines which area in the image has more attention and which area has less attention, and then determines the image annotation area. Figure 16a and Figure 16b As described above, the red part in the figure represents an area of ​​greater concern, that is, the image annotation area, which is also the image microenvironment area.

[0147] Step 804 converts areas in the two-dimensional heat map where pixel scores exceed a preset ratio into image annotations, where the image annotations are microenvironment-related image annotations. In this step, the score of each pixel in the two-dimensional heat map is extracted, and the top 60% of the pixel scores in the two-dimensional heat map are further converted into image annotations to obtain microenvironment-related image annotations. The two-dimensional heat map can be superimposed with disease images in an external validation set and labeled with image annotations, thereby forming a focus area representation on the image.

[0148] In this step, the preset ratio may be 60%, 65% or other values, and this application does not limit the value of the preset ratio.

[0149] This application uses the attention mechanism to achieve unsupervised image annotation, which does not require pre-training with annotated images. Manual annotation cannot obtain microenvironment-related information on the image.

[0150] Figure 9 The figure shows a schematic diagram of the structure of an imaging genomics prediction model construction device according to an embodiment of this specification. This figure describes the basic structure of the imaging genomics prediction model construction device. The functional units and modules therein can be implemented in software, or a general-purpose chip or a specific chip can be used to implement the imaging genomics prediction model construction. The device specifically includes:

[0151] The data collection unit 901 is used to collect batch gene sequencing sample data and disease sample image data that have a paired relationship;

[0152] A determining unit 902 is configured to determine, based on the proportion of each first cell type in the batch gene sequencing sample data, a first proportion group and a second proportion group corresponding to the proportion of each second cell type in the gene sequencing sample data, wherein the first cell type represents a cell type associated with a disease, and the second cell type represents a cell type within the first cell type that meets a preset difference;

[0153] The model construction unit 903 is used to use the first proportion group and the second proportion group corresponding to each second cell type proportion as labels, combine the disease sample image data and batch gene sequencing sample data, and iteratively construct an imaging genomics prediction model corresponding to each cell type.

[0154] Figure 10 The figure shows a schematic diagram of the structure of an image annotation device according to an embodiment of this specification. This figure describes the basic structure of the image annotation device, wherein the functional units and modules can be implemented in software, or a general-purpose chip or a specific chip can be used to implement image annotation. The device specifically includes:

[0155] A prediction score determination unit 1001 is configured to determine, based on an attention mechanism, a prediction score for a prediction category corresponding to a disease image in an external validation set, where the prediction category is a prediction result obtained by inputting the disease image in the external validation set into the imaging genomics prediction model;

[0156] A weight determination unit 1002 is configured to determine the weight of a feature map channel based on the prediction score and the feature layer in the imaging genomics prediction model;

[0157] a heat map determining unit 1003, configured to multiply the weight by the feature map to obtain a weighted feature map for each feature map channel and generate a two-dimensional heat map;

[0158] The annotation unit 1004 is configured to convert regions in the two-dimensional heat map where pixel scores exceed a preset ratio into image annotations, where the image annotations are image annotations related to the microenvironment.

[0159] Figure 11 The figure shows a schematic diagram of the structure of an image annotation device based on an imaging genomics prediction model according to an embodiment of this specification. This figure describes the basic structure of the image annotation device based on an imaging genomics prediction model. The functional units and modules therein can be implemented using software, or a general-purpose chip or a specific chip can be used to implement image annotation based on the imaging genomics prediction model. The device specifically includes:

[0160] An image acquisition unit 1101 is used to acquire a target disease image;

[0161] The annotated region determining unit 1102 is configured to input the target disease image into a genomics prediction system to obtain an annotated region of the disease image.

[0162] Figure 12 The figure shows a schematic diagram of a UNet module according to an embodiment of this specification. The UNet module consists of four downsampling modules (encoder modules), four upsampling modules (decoder modules), and a bottom connection layer. The bottom connection layer is used to link the upsampling modules and downsampling modules. Each module contains two convolutional layers.

[0163] Figure 13 The figure shows a schematic diagram of an SMLP module in an embodiment of this specification. The input layer of SMLP is M-dimensional data, and the output layer is N-dimensional data. SMLP compresses M-dimensional data into N-dimensional data, which is used to screen gene expressions related to breast cancer from a large amount of gene sequencing expression data.

[0164] Figure 14 Shown is a structural schematic diagram of an initial model of an embodiment of this specification.

[0165] The initial model includes a UNet module and a SMLP module (also known as an MLP module). The UNet module is used to extract image features (yellow in the figure) from disease sample image data, while the SMLP module is used to extract gene features related to the target disease type in gene sequencing samples (green in the figure).

[0166] Figure 15 The flowchart of an image annotation method based on an imaging genomics prediction model according to an embodiment of the present specification is shown, specifically including steps 1501 to 1502:

[0167] Step 1501: Obtain target disease images. In this step, the target disease images are unanalyzed images to be studied. The target disease images can come from an external validation set or other sources.

[0168] Step 1502 : Input the target disease image into an imaging genomics prediction system to obtain a labeled region of the disease image. The imaging genomics prediction system is composed of the imaging genomics prediction model and an attention mechanism.

[0169] In the embodiment of this specification, the imaging genomics prediction system is composed of an imaging genomics prediction model and Figure 8 The attention mechanism in [1] is composed of [2]. Without genomic data, only the target disease image to be studied is input into the system, and the target disease image microenvironment annotation, such as a clear tumor annotation area, can be obtained. Figure 16a and Figure 16b shown.

[0170] Figure 16a Shown is a schematic diagram of the image annotation results of an image-CAF model on an external validation set according to an embodiment of this disclosure.

[0171] Image annotation verification was performed on the ISPY1_1003 image from the external image validation set. Even without genomic data, image microenvironment annotation was performed, resulting in clear tumor annotated regions, including the tumor area and the surrounding microenvironment. This image annotation example shows that red areas are highly correlated with the microenvironment, while blue, yellow, and green areas are less correlated.

[0172] Figure 16b Figure 2 shows an image annotation result of another image-CAF model from this illustrative embodiment on an external validation set. Image annotation validation was performed on the UCSF-BR-03 image from the external validation set. The blue, yellow, and green areas in the figure represent regions with low microenvironment correlation, indicating that these regions have a low correlation with and influence on the tumor microenvironment.

[0173] like Figure 17 , which is a schematic diagram of a computer device provided in an embodiment of this specification. The oil and gas well engineering cost estimation method described in this application can be applied to the computer device. The computer device 1702 may include one or more processors 1704, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 1702 may also include any memory 1706 for storing any type of information, such as code, settings, data, etc. For example, without limitation, the memory 1706 may include any one or more combinations of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any memory may use any technology to store information. Furthermore, any memory may provide volatile or non-volatile retention of information. Furthermore, any memory may represent a fixed or removable component of the computer device 1702. In one embodiment, when the processor 1704 executes associated instructions stored in any memory or combination of memories, the computer device 1702 may perform any operation of the associated instructions. The computer device 1702 also includes one or more drive mechanisms 1708 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like.

[0174] The computer device 1702 may also include an input / output module 1710 (I / O) for receiving various inputs (via input devices 1712) and for providing various outputs (via output devices 1714). A specific output mechanism may include a presentation device 1716 and an associated graphical user interface (GUI) 1718. In other embodiments, the input / output module 1710 (I / O), input devices 1712, and output devices 1714 may not be included, and the computer device 1702 may simply be a computer device in a network. The computer device 1702 may also include one or more network interfaces 1720 for exchanging data with other devices via one or more communication links 1722. One or more communication buses 1724 couple the components described above together.

[0175] The communication link 1722 may be implemented in any manner, for example, via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0176] Corresponding to Figures 2 to 7 and Figure 15 The method in this specification also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are executed.

[0177] The embodiment of this specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 2 to 7 and Figure 15 The method shown.

[0178] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0179] It should also be understood that in the embodiments of this specification, the term "and / or" is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " in this specification generally indicates that the associated objects are in an "or" relationship.

[0180] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0181] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0182] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be an electrical, mechanical or other form of connection.

[0183] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0184] In addition, the functional units in the various embodiments of this specification may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0185] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0186] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, based on the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this specification.

Claims

1. A method for constructing an imaging genomics prediction model, characterized in that: The method comprises: Collect batch gene sequencing sample data and disease sample image data with paired relationships respectively; Determining, based on the proportion of each first cell type in the batch gene sequencing sample data, a first proportion group and a second proportion group corresponding to the proportion of each second cell type in the batch gene sequencing sample data, wherein the first cell type represents a cell type associated with a disease, and the second cell type represents a cell type in the first cell type that meets a preset difference; The first proportion group and the second proportion group corresponding to the proportion of each second cell type are used as labels, and combined with the disease sample image data and batch gene sequencing sample data, an imaging genomics prediction model corresponding to each cell type is iteratively constructed.

2. The method according to claim 1, characterized in that The batch gene sequencing sample data and disease sample image data are collected from a variety of different disease types. The second cell type is determined by: Using a t-test and a Wilcox test, respectively, to determine the proportion of each first cell type in target disease types and non-target disease types in multiple different disease types, and to determine a first probability value corresponding to the t-test and a second probability value corresponding to the Wilcox test; The cell type for which both the first probability value and the second probability value are lower than a preset threshold is determined to be a cell type that meets the preset difference.

3. The method according to claim 1, characterized in that The first cell type is determined by: Determining the proportion of multiple cell types and the expression of each cell type in the batch gene sequencing data based on a deconvolution algorithm; A first cell type that meets preset requirements is screened from multiple cell types.

4. The method according to claim 1, wherein Iteratively constructing the imaging genomics prediction model corresponding to each cell type further includes: Inputting the disease sample image data and batch gene sequencing sample data into the initial model to obtain an initial prediction result; Classify the initial prediction results according to a preset threshold to obtain a category corresponding to each initial prediction result; Calculate the loss value of the first proportion group or the second proportion group corresponding to the category between the initial prediction result after classification, iteratively update the initial model according to the loss value until the preset conditions are met, and construct a radiogenomics prediction model.

5. The method according to claim 4, characterized in that The initial prediction result is determined by: Extracting image features from the disease sample image data, and extracting expression features of target disease type cells from the batch gene sequencing sample data; fusing the image features and the expression features of the target disease type cells to obtain a combined feature; The combined features are input into a multi-layer perceptron to obtain an initial prediction result.

6. The method according to claim 5, characterized in that The first ratio group and the second ratio group are determined as follows: Sort the proportion of each second cell type according to a preset order to obtain a cell type proportion data set; The cell type proportion data set is divided according to the median, and the group whose cell type proportion exceeds the median is regarded as the first proportion group, and the group whose cell type proportion does not exceed the median is regarded as the second proportion group.

7. An image annotation method, characterized in that: The method applies the imaging genomics prediction model construction method according to any one of claims 1 to 6, comprising: Determine, based on an attention mechanism, a prediction score for a prediction category corresponding to a disease image in an external validation set, where the prediction category is a prediction result obtained by inputting the disease image in the external validation set into the imaging genomics prediction model; Determining weights of feature map channels based on the prediction scores and the feature layers in the imaging genomics prediction model; Multiplying the weight by the feature map to obtain a weighted feature map for each feature map channel and generating a two-dimensional heat map; The areas in the two-dimensional heat map where the pixel score values ​​exceed a preset ratio are converted into image annotations, where the image annotations are image annotations related to the microenvironment.

8. An image annotation method based on an imaging genomics prediction model, characterized in that: The method is applied to the method according to any one of claims 1 to 7, comprising: Acquire target disease images; The target disease image is input into the imaging genomics prediction system to obtain the annotated area of ​​the disease image. The imaging genomics prediction system is composed of the imaging genomics prediction model and the attention mechanism in claim 7.

9. A device for constructing an imaging genomics prediction model, characterized in that: The device comprises: A data acquisition unit, used to respectively acquire batch gene sequencing sample data and disease sample image data having a paired relationship; a determining unit, configured to determine, based on a proportion of each first cell type in the batch gene sequencing sample data, a first proportion group and a second proportion group corresponding to each second cell type in the batch gene sequencing sample data, wherein the first cell type represents a cell type associated with a disease, and the second cell type represents a cell type within the first cell type that meets a preset difference; The model construction unit is used to use the first proportion group and the second proportion group corresponding to each second cell type as labels, combine the disease sample image data and the batch gene sequencing sample data, and iteratively construct an imaging genomics prediction model corresponding to each cell type.

10. An image annotation device, characterized in that: The device comprises: a prediction score determination unit, configured to determine, based on an attention mechanism, a prediction score for a prediction category corresponding to a disease image in an external validation set, wherein the prediction category is a prediction result obtained by inputting the disease image in the external validation set into an imaging genomics prediction model; a weight determination unit, configured to determine the weight of a feature map channel based on the prediction score and the feature layer in the imaging genomics prediction model according to the method of any one of claims 1 to 8; a heat map determining unit, configured to multiply the weight by the feature map to obtain a weighted feature map for each feature map channel and generate a two-dimensional heat map; The annotation unit is used to convert the area in the two-dimensional heat map where the pixel score value exceeds a preset ratio into an image annotation, wherein the image annotation is an image annotation related to the microenvironment.

11. An image annotation device based on an imaging genomics prediction model, characterized in that: The device comprises: An image acquisition unit, configured to acquire a target disease image; The labeled area determination unit is used to input the target disease image into the genomics prediction system to obtain the labeled area of ​​the disease image, and the imaging genomics prediction system is composed of the imaging genomics prediction model and the attention mechanism in claim 7.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images