Gene expression level prediction method and device based on deep learning, and electronic equipment

Abstract

The disclosure provides a gene expression level prediction method based on deep learning, which can be applied to the fields of bioinformatics technology and artificial intelligence technology. The method comprises: acquiring a lung cancer tissue pathological section staining image of a gene expression level to be detected; preprocessing the lung cancer tissue pathological section staining image to obtain a target prediction image; inputting the target prediction image into a lung cancer subtype classification model to output a lung cancer subtype representation vector matched with the target prediction image, wherein the lung cancer subtype classification model is trained according to a first lung cancer tissue pathological section staining image training sample with lung cancer subtype information; inputting the lung cancer subtype representation vector and the target prediction image into a gene expression level prediction model to output a gene expression level corresponding to a target gene, wherein the gene expression level prediction model is trained according to a second lung cancer tissue pathological section staining image training sample with a gene expression level.

Description

Technical Field

[0001] This disclosure relates to the fields of bioinformatics and artificial intelligence, and more specifically, to a method, apparatus, and electronic device for predicting gene expression levels based on deep learning. Background Technology

[0002] Lung cancer is a malignant tumor with high incidence and mortality rates. Clinical data shows that PD-1 / PD-L1 immune checkpoint inhibitor therapy can significantly improve the overall survival of lung cancer patients; however, most patients develop innate or acquired resistance during treatment. RNA expression profiling analysis has revealed that the innate or acquired resistance to PD-1 / PD-L1 inhibitors in lung cancer patients is regulated by multiple genes. Understanding the expression of these resistance genes could predict patient responses to PD-1 / PD-L1 inhibitor therapy and lead to the development of appropriate combination therapy regimens, potentially greatly improving treatment outcomes for lung cancer patients.

[0003] In realizing the present invention, the inventors discovered at least the following problems in the related technologies: the detection of gene expression levels takes a long time and is costly. Summary of the Invention

[0004] In view of this, this disclosure provides a method, apparatus and electronic device for predicting gene expression levels based on deep learning.

[0005] One aspect of this disclosure provides a deep learning-based method for predicting gene expression levels, comprising:

[0006] Obtain stained images of lung cancer tissue sections with the expression level of the gene to be detected.

[0007] Preprocessing of stained images of lung cancer tissue pathological sections yields target prediction images;

[0008] The target prediction image is input into the lung cancer subtype classification model, and the output is a lung cancer subtype representation vector that matches the target prediction image. The lung cancer subtype classification model is trained based on training samples of the first lung cancer tissue pathological section staining image with lung cancer subtype information.

[0009] The lung cancer subtype characterization vector and the target prediction image are input into the gene expression level prediction model, which outputs the gene expression level corresponding to the target gene. The gene expression level prediction model is trained on training samples of stained images of second lung cancer tissue pathological sections with gene expression levels. The gene expression level of the second lung cancer tissue pathological section stained image training samples is determined based on the target gene, which is related to resistance to PD-1 / PD-L1 inhibitors.

[0010] According to embodiments of this disclosure, the lung cancer subtype characterization vector and the target prediction image are input into a gene expression level prediction model, and the gene expression level corresponding to the target gene is output, including:

[0011] The lung cancer subtype characterization vector and the target prediction image are input into the data fusion layer of the gene expression level prediction model, and the initial image fusion data is output.

[0012] The initial image fusion data is input into the fusion feature extraction layer of the gene expression level prediction model, and the fusion feature vector is output.

[0013] The fusion features are input into the fusion feature enhancement layer of the gene expression level prediction model, and the enhanced feature vector is output.

[0014] The enhanced feature vector is input into the gene expression level prediction layer of the gene expression level prediction model, and the output is the gene expression level corresponding to the target gene.

[0015] According to embodiments of this disclosure, inputting the target predicted image into a lung cancer subtype classification model and outputting a lung cancer subtype representation vector matching the target predicted image includes:

[0016] The target prediction image is input into the input layer of the lung cancer subtype classification model, and the output is lung cancer subtype processing data;

[0017] The lung cancer subtype processing data is input into the convolutional layer of the lung cancer subtype classification model, and the lung cancer subtype feature data of the target prediction image is output.

[0018] The input is fed into the output layer of the lung cancer subtype classification model, which outputs a lung cancer subtype representation vector that matches the target predicted image.

[0019] According to embodiments of this disclosure, the fusion feature vector is input into the fusion feature enhancement layer of the gene expression level prediction model, and the enhanced feature vector is output, including:

[0020] The attention module in the fusion feature enhancement layer is used to determine the fusion feature weights of the fusion feature vector;

[0021] Based on the fusion feature weights, an enhanced feature vector is output.

[0022] According to embodiments of this disclosure, the enhanced feature vector is input into the gene expression level prediction layer of the gene expression level prediction model, and the gene expression level corresponding to the target gene is output, including:

[0023] The enhanced feature vector is input into a fully connected neural network structure in the gene expression level prediction layer, and the mapping relationship between the enhanced feature vector and the target gene is output.

[0024] Based on the mapping relationship, the gene expression level corresponding to the target gene is output.

[0025] According to embodiments of this disclosure, preprocessing the stained image of a lung cancer tissue pathological section to obtain a target prediction image includes:

[0026] The target region image is determined based on the pixels in the stained image of lung cancer tissue pathological sections;

[0027] Based on the pixels in the target area image, the target area image is adjusted to obtain a target size image that conforms to the preset size;

[0028] The pixels in the target-sized image are normalized to obtain the target prediction image.

[0029] According to embodiments of this disclosure, a method for training a gene expression level prediction model includes:

[0030] Obtain a first training sample, wherein the first training sample includes a second lung cancer tissue pathological section staining image training sample and a third lung cancer tissue pathological section staining image training sample. The second lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image with gene expression level, and the third lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image without gene expression level.

[0031] The first deep learning model was trained using the second lung cancer slide stained image training sample as the label and the third lung cancer slide stained image training sample as the input, to obtain the gene expression level prediction model.

[0032] According to embodiments of this disclosure, the training method for the lung cancer subtype classification model includes:

[0033] Obtain a second training sample, wherein the second training sample includes a first lung cancer tissue pathological section staining image training sample and a fourth lung cancer tissue pathological section staining image training sample. The first lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image with lung cancer subtype information, and the fourth lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image without lung cancer subtype information.

[0034] The first lung cancer slide stained image training sample is used as the label, and the fourth lung cancer slide stained image training sample is used as the input to train the second deep learning model, thus obtaining the lung cancer subtype classification model.

[0035] Another aspect of this disclosure provides a deep learning-based gene expression level prediction device, comprising:

[0036] The acquisition module is used to acquire stained images of lung cancer tissue pathological sections with the gene expression level to be detected.

[0037] The module is used to preprocess the stained images of lung cancer tissue pathological sections to obtain the target prediction image;

[0038] The first output module is used to input the target prediction image into the lung cancer subtype classification model and output the lung cancer subtype representation vector that matches the target prediction image. The lung cancer subtype classification model is trained based on the training samples of the first lung cancer tissue pathological section staining image with lung cancer subtype information.

[0039] The second output module is used to input the lung cancer subtype characterization vector and the target prediction image into the gene expression level prediction model, and output the gene expression level corresponding to the target gene. The gene expression level prediction model is trained based on the second lung cancer tissue pathological section staining image training sample with gene expression level. The gene expression level of the second lung cancer tissue pathological section staining image training sample is determined based on the target gene, which is related to the drug resistance of PD-1 / PD-L1 inhibitors.

[0040] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the deep learning-based gene expression level prediction method.

[0041] According to embodiments of this disclosure, a target prediction image is obtained by preprocessing the stained image of lung cancer tissue pathological sections. A lung cancer subtype representation vector is then obtained through a lung cancer subtype classification model. The lung cancer subtype representation vector and the target prediction image are input into a gene expression level prediction model to obtain the gene expression level of the target gene. This method can predict the gene expression level of genes related to PD-1 / PD-L1 inhibitor resistance based solely on stained images of lung cancer tissue pathological sections. Compared to traditional methods of predicting gene expression levels using multi-protein section staining, the prediction method of this disclosure significantly improves diagnostic efficiency and reduces costs. Furthermore, by pre-classifying lung cancer types using a lung cancer subtype classification model, the prediction accuracy can be further increased. Therefore, it at least partially overcomes the technical problems of long detection times and high costs associated with gene expression level detection in related technologies. Attached Figure Description

[0042] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0043] Figure 1An exemplary system architecture for predicting gene expression levels based on deep learning, according to embodiments of the present disclosure, is illustrated.

[0044] Figure 2 A flowchart illustrating a method for training a gene expression level prediction model according to an embodiment of the present disclosure is shown schematically.

[0045] Figure 3 A flowchart illustrating a method for training a lung cancer subtype classification model according to an embodiment of the present disclosure is shown schematically.

[0046] Figure 4 A flowchart illustrating a deep learning-based gene expression level prediction method according to an embodiment of the present disclosure is shown.

[0047] Figure 5 A block diagram of a deep learning-based gene expression level prediction device according to an embodiment of the present disclosure is illustrated schematically.

[0048] Figure 6 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Detailed Implementation

[0049] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0050] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0051] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0052] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0053] In the technical solution disclosed herein, the acquisition, storage, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0054] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0055] In related technologies, gene expression levels in lung cancer tissues are usually detected based on RNA and protein levels. However, this method is time-consuming and costly, making it unsuitable for everyday use.

[0056] In view of this, embodiments of the present disclosure provide a deep learning-based method for predicting gene expression levels. The method includes acquiring a stained image of a lung cancer tissue pathological section containing the gene expression level to be detected; preprocessing the stained image to obtain a target prediction image; inputting the target prediction image into a lung cancer subtype classification model, and outputting a lung cancer subtype representation vector matching the target prediction image, wherein the lung cancer subtype classification model is trained using a first lung cancer tissue pathological section stained image training sample containing lung cancer subtype information; inputting the lung cancer subtype representation vector and the target prediction image into a gene expression level prediction model, and outputting the gene expression level corresponding to the target gene, wherein the gene expression level prediction model is trained using a second lung cancer tissue pathological section stained image training sample containing the gene expression level, the gene expression level of the second lung cancer tissue pathological section stained image training sample being determined based on the target gene, and the target gene being associated with resistance to PD-1 / PD-L1 inhibitors.

[0057] Figure 1 An exemplary system architecture 100, according to embodiments of the present disclosure, on which a deep learning-based gene expression level prediction method can be applied, is illustrated. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0058] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0059] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0060] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0061] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0062] It should be noted that the deep learning-based gene expression level prediction method provided in this embodiment can generally be executed by server 105. Correspondingly, the deep learning-based gene expression level prediction device provided in this embodiment can generally be located in server 105. The deep learning-based gene expression level prediction method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the deep learning-based gene expression level prediction device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Alternatively, the deep learning-based gene expression level prediction method provided in this embodiment can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103. Accordingly, the gene expression level prediction device based on deep learning provided in this embodiment can also be set in terminal device 101, 102, or 103, or in other terminal devices different from terminal device 101, 102, or 103.

[0063] For example, the stained image of a lung cancer tissue pathology slide can be originally stored in any one of terminal devices 101, 102, or 103 (e.g., terminal device 101, but not limited thereto), or stored on an external storage device and can be imported into terminal device 101. Then, terminal device 101 can locally execute the deep learning-based gene expression level prediction method provided in this disclosure embodiment, or send the stained image of the lung cancer tissue pathology slide to other terminal devices, servers, or server clusters, and have the other terminal devices, servers, or server clusters receiving the stained image of the lung cancer tissue pathology slide execute the deep learning-based gene expression level prediction method provided in this disclosure embodiment.

[0064] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0065] According to embodiments of this disclosure, since a lung cancer subtype classification model and a gene expression level prediction model are used in the gene expression level prediction method, the training of the two models will be explained first.

[0066] According to embodiments of this disclosure, before training the model, target genes associated with resistance to PD-1 / PD-L1 inhibitors are screened. Gene expression level analysis can be performed using published RNA-seq datasets from lung cancer cell lines or tumor tissues that are resistant or sensitive to PD-1 / PD-L1 inhibitors. Genes enriched in lung cancer cell lines or tumor tissues resistant to anti-PD-1 / PD-L1 inhibitor therapy are screened using Gene Set Enrichment Analysis (GSEA), and are considered to be associated with resistance to PD-1 / PD-L1 inhibitors. Conversely, genes enriched in lung cancer cell lines or tumor tissues sensitive to anti-PD-1 / PD-L1 inhibitor therapy are considered to be associated with the responsiveness of lung cancer cells to PD-1 / PD-L1 inhibitors. To ensure that the screened genes can predict the resistance or response of lung cancer patients to PD-1 / PD-L1 inhibitor therapy, genes that have been validated through gene function exploration experiments such as gene knockout or overexpression in published works are further screened. Ultimately, we can obtain resistance-related target genes including: PLSCR1, STAT1, CXCL10, USP18, OASL, IFI27, ISG15, IFI6, OAS3, OAS1, DDX60, HERC6, MX1, IFI44, IFI44L, IFIT3, IFIT1, PVR, COL4A5, COL7A1, COL18A1, COL14A1, COL4A2, COL4A1, COL5A1, and COL8A1. These target genes are upregulated in lung cancer cell lines or tumor tissues resistant to PD-1 / PD-L1 inhibitor blockade therapy, and their deficiency increases the sensitivity of cancer cells to treatment. Sensitivity-related target gene: CD274 (PD-L1), which is downregulated in lung cancer cell lines or tumor tissues resistant to PD-1 / PD-L1 inhibitor blockade therapy, and its deficiency increases the resistance of cancer cells to treatment. A total of 27 target genes were obtained.

[0067] Figure 2 A flowchart illustrating a method for training a gene expression level prediction model according to an embodiment of the present disclosure is shown.

[0068] like Figure 2 As shown, the method includes operations S201 to S202.

[0069] In operation S201, a first training sample is obtained, wherein the first training sample includes a second lung cancer tissue pathological section staining image training sample and a third lung cancer tissue pathological section staining image training sample. The second lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image with gene expression level, and the third lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image without gene expression level.

[0070] In operation S202, the training samples of the second lung cancer slide stained image are used as labels, and the training samples of the third lung cancer slide stained image are used as inputs to train the first deep learning model, thereby obtaining the gene expression level prediction model.

[0071] According to embodiments of this disclosure, the first training sample can be obtained from a database, wherein the second lung cancer tissue pathological section staining image training sample can be a lung cancer tissue pathological section staining image with labeled gene expression levels, and the third lung cancer tissue pathological section staining image training sample can be a lung cancer tissue pathological section staining image without labeled gene expression levels. Using the second lung cancer tissue section staining image training sample as a label and the third lung cancer tissue section staining image training sample as input, the predicted gene expression level can be output. The gene expression level in the training sample of the second lung cancer tissue pathological section staining image can be y, and the predicted gene expression level can be calculated by the following formula (1). Mean squared error (MSE) between the actual gene expression level (y) and the true gene expression level (y):

[0072]

[0073] Where K represents the sum of all the first training samples that participate in the calculation of the loss value of the current batch, and the possible values ​​are [1, 1000].

[0074] Based on the mean squared error, the trainable parameter values ​​in the gene expression level prediction model are updated through the backpropagation process of the gradient until the mean squared error converges to below the set threshold, or the total number of training rounds exceeds the preset range. In this way, the final trained gene expression level prediction model can be obtained.

[0075] Figure 3 A flowchart illustrating a training method for a lung cancer subtype classification model according to an embodiment of the present disclosure is shown.

[0076] like Figure 3 As shown, the method includes operations S301 to S302.

[0077] In operation S301, a second training sample is obtained, wherein the second training sample includes a first lung cancer tissue pathological section staining image training sample and a fourth lung cancer tissue pathological section staining image training sample. The first lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image with lung cancer subtype information, and the fourth lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image without lung cancer subtype information.

[0078] In operation S302, the first lung cancer slide stained image training sample is used as the label, and the fourth lung cancer slide stained image training sample is used as the input to train the second deep learning model, thereby obtaining the lung cancer subtype classification model.

[0079] According to embodiments of this disclosure, the second training sample can be obtained from a database. The first lung cancer tissue pathology slide staining image training sample can be a lung cancer tissue pathology slide staining image that has been labeled with lung cancer subtype information. The fourth lung cancer tissue pathology slide staining image training sample can be a lung cancer tissue pathology slide staining image that has not been labeled with lung cancer subtype information. The first lung cancer slide staining image training sample is used as a label, and the fourth lung cancer slide staining image training sample is used as input to train the lung cancer subtype classification model.

[0080] According to the embodiments of this disclosure, since the lung cancer subtype classification model requires sufficient second training samples for training, and there are many types of lung cancer subtypes, it is difficult to obtain sufficient training samples for the rarer subtypes. Therefore, this article takes the most common type of lung cancer - non-small cell lung cancer, and the two most important subtypes of non-small cell lung cancer - adenocarcinoma (LUAD) and squamous cell carcinoma (LUSC) as examples to introduce specific implementation methods and prediction results.

[0081] According to embodiments of this disclosure, the lung cancer subtype classification model can be constructed based on the ResNet50 neural network classification model as a second deep learning model, or it can be constructed based on other neural network models. Here, only the ResNet50 neural network classification model is used for illustration. The ResNet50 neural network classification model includes an input layer with a large-size convolutional kernel, typically 7×7. It is followed by four feature extraction convolutional layers, referred to here as convolutional layers 1 to 4. Each convolutional layer is composed of a varying number of residual blocks, each containing a 1×1, 3×3, or 1×1 convolutional combination. Finally, a fully connected output layer outputs the prediction results. The specific network structure of the ResNet50 neural network classification model is given in Table 1.

[0082] Table 1. Network structure of the ResNet50 neural network classification model

[0083]

[0084] In Table 1, 7×7,64 represents processing the second training sample of the current input using a convolution with a kernel size of 7×7 and 64 channels. The residual blocks in () represent a set, and ()×3 represents using 3 sets of such residual blocks to construct the current convolutional layer. FC represents the output layer used to map the input results to the final output lung cancer subtype representation vector.

[0085] According to embodiments of this disclosure, the difference between the output layer's predicted result and the actual result, i.e., the error value, can be calculated using a binary classification cross-entropy loss function. This error value is then used to backpropagate gradients and update the parameters of the second deep learning model using the Adam optimization algorithm. The binary classification cross-entropy loss function is defined by the following formula:

[0086]

[0087] in, denoted as the vector representing the lung cancer subtype predicted by the second deep learning model, and t as the corresponding true lung cancer subtype vector. The length of t is the same, denoted by n, and t[k] and Let represent the k-th element of the true lung cancer subtype vector and the predicted lung cancer subtype result vector, respectively. `log(·)` performs a base-10 logarithmic operation, `∑(·)` performs a summation operation, and `loss` represents the loss function. BCE (·) represents the final calculated loss value.

[0088] According to embodiments of this disclosure, the parameters of all convolutional parts in the ResNet50 used in the model are provided by a pre-trained model trained on the ImageNet ultra-large image classification dataset, and these are used as initial parameter values. For model parameters not present in other original pre-trained models, random Gaussianization is used for initialization. The model training process can be... Training was performed using the PyTorch deep learning framework based on Python, in an environment with an i7-6800K@3.4GHz CPU, 64GB of RAM, an NVIDIA TITAN RTX GPU, and Ubuntu 14.10.

[0089] The following section explains the gene expression level prediction method based on deep learning.

[0090] Figure 4 A flowchart illustrating a deep learning-based gene expression level prediction method according to an embodiment of the present disclosure is shown.

[0091] like Figure 4 As shown, the method includes operations S401 to S404.

[0092] Using the S401, obtain stained images of lung cancer tissue pathological sections to detect the gene expression level.

[0093] According to embodiments of this disclosure, gene expression level can be the transcriptional-translational level of a gene.

[0094] According to embodiments of this disclosure, the stained images of lung cancer tissue pathological sections can be obtained by taking photographs of lung cancer tissue pathological sections stained with hematoxylin and eosin (HE) using a microscope.

[0095] In operation S402, the stained images of lung cancer tissue pathological sections are preprocessed to obtain the target prediction image.

[0096] According to embodiments of this disclosure, in order to better predict gene expression levels in stained images of lung cancer tissue pathology sections, preprocessing of the stained images of lung cancer tissue pathology sections can include flipping, denoising, sharpening, etc., to obtain a target prediction image that can be input into a lung cancer subtype classification model.

[0097] In operation S403, the target prediction image is input into the lung cancer subtype classification model, and the lung cancer subtype representation vector that matches the target prediction image is output. The lung cancer subtype classification model is trained based on training samples of the first lung cancer tissue pathological section staining image with lung cancer subtype information.

[0098] According to embodiments of this disclosure, since the lung cancer subtype classification model has been trained, inputting the target prediction image into the lung cancer subtype classification model can output a lung cancer subtype representation vector that matches the target prediction image.

[0099] In operation S404, the lung cancer subtype characterization vector and the target prediction image are input into the gene expression level prediction model, and the gene expression level corresponding to the target gene is output. The gene expression level prediction model is trained based on the training sample of the second lung cancer tissue pathological section staining image with gene expression level. The gene expression level of the second lung cancer tissue pathological section staining image training sample is determined based on the target gene, which is related to the drug resistance of PD-1 / PD-L1 inhibitors.

[0100] According to embodiments of this disclosure, since the gene expression level prediction model is already trained to predict the gene expression level of the target gene, inputting the lung cancer subtype characterization vector and the target prediction image into the gene expression level prediction model can output the gene expression level corresponding to the target gene.

[0101] According to embodiments of this disclosure, a target prediction image is obtained by preprocessing the stained image of lung cancer tissue pathological sections. A lung cancer subtype representation vector is then obtained through a lung cancer subtype classification model. The lung cancer subtype representation vector and the target prediction image are input into a gene expression level prediction model to obtain the gene expression level of the target gene. This method can predict the gene expression level of genes related to PD-1 / PD-L1 inhibitor resistance based solely on stained images of lung cancer tissue pathological sections. Compared to traditional methods of predicting gene expression levels using multi-protein section staining, the prediction method of this disclosure significantly improves diagnostic efficiency and reduces costs. Furthermore, by pre-classifying lung cancer types using a lung cancer subtype classification model, the prediction accuracy can be further increased. Therefore, it at least partially overcomes the technical problems of long detection times and high costs associated with gene expression level detection in related technologies.

[0102] According to embodiments of this disclosure, the lung cancer subtype characterization vector and the target prediction image are input into a gene expression level prediction model, and the gene expression level corresponding to the target gene is output, including:

[0103] The lung cancer subtype characterization vector and the target prediction image are input into the data fusion layer of the gene expression level prediction model, and the initial image fusion data is output.

[0104] The initial image fusion data is input into the fusion feature extraction layer of the gene expression level prediction model, and the fusion feature vector is output.

[0105] The fusion features are input into the fusion feature enhancement layer of the gene expression level prediction model, and the enhanced feature vector is output.

[0106] The enhanced feature vector is input into the gene expression level prediction layer of the gene expression level prediction model, and the output is the gene expression level corresponding to the target gene.

[0107] According to embodiments of this disclosure, the main function of the data fusion layer is to receive two inputs of different dimensions: the target prediction image and its corresponding lung cancer subtype representation vector. Preliminary fusion of the input data is performed using a parameterized adaptive fusion method. Specifically, the lung cancer subtype representation vector is one-dimensional vector data, while the preprocessed target prediction image is three-channel vector data, where each element is a floating-point number within the range [0,1]. The lung cancer subtype representation vector t and the target prediction image I are fused as follows:

[0108]

[0109] in This represents the initial image fusion data after fusion, and conv(·,a) represents a two-dimensional convolution operation with a kernel size of a×a. This represents a vector expansion operation, which expands the one-dimensional lung cancer subtype representation vector in space to a vector with the same length and width as the target prediction image I. All missing parts are filled with the same values ​​repeatedly. This represents a vector concatenation operation, where data concatenation is performed at the channel level.

[0110] According to embodiments of this disclosure, all parameters in the data fusion layer are filled with random values ​​that follow a Gaussian distribution during model construction.

[0111] According to embodiments of this disclosure, the data fusion layer outputs initial image fusion data consisting of the target prediction image and the lung cancer subtype representation vector. However, the prediction of gene expression levels of the target gene relies on more efficient and complex feature representations. Therefore, a fusion feature extraction layer is constructed to extract feature information from the fused initial image fusion data that better matches the gene expression level prediction. The structure of the fusion feature extraction layer can refer to the structure in the lung cancer subtype classification model.

[0112] According to embodiments of this disclosure, since the structure of the fusion feature extraction layer is the same as that in the lung cancer subtype classification model (both are ResNet50 neural network classification models), and the features extracted by the ResNet50 neural network classification model are more suitable for the representation of classification tasks, directly applying them to gene expression level prediction is not the optimal approach. Therefore, by fusing a feature enhancement layer, a composite attention mechanism is used to strengthen the features from within, thereby improving the representational ability for gene expression level prediction and outputting an enhanced feature vector.

[0113] According to embodiments of this disclosure, the gene expression level prediction layer can calculate a gene expression level representation vector that matches the number of target genes, and finally output the gene expression level corresponding to the target genes.

[0114] According to embodiments of this disclosure, the target prediction image and the lung cancer subtype representation vector are fused and feature extracted through different network layers in the gene expression level prediction model, and finally the gene expression level corresponding to the target gene is output, thus realizing rapid and low-cost prediction of the gene expression level of the target gene.

[0115] According to embodiments of this disclosure, inputting the target predicted image into a lung cancer subtype classification model and outputting a lung cancer subtype representation vector matching the target predicted image includes:

[0116] The target prediction image is input into the input layer of the lung cancer subtype classification model, and the output is lung cancer subtype processing data;

[0117] The lung cancer subtype processing data is input into the convolutional layer of the lung cancer subtype classification model, and the lung cancer subtype feature data of the target prediction image is output.

[0118] The input is fed into the output layer of the lung cancer subtype classification model, which outputs a lung cancer subtype representation vector that matches the target predicted image.

[0119] According to embodiments of this disclosure, a lung cancer subtype classification model may include an input layer, a convolutional layer, and an output layer. The input layer performs preliminary processing on the target prediction image to obtain lung cancer subtype processing data. The convolutional layer may have three layers. After calculation by the convolutional layer, the lung cancer subtype features in the target prediction image can be extracted to obtain lung cancer subtype feature data. Finally, the output layer outputs the lung cancer subtype representation vector.

[0120] According to embodiments of this disclosure, a lung cancer subtype classification model is used to classify lung cancer subtypes in a target prediction image, thereby obtaining a lung cancer subtype representation vector that matches the target prediction image. This allows for more accurate prediction of gene expression levels in a gene expression level prediction model by utilizing the lung cancer subtype representation vector.

[0121] According to embodiments of this disclosure, the fusion feature vector is input into the fusion feature enhancement layer of the gene expression level prediction model, and the enhanced feature vector is output, including:

[0122] The attention module in the fusion feature enhancement layer is used to determine the fusion feature weights of the fusion feature vector;

[0123] Based on the fusion feature weights, an enhanced feature vector is output.

[0124] According to embodiments of this disclosure, the fusion feature enhancement layer achieves channel-level feature information flow through a spatial convolutional neural network, breaking the channel-to-channel blockage caused by direct splicing; then, a convolution-based block attention module generates fusion feature weights for the fusion feature vector and applies them to the fusion feature vector itself, further highlighting the effective regions in the fusion feature vector. The enhanced feature vector output by the fusion feature enhancement layer is represented by the enhanced feature vector. The enhancement process of the fusion feature enhancement layer on the fusion feature vector can be described by the following formula (4):

[0125]

[0126] Where i is the channel number of the corresponding feature. This is a vector slice of the i-th channel of the feature vector output by the feature extraction module. This represents a vector slice of the i-th channel of the feature representation enhanced by a spatial convolutional neural network. `conv(·,a)` represents a two-dimensional convolution operation with a kernel size of a×a, and `+` represents pixel-wise vector addition. Here, we use... Indicates all The feature vector obtained after stacking the channel dimensions is the feature representation of the complete spatial convolutional neural network after enhancement. Therefore, the attention module is calculated as follows:

[0127]

[0128]

[0129] in, The vector represents the feature vector generated during the operation, δ(·) is the Sigmoid function, MLP(·) and AP(·) represent multilayer perceptron operation and average pooling operation, respectively. conv(·,a) represents a two-dimensional convolution operation with a kernel size of a×a. This is the enhanced feature vector output by the fusion feature enhancement layer.

[0130] According to embodiments of this disclosure, the enhanced feature vector is input into the gene expression level prediction layer of the gene expression level prediction model, and the gene expression level corresponding to the target gene is output, including:

[0131] The enhanced feature vector is input into a fully connected neural network structure in the gene expression level prediction layer, and the mapping relationship between the enhanced feature vector and the target gene is output.

[0132] Based on the mapping relationship, the gene expression level corresponding to the target gene is output.

[0133] According to embodiments of this disclosure, the gene expression level prediction layer can be composed of three fully connected neural networks, with the number of neurons in each fully connected neural network decreasing progressively. The number of neurons in the input layer of the first fully connected layer is consistent with the total number of elements in the enhanced feature vector F3 output by the fusion feature enhancement layer. The number of neurons in the input layer of the middle layer is 1024, and the number of neurons in the output layer of the last layer is the same as the number of target genes, which can be 27, i.e., the number of genes contained in the target genes related to PD-1 / PD-L1 inhibitor resistance.

[0134] According to embodiments of this disclosure, based on the number of features output by the gene expression level prediction layer, a mapping relationship is established between the gene expression level representation of each target gene and the corresponding target gene, thereby enabling the output of the gene expression level corresponding to the target gene based on the mapping relationship.

[0135] According to embodiments of this disclosure, preprocessing the stained image of a lung cancer tissue pathological section to obtain a target prediction image includes:

[0136] The target region image is determined based on the pixels in the stained image of lung cancer tissue pathological sections;

[0137] Based on the pixels in the target area image, the target area image is adjusted to obtain a target size image that conforms to the preset size;

[0138] The pixels in the target-sized image are normalized to obtain the target prediction image.

[0139] According to embodiments of this disclosure, in order to ensure the efficient operation of the method and improve its robustness while reducing the interference of background noise information in complex scenes, the input lung cancer tissue pathological slide staining image needs to be preprocessed during the gene expression level prediction process. The preprocessing process includes three parts: extraction of effective regions of lung cancer tissue pathological slide staining image, adjustment of effective regions, and image normalization.

[0140] According to embodiments of this disclosure, lung cancer tissue pathological section staining image I ori It is a color image, therefore I ori It contains three channels: red, green, and blue. Here, we use I... r ,I g ,I b To represent, the following general formula is used to calculate and generate the grayscale image I. G :

[0141] I G =0.299×I r +0.587×I g +0.114×I b (7)

[0142] Among them, [I r ,I g ,I b ] = I ori This indicates that the data from the three channels can be obtained directly from stained images of lung cancer tissue pathological sections. The initially generated grayscale image will undergo further binarization to obtain a binarized image I. B =F B (I G ,T), where F B (·) represents a binary equation, and the specific algorithm for this equation is as follows:

[0143]

[0144] Where the threshold T can be 100, and j is I G Any element in.

[0145] According to embodiments of this disclosure, the binarized image I is calculated using the following connected component calculation method. B Obtain the target region image I U Connectivity calculation methods will affect the binarized image I.B Perform two consecutive line-by-line scans. During the first line-by-line scan, each pixel value in the image is scanned sequentially from top to bottom and left to right. Each valid pixel value with a value of 1 is assigned a label, with the label value being an integer value starting from zero and incrementing by 1. The label assignment rules are as follows:

[0146] If the left and top pixel values ​​of a pixel's 4-neighborhood are both 0 and have no label, then assign a new label value to the pixel. If either the left or top pixel value of a pixel's 4-neighborhood is 1, then the pixel's label is the same as the label of the adjacent pixel with a value of 1. If both the left and top pixel values ​​of a pixel's 4-neighborhood are 1 and have the same label, then this label is used to label the pixel. If both the left and top pixel values ​​of a pixel's 4-neighborhood are 1 and have different labels, then the smaller label is used as the pixel's label value. If, after this labeling, several different labels appear in a connected component, then the labels of the left and top pixels of that pixel are assigned the same label.

[0147] During the second line-by-line scan of the binarized image, labels with equal relationships are merged, and the smallest label is selected as the final label. This process ultimately obtains all connected components in the binarized image. The largest connected component in the binarized image is selected as the effective region in the stained image of the lung cancer tissue pathological section. The coordinate information of each pixel in the effective region is scanned and recorded, and the maximum and minimum values ​​of the horizontal and vertical coordinates are recorded as x. max ,y max ,x min ,y min Based on these coordinate values, the target region image I is extracted from the stained image of a lung cancer tissue pathological section. U =I ori [x min :x max ,y min :y max ].

[0148] According to embodiments of this disclosure, since the target region image size varies in each lung cancer tissue pathology slide staining image, to ensure effective prediction, the target region image size needs to be adjusted to a preset size W×H, typically 1024×1024. Bilinear interpolation is used to adjust the target region image. For a point in the target region image at position (x,y), its corresponding value f(x,y) is calculated from the four points Q in the lung cancer tissue pathology slide staining image that are proportionally scaled to the current position. 12 (x1, y2), Q 22 (x2,y2), Q 11(x1, y1), Q 21 The values ​​of (x2, y1) together determine the value of the x2, y1, and the specific calculation method is as follows:

[0149] f(x,y)=f(Q 11 )w 11 +f(Q 21 )w 21 +f(Q 12 )w 12 +f(Q 22 )w 22 (9)

[0150] Where f(·) represents the value of the image at a specified point or location, w 11 =(x²-x)(y²-y), w 21 =(x-x1)(y2-y), w 12 =(x2-x)(y-y1), w 22 = (x-x1)(y-y1). The final adjusted target size image is obtained using I. UR express.

[0151] According to embodiments of this disclosure, a target-size image I can be normalized using a statistical Z-score-based method. UR Normalization is performed. The target prediction image I obtained after normalization can be calculated using the following formula (10):

[0152]

[0153] Among them, a * denoted as the normalized pixel value in the target prediction image I, where μ, σ, and a represent the mean, standard deviation, and current position pixel value obtained statistically from the three-channel pixel values, respectively.

[0154] According to embodiments of this disclosure, by preprocessing the stained images of lung cancer tissue pathological sections, a target prediction image with uniform size and normalized pixel values ​​can be generated, enabling the lung cancer subtype classification model to perform lung cancer subtype classification more efficiently and the gene expression level prediction model to perform gene level prediction tasks more efficiently.

[0155] According to embodiments of this disclosure, in order to demonstrate the effectiveness of the prediction method in the embodiments of this disclosure, in The experiment and simulation were conducted using a Silver 4110@2.10GHz CPU, NVIDIA TITAN RTX, 128GB RAM, and Ubuntu 18.04.4LTS, built with Python and PyTorch frameworks. Data used in the experiment was collected and screened from the Genomic Data Commons Database, ultimately yielding 200 cases each of LUSC and LUAD lung cancer patient datasets. 160 cases were randomly selected from each lung cancer type as the training set, and the remaining 40 cases as the test set. Therefore, a total of 400 lung cancer patient datasets were used, including 320 training data and 80 test data. Each lung cancer patient dataset included HE-stained images of lung cancer tissue pathological sections, RNA-seq data, and information on the cancer type of the lung cancer tissue pathological sections. The cancer type of each lung cancer sample was converted into a corresponding numerical label for subsequent training and testing. The expression levels of PD-1 / PD-L1 inhibitor resistance-related genes were screened from the RNA-seq data and characterized by the FPKM value of each gene. The final test results and average results for some genes are shown in Table 2. It can be seen that after sufficient training, the predicted expression levels of PD-1 / PD-L1 inhibitor resistance-related genes showed a high degree of consistency with the actual data in both LUSC and LUAD lung cancer subtypes. The prediction method in this embodiment achieved the expected results, with an average prediction time of less than 300 milliseconds, which can provide the data required for auxiliary diagnosis and treatment at a very fast speed and with very high efficiency.

[0156] Table 2. Experimental results of the deep learning-based gene expression level prediction method

[0157]

[0158] Figure 5 A block diagram of a deep learning-based gene expression level prediction device according to an embodiment of the present disclosure is illustrated schematically.

[0159] like Figure 5 As shown, the gene expression level prediction device 500 based on deep learning includes an acquisition module 510, an acquisition module 520, a first output module 530, and a second output module 540.

[0160] The acquisition module 510 is used to acquire stained images of lung cancer tissue pathological sections with the gene expression level to be detected.

[0161] Module 520 is obtained for preprocessing the stained images of lung cancer tissue pathological sections to obtain the target prediction image;

[0162] The first output module 530 is used to input the target prediction image into the lung cancer subtype classification model and output the lung cancer subtype representation vector that matches the target prediction image. The lung cancer subtype classification model is trained based on the training samples of the first lung cancer tissue pathological section staining image with lung cancer subtype information.

[0163] The second output module 540 is used to input the lung cancer subtype characterization vector and the target prediction image into the gene expression level prediction model, and output the gene expression level corresponding to the target gene. The gene expression level prediction model is trained based on the training sample of the second lung cancer tissue pathological section staining image with gene expression level. The gene expression level of the second lung cancer tissue pathological section staining image training sample is determined based on the target gene, which is related to the drug resistance of PD-1 / PD-L1 inhibitors.

[0164] According to embodiments of this disclosure, a second output module for inputting a lung cancer subtype characterization vector and a target prediction image into a gene expression level prediction model, and outputting the gene expression level corresponding to the target gene, includes:

[0165] The first output unit is used to input the lung cancer subtype characterization vector and the target prediction image into the data fusion layer of the gene expression level prediction model, and output the initial image fusion data.

[0166] The second output unit is used to input the initial image fusion data into the fusion feature extraction layer of the gene expression level prediction model and output the fusion feature vector.

[0167] The third output unit is used to input the fusion features into the fusion feature enhancement layer of the gene expression level prediction model and output the enhanced feature vector.

[0168] The fourth output unit is used to input the enhanced feature vector into the gene expression level prediction layer of the gene expression level prediction model and output the gene expression level corresponding to the target gene.

[0169] According to embodiments of this disclosure, the first output module for inputting a target predicted image into a lung cancer subtype classification model and outputting a lung cancer subtype representation vector matching the target predicted image includes:

[0170] The fifth output unit is used to input the target prediction image into the input layer of the lung cancer subtype classification model and output lung cancer subtype processing data.

[0171] The sixth output unit is used to input lung cancer subtype processing data into the convolutional layer of the lung cancer subtype classification model and output lung cancer subtype feature data of the target prediction image.

[0172] The seventh output unit is used to input the lung cancer subtype classification model into the output layer and output a lung cancer subtype representation vector that matches the target predicted image.

[0173] According to embodiments of this disclosure, the third output unit for inputting the fused feature vector into the fused feature enhancement layer of the gene expression level prediction model and outputting the enhanced feature vector includes:

[0174] The first output subunit is used to determine the fusion feature weights of the fusion feature vector using the attention module in the fusion feature enhancement layer;

[0175] The second output subunit is used to output an enhanced feature vector based on the fused feature weights.

[0176] According to embodiments of this disclosure, the fourth output unit for inputting the enhanced feature vector into the gene expression level prediction layer of the gene expression level prediction model and outputting the gene expression level corresponding to the target gene includes:

[0177] The third output subunit is used to input the enhanced feature vector into the fully connected neural network structure in the gene expression level prediction layer and output the mapping relationship between the enhanced feature vector and the target gene.

[0178] The fourth output subunit is used to output the gene expression level corresponding to the target gene based on the mapping relationship.

[0179] According to embodiments of this disclosure, the module for preprocessing stained images of lung cancer tissue pathological sections to obtain a target prediction image includes:

[0180] The first obtaining unit is used to determine the target region image based on the pixels in the stained image of lung cancer tissue pathological sections;

[0181] The second obtaining unit is used to adjust the target region image according to the pixels in the target region image to obtain a target size image that conforms to the preset size;

[0182] The third unit is used to normalize the pixels in the target-sized image to obtain the target prediction image.

[0183] According to embodiments of this disclosure, the training module of the gene expression level prediction model includes:

[0184] The first training unit is used to acquire the first training sample, wherein the first training sample includes a second lung cancer tissue pathological section staining image training sample and a third lung cancer tissue pathological section staining image training sample. The second lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image with gene expression level, and the third lung cancer tissue pathological section staining image training sample represents a lung cancer tissue pathological section staining image without gene expression level.

[0185] The second training unit is used to train the first deep learning model by using the second lung cancer slide stained image training sample as a label and the third lung cancer slide stained image training sample as input, so as to obtain a gene expression level prediction model.

[0186] According to embodiments of this disclosure, the training module of the lung cancer subtype classification model includes:

[0187] The third training unit is used to acquire the second training samples, wherein the second training samples include the first lung cancer tissue pathological section staining image training sample and the fourth lung cancer tissue pathological section staining image training sample. The first lung cancer tissue pathological section staining image training sample represents the lung cancer tissue pathological section staining image with lung cancer subtype information, and the fourth lung cancer tissue pathological section staining image training sample represents the lung cancer tissue pathological section staining image without lung cancer subtype information.

[0188] The fourth training unit is used to train the second deep learning model by using the first lung cancer slide stained image training sample as a label and the fourth lung cancer slide stained image training sample as input, thereby obtaining a lung cancer subtype classification model.

[0189] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0190] For example, any plurality of the acquisition module 510, the obtaining module 520, the first output module 530, and the second output module 540 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present disclosure, at least one of the acquisition module 510, the obtaining module 520, the first output module 530, and the second output module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 510, the obtaining module 520, the first output module 530, and the second output module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0191] It should be noted that the deep learning-based gene expression level prediction device part in the embodiments of this disclosure corresponds to the deep learning-based gene expression level prediction method part in the embodiments of this disclosure. For a detailed description of the deep learning-based gene expression level prediction device part, please refer to the deep learning-based gene expression level prediction method part, which will not be repeated here.

[0192] Figure 6 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0193] like Figure 6As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 308 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0194] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0195] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The system 600 may also include one or more of the following components connected to the I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0196] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0198] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A deep learning-based gene expression level prediction method, comprising: obtaining a lung cancer tissue pathological section staining image to be detected for gene expression level; preprocessing the lung cancer tissue pathological section staining image to obtain a target prediction image; inputting the target prediction image into a lung cancer subtype classification model to output a lung cancer subtype representation vector matched with the target prediction image, wherein the lung cancer subtype classification model is trained according to a first lung cancer tissue pathological section staining image training sample with lung cancer subtype information; inputting the lung cancer subtype representation vector and the target prediction image into a gene expression level prediction model to output a gene expression level corresponding to a target gene, wherein the gene expression level prediction model is trained according to a second lung cancer tissue pathological section staining image training sample with gene expression level, the gene expression level of the second lung cancer tissue pathological section staining image training sample is determined according to the target gene, and the target gene is related to the drug resistance of a PD-1 / PD-L1 inhibitor.

2. The method of claim 1, wherein, The inputting of the lung cancer subtype representation vector and the target prediction image into the gene expression level prediction model to output the gene expression level corresponding to the target gene comprises: inputting the lung cancer subtype representation vector and the target prediction image into a data fusion layer of the gene expression level prediction model to output initial image fusion data; inputting the initial image fusion data into a fusion feature extraction layer of the gene expression level prediction model to output a fusion feature vector; inputting the fusion feature into a fusion feature enhancement layer of the gene expression level prediction model to output an enhanced feature vector; inputting the enhanced feature vector into a gene expression level prediction layer of the gene expression level prediction model to output the gene expression level corresponding to the target gene.

3. The method of claim 1, wherein, The inputting of the target prediction image into the lung cancer subtype classification model to output the lung cancer subtype representation vector matched with the target prediction image comprises: inputting the target prediction image into an input layer of the lung cancer subtype classification model to output lung cancer subtype processing data; inputting the lung cancer subtype processing data into a convolution layer of the lung cancer subtype classification model to output lung cancer subtype feature data of the target prediction image; inputting into an output layer of the lung cancer subtype classification model to output the lung cancer subtype representation vector matched with the target prediction image.

4. The method of claim 2, wherein, The inputting of the fusion feature vector into the fusion feature enhancement layer of the gene expression level prediction model to output the enhanced feature vector comprises: determining a fusion feature weight of the fusion feature vector by using an attention module in the fusion feature enhancement layer; outputting the enhanced feature vector based on the fusion feature weight.

5. The method of claim 2, wherein, The inputting of the enhanced feature vector into the gene expression level prediction layer of the gene expression level prediction model to output the gene expression level corresponding to the target gene comprises: inputting the enhanced feature vector into a full-connection neural network structure in the gene expression level prediction layer, and outputting a mapping relationship between the enhanced feature vector and the target gene; outputting the gene expression level corresponding to the target gene based on the mapping relationship.

6. The method of claim 1, wherein, The preprocessing of the lung cancer histopathological section staining image to obtain a target prediction image comprises: determining a target region image according to the pixel points in the lung cancer histopathological section staining image; adjusting the target region image according to the pixel points in the target region image to obtain a target size image conforming to a preset size; normalizing the pixel points in the target size image to obtain the target prediction image.

7. The method of claim 1, wherein the training method of the gene expression level prediction model comprises: obtaining a first training sample, wherein the first training sample comprises a second lung cancer histopathological section staining image training sample and a third lung cancer histopathological section staining image training sample, the second lung cancer histopathological section staining image training sample representing a lung cancer histopathological section staining image with a gene expression level, and the third lung cancer histopathological section staining image training sample representing a lung cancer histopathological section staining image without a gene expression level; training a first deep learning model by taking the second lung cancer histopathological section staining image training sample as a label and the third lung cancer histopathological section staining image training sample as input, to obtain the gene expression level prediction model.

8. The method of claim 1, wherein, The training method of the lung cancer subtype classification model comprises: obtaining a second training sample, wherein the second training sample comprises a first lung cancer histopathological section staining image training sample and a fourth lung cancer histopathological section staining image training sample, the first lung cancer histopathological section staining image training sample representing a lung cancer histopathological section staining image with lung cancer subtype information, and the fourth lung cancer histopathological section staining image training sample representing a lung cancer histopathological section staining image without lung cancer subtype information; training a second deep learning model by taking the first lung cancer histopathological section staining image training sample as a label and the fourth lung cancer histopathological section staining image training sample as input, to obtain the lung cancer subtype classification model.

9. A gene expression level prediction device based on deep learning, comprising: an acquisition module configured to acquire a lung cancer histopathological section staining image with a gene expression level to be detected; a obtaining module configured to preprocess the lung cancer histopathological section staining image to obtain a target prediction image; a first output module configured to input the target prediction image into a lung cancer subtype classification model, and output a lung cancer subtype representation vector matched with the target prediction image, wherein the lung cancer subtype classification model is trained according to a first lung cancer histopathological section staining image training sample with lung cancer subtype information. The second output module is configured to input the lung cancer subtype characterization vector and the target prediction image into a gene expression level prediction model to output a gene expression level corresponding to a target gene, wherein the gene expression level prediction model is trained according to a second lung cancer histopathological section staining image training sample with a gene expression level, the gene expression level of the second lung cancer histopathological section staining image training sample is determined according to the target gene, and the target gene is related to the drug resistance of the PD-1 / PD-L1 inhibitor.

10. An electronic device, comprising: one or more processors; memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-8.

Images