Medical information processing apparatus and medical information processing method

Abstract

To provide a medical information processing apparatus and method for imposing an ordering to observe the proximity relationship in both low-level features and high-level features.SOLUTION: In a medical information processing apparatus 10, a processing circuit 22 of a computing device 12 acquires convolutional neural network (CNN) layers, and trains a first mapping layer that connects to an input layer of the CNN layers, and a second mapping layer that connects to an output layer of the CNN layers. The first mapping layer converts input omics data representing gene information into first N-dimensional data, and the second mapping layer receives second N-dimensional data that is output by the CNN layers and converts the second N-dimensional data into output omics data representing gene information. The training of the first mapping layer and the second mapping layer includes minimizing a loss function in predetermined parameters of the CNN layers. The loss function is obtained on the basis of the input omics data and the output omics data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to a medical information processing apparatus and a medical information processing method. [Background technology]

[0002] Omics research refers to fields of science that aim to characterize biomolecules to understand the structure and function of living organisms. For example, transcriptomics is the analysis of gene expression data to determine the number and diversity of RNA transcripts found in a biological sample.

[0003] Gene expression data in the transcriptome may be thought of as including a normalized count of mRNA molecules found in a tissue sample (e.g., a colon polyp or a biopsy tumor). Using the known relationships between DNA, RNA, and protein, the transcriptome may be thought of as forming a proxy for protein expression in the tissue.

[0004] Transcriptomes can be obtained through RNA sequencing methods, including microarrays, RNA sequences, or probe-based analyses using libraries of targeted primers, such as Temp'O Seq.

[0005] A typical gene expression dataset from a sample will contain about 10,000 to 60,000 measurements. If only protein-coding RNA (mRNA) is considered and low-expressing genes are filtered out, the dataset can be reduced to about 10,000 measurements. Inspection of the dataset shows that there is no explicit ordering of the transcriptomic results; by default, genes are often listed alphabetically by name. Ordering based on chromosomal location is also possible, but is less relevant for functional significance.

[0006] Figure 1 shows an example of an individual transcriptome stored in a spreadsheet. Each gene is given an individual index number. Each gene has an associated normalized gene expression value. In practice, the transcriptomes of many patients or other subjects may be stored in a single spreadsheet.

[0007] In particular, the transcriptome of tumor biopsies has been widely studied and shown to have prognostic value, for example, the transcriptome of resected polyps may be analyzed to improve stratification in colon cancer screening.

[0008] Transcriptomic datasets, consisting of gene expression values ​​of many samples, are typically processed by different gene expression analyses for each cohort. This is often followed by gene group enrichment analysis and visualization in a Volcano Plot or heat map. Volcano plots contain multiple points, each representing an individual gene, plotted as fold change versus p-value for the whole cohort, not as patient-specific plots.

[0009] These methods will often be useful for understanding perturbations in individual genes and groups of genes, but they typically do not provide an easily interpretable visual display of the entire dataset across all samples.

[0010] Both the human visual cortex and modern neural network models contain convolutional network architectures and are adept at spotting patterns in 2D images for pattern classification. Images can be meaningfully compared if they are defined or structured in some absolute way. Therefore, it is necessary to present omics data in a way that allows humans and modern neural network models to make sense of them.

[0011] Convolutional Neural Networks (CNNs) can be considered the state of the art in Deep Learning (DL) techniques. Their success can be attributed to the reduction of free parameters by processing data locally instead of globally. This reduces the capacity for over-fitting and for fitting into local minima with respect to other DL methods, such as dense neural networks.

[0012] We have found that for CNNs to be useful, they need to be applied to input data that has a meaningful ordering. A meaningful ordering may be one in which highly related data are placed close to each other in a data sequence and less related data are placed farther apart. A meaningful ordering may be meaningful at both low and high levels. In the case of processing the input data into the form of 2D or 3D images, the low level ordering may include texture and edges. The high level ordering may involve a hierarchical grouping of low level features. The high level features may be of a more abstract nature.

[0013] The concept of low and high level features can be clarified by considering an example image, e.g. a photo of a car. An example of a low level feature would be the texture of the road the car is driving on or the edges of the car. An example of a mid level feature would be the car itself or further cars in the image. An example of a high level feature would be the risk associated with the car.

[0014] A CNN may typically decompose an image into both high-level and low-level features, for example, different layers of the CNN may be associated with different levels of features. [Prior art documents] [Patent documents]

[0015] [Patent Document 1] China Patent Application Publication No. 102184349 Summary of the Invention [Problem to be solved by the invention]

[0016] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to impose an order for observing proximity relationships in both low-level features and high-level features. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0017] A medical information processing device according to an embodiment includes a processing circuit, which acquires a Convolutional Neural Network (CNN) layer, trains a first mapping layer coupled to an input layer of the CNN layer and a second mapping layer coupled to an output layer of the CNN layer, the first mapping layer converts input omics data representing genetic information into first N-dimensional data, the second mapping layer receives second N-dimensional data output by the CNN layer and converts the second N-dimensional data into output omics data representing genetic information, and the training of the first mapping layer and the second mapping layer includes minimizing a loss function at a predetermined parameter of the CNN layer, and the loss function is obtained based on the input omics data and the output omics data. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 shows an extract of the transcriptome in spreadsheet form. [Diagram 2] FIG. 2 is a schematic diagram of an apparatus according to an embodiment. [Diagram 3] FIG. 3 is a flow chart outlining a method according to an embodiment. [Figure 4]FIG. 4 is an exemplary diagram illustrating points output by the manifold learning method. [Diagram 5] FIG. 5 is an exemplary diagram showing the points after alignment. [Figure 6] FIG. 6 is an example of a genetic map. [Figure 7] FIG. 7 shows an example of displayed images of six subjects. [Figure 8] FIG. 8 is an example of a display image with a mouse-over function. [Figure 9] FIG. 9 is a flow chart outlining a method according to an embodiment. [Figure 10] FIG. 10 is a flow chart outlining a method according to an embodiment. [Figure 11] FIG. 11 shows a set of images processed by the autoencoder. [Figure 12] FIG. 12 shows an example of the output format. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] An embodiment provides a medical information processing device including a processing circuit. The processing circuit acquires a Convolutional Neural Network (CNN) layer, and trains a first mapping layer coupled to an input layer of the CNN layer and a second mapping layer coupled to an output layer of the CNN layer. The first mapping layer converts input omics data representing genetic information into first N-dimensional data, and the second mapping layer receives second N-dimensional data output by the CNN layer and converts the second N-dimensional data into output omics data representing genetic information. The training of the first mapping layer and the second mapping layer includes minimizing a loss function for a predetermined parameter of the CNN layer. The loss function is obtained based on the input omics data and the output omics data.

[0020] An embodiment provides a medical information processing method including: obtaining a Convolutional Neural Network (CNN) layer; and training a first mapping layer coupled to an input layer of the CNN layer and a second mapping layer coupled to an output layer of the CNN layer. The first mapping layer converts input omics data representing genetic information into first N-dimensional data, and the second mapping layer receives second N-dimensional data output by the CNN layer and maps the second N-dimensional data to output omics data representing genetic information. The training of the first mapping layer and the second mapping layer includes minimizing a loss function at a predetermined parameter of the CNN layer. The loss function is obtained based on the input omics data and the output omics data.

[0021] An embodiment provides a medical information processing device including a processing circuit, which obtains a trained first mapping layer that converts input omics data representing genetic information into a first N-dimensional data, converts the input omics dataset into a spatially ordered dataset using the trained first mapping layer, and applies a task-specific model to the spatially ordered dataset to obtain a task-specific output.

[0022] One embodiment provides a medical information processing method including: obtaining a trained first mapping layer that maps omics input data to N-dimensional data; converting an input omics dataset representing genetic information into a spatially ordered dataset using the trained first mapping layer; and applying a task-specific model to the spatially ordered dataset to obtain a task-specific output.

[0023] An apparatus 10 according to an embodiment is shown diagrammatically in Fig. 2. The apparatus 10 may also be referred to as a medical information processing apparatus. The apparatus 10 processes omics data. The apparatus 10 further displays an image based on the omics data.

[0024] The omics data may be any type of omics data including a plurality of values, each of which is associated with a distinct one of the biomolecules. The data may be, for example, transcriptome data, where the biomolecules are genes and the associated values ​​are expression levels of the genes. Or, the omics data may be other gene-related omics data, where the biomolecules are germline DNA, SNPs, or somatic mutations and the associated values ​​are, for example, the magnitude of mutation impact. Or, the omics data corresponds to biomolecules such as proteins or metabolites and the associated values ​​are the abundances of the proteins or metabolites, respectively (i.e., the omics data is proteome or metabolome data).

[0025] In other embodiments, the apparatus 10 may process any suitable data, which may include non-omics data, such as any unordered data. For example, in some embodiments, the apparatus 10 may process any data that includes a plurality of values, each of which is associated with a distinct variable.

[0026] The apparatus 10 comprises a computing device 12, which in this example is a personal computer (PC) or workstation. The computing device 12 is connected to a display screen 16 or other display device and one or more input devices 18, such as a computer keyboard and a mouse. The computing device 12 receives data from a memory 40, which may also be referred to as a data store or storage. In alternative embodiments, the computing device 12 receives data from one or more further data stores (not shown) instead of or in addition to the memory 40. For example, the computing device 12 may receive data from one or more remote data stores (not shown), which may include a cloud-based store.

[0027] The memory 40 stores a two-dimensional image format that maps a plurality of biomolecules to respective display positions of the two-dimensional image. According to the two-dimensional image format, each of the plurality of biomolecules is mapped to a pixel position that is displayed as an image on the display screen 16. The two-dimensional image format can be, for example, a matrix in which each cell of the matrix in the two-dimensional image is associated with an individual biomolecule, and multiple ways can be defined as to which matrix each individual biomolecule is located. In another example, instead of a two-dimensional array, the two-dimensional image format can be an image file format, such as a JPG or BMP file, in which pixel positions of the image are associated with individual biomolecules.

[0028] The memory 40 further stores the omics data. In other embodiments, the omics data may be stored in another suitable memory, for example in another device, or in a cloud-based memory. The omics data may be stored in any file format suitable for storing text-based data, such as TSV, CSV, XLS, XML, etc.

[0029] The computing device 12 includes a processing circuit 22 for processing data. The processing circuit 22 includes a Central Processing Unit (CPU) and a Graphical Processing Unit (GPU) and / or a Tensor Processing Unit (TPU). The processing circuit 22 provides processing resources for automatically or semi-automatically processing the omics data sets.

[0030] The processing circuit 22 includes a distance circuit 24 that calculates the distance between the biomolecules of each biomolecule pair based on the multiple biomolecules in the omics data, an embedding circuit 26 that maps each of the multiple biomolecules to a position based on the calculated distance, an adjustment circuit 28 that adjusts the position to obtain a two-dimensional image format, and a display circuit 30 that displays the omics data based on the two-dimensional image format.

[0031] In some embodiments, the processing circuitry further comprises training circuitry 32 for training one or more mappings and for training one or more Convolutional Neural Networks (CNNs). In other embodiments, different circuits or devices may be used to train different mappings or CNNs.

[0032] In some embodiments, at least one of distance circuitry 24 , embedding circuitry 26 , and conditioning circuitry 28 may be omitted from processing circuitry 22 .

[0033] In this embodiment, the circuits 24, 26, 28, 30 are each implemented in a CPU and / or GPU and / or TPU by a computer program having computer readable instructions executable to perform the method of the embodiment. In other embodiments, the circuits may be implemented as one or more Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs).

[0034] Computing device 12 also has other components of a PC, including a hard drive and RAM, ROM, a data bus, an operating system including various device drivers, and hardware devices including a graphics card, although such components are not shown in FIG.

[0035] The processing circuitry 22 of Figure 2 executes a method according to Figure 3. Figure 3 is a flow chart outlining a method for generating a two-dimensional image format for representing a plurality of biomolecules as a two-dimensional image, and then obtaining and displaying the image based on the two-dimensional image format, according to an embodiment.

[0036] At stage 50, distance circuit 24 receives omics data from memory 40 or any suitable data storage. The omics data includes a plurality of values, each value of which is associated with an individual biomolecule of the plurality of biomolecules. The plurality of biomolecules includes N biomolecules. In this embodiment, the biomolecules are genes associated with respective gene expression values. In other embodiments, the omics data received by distance circuit 24 at stage 50 may include only a list of N biomolecules, such as a list of N genes, without including values ​​associated with the N biomolecules. Note that, although this embodiment uses data that associates individual gene types with their expression values ​​as an example, the omics data may be data that indicates bioinformation that associates a type of bioinformation with a parameter, such as data that associates a type of protein with its amount.

[0037] Performing distance evaluation such that the distance between highly related biomolecules is short and the distance between less related biomolecules is long, and formatting the data for two-dimensional imaging is advantageous for subsequent ML (Machine Learning) model analysis. Thus, stages 54 and 58 each provide a method for calculating the distance between N biomolecules. In some embodiments, stage 54 is performed and stage 58 is omitted. In other embodiments, stage 58 is performed and stage 54 is omitted. In further embodiments, stages 54 and 58 or outputs from stages 54 and 58 may be combined in any suitable combination to obtain the distance between the N biomolecules.

[0038] The input to stage 54 is data 52 from one or more bioinformatics databases, sometimes referred to as biological databases. The bioinformatics database or biological database may be a database that uses a network to store knowledge or a knowledge graph, such as the Search Tool for the Retrieval of Interacting Genes / Proteins (STRING) or the Kyoto Encyclopedia of Genes and Genomes (KEGG) database. In such a database, biomolecules are represented by nodes that are connected to each other by edges, if the scientific literature indicates that they are interrelated. The edges of the nodes may be weighted based on the strength of the interaction or based on the confidence that the interaction is likely to be true.

[0039] At stage 54, the distance circuitry 24 selects each biomolecule pair in turn from the plurality of biomolecules and calculates a distance for each biomolecule pair from the plurality of biomolecules using data from one or more bioinformatics databases. A distance is calculated for each possible pair.

[0040] Using the STRING network as an example where proteins are represented as nodes and edges are weighted by a confidence value for the likelihood that the interaction is true, the distance between two connected proteins may be calculated using the formula (1-confidence value). If the two proteins are not directly connected in the node network, the distance may be calculated based on the minimum path distance between the two nodes.

[0041] If the genes code for proteins, a protein-protein network such as STRING may be used to determine the distance between said genes. For example, if there are N genes, then for each gene pair that defines a cell of the distance matrix, a distance value between the two proteins that the gene pair codes for may be calculated using information from the STRING network to calculate an N×N distance matrix.

[0042] As an alternative to the STRING database, other databases may be used, such as databases that store links between genes based on co-expression, or databases that store links between DNA, RNA, or protein sequences based on sequence homology, or other databases that store links between biomolecules based on shared activity in biochemical pathways, or based on descriptions of co-occurring in the scientific literature.

[0043] Distance circuitry 24 outputs distances 60 obtained at stage 54. The distances may be stored as a distance matrix of dimension N×N, where N is the number of biomolecules, or as a list, or any suitable data structure for storing numerical values.

[0044] The input to stage 58 is study cohort data 56, which includes data, such as transcriptome data, for a number of individuals in a study cohort. For example, a cohort may include data for hundreds or thousands of patients or other subjects. Any suitable cohort data may be used.

[0045] At stage 58, the distance circuit 24 selects each biomolecule pair from the plurality of biomolecules in turn and calculates the distance between each biomolecule pair from the plurality of biomolecules using data from the study cohort. A distance is calculated for each possible pair.

[0046] The distance between the biomolecule pairs is calculated by performing a correlation analysis. For example, for each biomolecule pair, an individual correlation coefficient, such as a sample correlation coefficient, is calculated across the subjects of the cohort based on the values ​​associated with that biomolecule pair. Once the sample correlation coefficient is determined, the distance is calculated as the inverse of the correlation coefficient. In this way, highly correlated biomolecules will have a correspondingly shorter distance between the biomolecules. In other embodiments, any suitable method of obtaining distances from cohort data may be used.

[0047] For example, for transcript data containing N genes, we generate an N × N correlation matrix containing the correlation coefficients for each gene pair, where high correlation values ​​correspond to similar expression patterns between each gene pair. Then, for each cell of the distance matrix, we take the corresponding correlation coefficient from the correlation matrix: (1-correlation coefficient) or (1 / correlation coefficient) Then, the inverse of either of these is calculated to generate an N×N distance matrix.

[0048] In other embodiments, any suitable correlation measure may be used, and any suitable inverse may be used.

[0049] Distance circuitry 24 outputs distances 60 obtained at stage 58. The distances may be stored as a distance matrix of dimension N×N, where N is the number of biomolecules, or as a list, or any suitable data structure for storing numerical values.

[0050] In some embodiments, the distance 60 is obtained only at stage 54. In other embodiments, the distance 60 is obtained only at stage 58. In further embodiments, the distance 60 is obtained by combining stages 54 and 58, or by combining the outputs of stages 54 and 58. That is, although a method for evaluating the distance between gene pairs, i.e., the degree of association, using multiple scales and databases has been described, the distance may be calculated by combining multiple scales. For example, the confidence that the interaction between the gene pairs is likely to be true may be used as a first scale, and the distance obtained by correlation analysis of omics data in a cohort may be used as a second scale, and the distance between the gene pairs may be calculated by multiplying the first scale and the second scale.

[0051] At stage 62, the distance circuitry 24 passes the calculated distances to the embedding circuitry 26. The embedding circuitry 26 performs a mapping technique, which in this embodiment is a manifold learning technique, to embed the plurality of biomolecules into a two-dimensional space based on the calculated distances 60. In other embodiments, the mapping technique used to embed the biomolecules into a two-dimensional space may not include a manifold learning technique. Any suitable technique may be used.

[0052] A manifold learning method may be a method that projects a set of elements, such as a multidimensional scaling (MDS) algorithm, from a high dimensional space to a low dimensional space (e.g., 2D) while preserving as much as possible a defined distance between some elements, so that biomolecules with small calculated distances will be close to each other in the embedding.

[0053] Two-dimensional embedding assigns a location to each biomolecule of a plurality of biomolecules, the location being a position in two-dimensional space, e.g., expressed as x,y coordinate values, such that correlated biomolecules are expected to be assigned to nearby locations in the two-dimensional space.

[0054] In other embodiments, any suitable dimensionality reduction method may be used to obtain the coordinates of biomolecules in two-dimensional space using calculated distances, such as Uniform Manifold Approximation and Projection (UMAP) or Stochastic Neighbor Embedding (t-SNE).

[0055] In some embodiments where the biomolecules are genes, manifold learning is applied to an N×N gene distance matrix to obtain a two-dimensional embedding that assigns each gene a distinct location in two-dimensional space. The embedding preserves inter-gene distances such that closely mapped genes are correlated. It is also envisioned that manifold learning methods may be applied to any other suitable data organization that includes distance calculations.

[0056] The embedding circuit 26 outputs an embedding that includes a set of two-dimensional coordinate values ​​64 that includes distinct two-dimensional coordinate values ​​for each of the N biomolecules.

[0057] Figure 4 shows an example of a set of two-dimensional coordinate values ​​output from stage 64. In the example of Figure 4, the biomolecules are genes and the manifold learning method is MDS. The locations output by MDS are labeled with their associated genes.

[0058] Using this mapping between genes and locations, points can be colored depending on whether the corresponding genes are under-expressed, over-expressed, or have small changes compared to the cohort values. This allows for a two-dimensional image of the individual transcriptome when there is an individual transcriptome for one patient or other subject. Coloring can be based on Z-score, where the Z-score indicates the number of standard deviations a value is above or below the mean. In one example, points representing over-expressed genes are colored red, points representing under-expressed genes are colored blue, and points representing small changes are colored gray.

[0059] In the embodiment shown in Figure 4, the positions output by MDS are limited to an area defined by a circle. Therefore, the output positions do not cover the entire rectangular two-dimensional space. Typically, many output positions are on the edge of the circle, and there are no output positions outside the circle. The 2D arrangement of genes by MDS is unevenly distributed in the two-dimensional space.

[0060] At stage 66, adjustment circuitry 28 receives the embedding, including the two-dimensional coordinate values ​​64, and adjusts at least some of the positions in the embedding to more evenly distribute the positions across the two-dimensional space, making the resulting image easier to visually inspect.

[0061] In this embodiment, adjustment circuitry 28 receives the x, y coordinate values ​​and adjusts the coordinate values ​​so that they are more evenly distributed across two-dimensional space. In an alternative embodiment, adjustment circuitry 28 may map the x, y coordinate values ​​to individual pixel locations and adjust the pixel locations so that they are more evenly distributed across the image.

[0062] The adjustment circuit 28 adjusts the coordinate values ​​or pixel positions by applying, for example, an image distortion method. The image distortion method may be, for example, a spatial transformation such as a quadrilateral transformation. When the positions are adjusted by the spatial transformation, the positions are spread more uniformly throughout the two-dimensional space while maintaining the relative distance between the positions output from the MDS as much as possible. Compared to a two-dimensional data distribution in which there are no output positions outside the circle and the output positions are concentrated on the edge of the circle, the uniform distribution of data throughout the two-dimensional space is considered to be advantageous in that it is easier to understand the data even in the subsequent data analysis using an ML model.

[0063] In other embodiments, adjusting the coordinate values ​​or pixel positions may include perturbing the positions corresponding to each biomolecule such that the adjusted positions are evenly spaced on a two-dimensional grid.

[0064] Figure 5 shows an example of adjusted positions obtained by the position adjustment shown in Figure 4. In Figure 4, the 2D space is rectangular and the x, y coordinate values ​​before adjustment are distributed in a circular region of the 2D space. The adjustment includes a quadrilateral transformation that distributes the positions throughout the rectangular 2D space. The adjustment may include non-linear radial scaling that accounts for the clustering of positions at the edge of the circle.

[0065] Figure 5 shows the result of a rectangle transformation, which distorts two-dimensional coordinate values ​​to fill a rectangle. In the example of Figure 5, the biomolecules are genes, and each adjusted position is labeled with the corresponding gene.

[0066] At stage 68, adjustment circuitry 28 applies an image morphology technique to the adjusted locations. If the adjusted locations are x,y coordinate values, adjustment circuitry 28 first maps the coordinate values ​​to individual pixel locations. If the adjusted locations are pixel locations, adjustment circuitry 28 applies the image morphology technique directly to the pixel locations.

[0067] Prior to performing image morphology techniques, each of the N biomolecules is mapped to a separate single pixel in the two-dimensional image space. Since the number of pixels may be large compared to N, many pixels may not be mapped to a biomolecule. Such pixels may be referred to as unmapped pixels. If a gene is not substantially related to other genes, then a pixel representing that gene will be surrounded by more unmapped pixels than a pixel representing a gene with many related genes.

[0068] The adjustment circuitry 28 applies image morphology techniques, such as dilation, to expand at least some of the mapped pixels into adjacent unmapped pixels. A biomolecule is initially mapped to one pixel. After application of the image morphology techniques, each biomolecule may be mapped to two, three, four, or more pixels. The mapping may be maintained so that each biomolecule is mapped to adjacent pixels. This results in a biomolecule map that arranges the biomolecules in a two-dimensional space. If transcriptomics data is used, the resulting map is a genetic map. The arrangement of the data distribution may be optimized such that no unmapped pixels remain due to the image deformation by the image morphology techniques. In an alternative embodiment, some unmapped pixels may be left, for example to form boundaries between regions representing different biomolecules.

[0069] The conditioning circuitry 28 outputs a two-dimensional image format 70 that includes a representation location of each of a plurality of biomolecules. In the embodiment of Figure 3, the representation locations are pixel locations. The image format associates each biomolecule with a separate pixel or pixels.

[0070] In some embodiments, the display circuitry 30 renders an index image, such as a genetic map on a grid, with the display locations in a two-dimensional image format. In the index image, the display locations (e.g., pixels or groups of pixels) are colored according to the genes they represent. FIG. 6 shows an example of a genetic map showing about 900 genes. The color of the pixels can be used as a scale for the genetic map, with the color scale (represented as grayscale in FIG. 6) representing the gene index number. Different colors represent different genes. Genes with consecutive index numbers are not typically expected to be adjacent in a genetic map, since the index numbers themselves do not usually show correlation in genes, and genes with close index numbers are not necessarily closely correlated. Adjacent regions of a map often do not have similar colors. In other embodiments, the index image is not rendered.

[0071] The two-dimensional image format is stored in memory 40. Alternatively, the two-dimensional image format may be stored in one or more remote data stores (not shown), which may include cloud-based storage.

[0072] At stage 74, display circuitry 30 receives the two-dimensional image format from memory 40 or any suitable storage. Display circuitry 30 also receives omics data 72 from memory 40 or any suitable storage. In the embodiment of Figure 5, the omics data includes individual data for a patient or subject, such as transcriptome data. In other embodiments, the omics data includes data for multiple subjects. The omics data may be stored in the format depicted in Figure 1.

[0073] The display circuit 30 generates the display image 20 by determining a color at the display position of each of a plurality of biomolecules, the display positions of which are defined based on the relevance of the biomolecules in the two-dimensional image format, using values ​​in the omics data. Although individual colors may be assigned to the expression levels of individual genes simply using a color scale, the color may be determined using a color map that converts any value associated with a biomolecule into a color scale. The display image 20 is rendered using the colors determined for all display positions in the display image 20. Thus, values ​​from the omics data are mapped to the display image 20 based on the display positions defined in the two-dimensional image format.

[0074] In some embodiments, the values ​​associated with the biomolecules are normalized values. For example, if the omics data includes only one sample, the values ​​may be normalized with a Z-score based on all values ​​in that sample. If the omics data includes only two or more samples, the values ​​associated with a particular biomolecule may be normalized with a Z-score based on values ​​from all samples of the particular biomolecule. Displaying the normalized values ​​makes it easier to visually inspect the data and determine which biomolecules are overexpressed, underexpressed, or activated compared to the average. Alternatively, other normalization methods, such as min-max scaling, may be used.

[0075] In the embodiment of FIG. 3, the omics data is transcriptome data and the color scale represents gene expression values. Gene expression values ​​in the omics data that are specific to a patient are converted to Z-score values ​​for a cohort. The Z-score values ​​are then used to determine colors in the displayed image. For example, the color scale may color pixels representing genes that are overexpressed red, pixels representing genes that are underexpressed blue, and points representing genes with small changes gray. In other embodiments, any suitable color scale may be used to represent any suitable value contained in the omics data.

[0076] If the omics data includes data from more than one subject, the display image 20 can create a separate image for each subject, which can be used to represent the individual values ​​for the mapped patient. The same two-dimensional image format is used for each of the images. Thus, the two-dimensional image format is generated only once for each omics data. Using the same two-dimensional image format for each image facilitates comparison of data from each patient. For example, a two-dimensional location representing a given gene in a first image will represent the same gene in a second, different image.

[0077] At stage 76, display circuitry 30 displays display image 20 on a display screen, such as display screen 16. Display image 20 is displayed for human review.

[0078] As shown in FIG. 7, in one embodiment, the display circuitry 30 displays the images for each subject side-by-side for quick comparison of each patient. In FIG. 7, six individual transcriptome images are rendered using the same two-dimensional image format. Color (shown in grayscale in FIG. 7) represents over- or under-expression. Areas of over- or under-expression indicate coherence. Coherence may further suggest that related genes map close to each other. It can be seen that the images for the six patients are visually different due to the different expression levels of their genes.

[0079] In another embodiment, display circuitry 30 presents one image at a time and provides the ability for a user to switch between images within the same window, for example by providing an input using input device 18 that triggers the display of a different image. For example, different images may be displayed in response to a user input, such as a click, a keystroke, or a repositioning of a slider.

[0080] In some embodiments, as shown in FIG. 8, the display circuitry 30 provides mouse-over functionality to the display image 20. When a cursor 90 controlled by the input device 18 hovers over a display location of a biomolecule, the display circuitry 30 receives an input representing the position of the cursor 90. The display circuitry 30 uses the two-dimensional image format to obtain a name of the biomolecule associated with the cursor position in the two-dimensional image format and / or obtains functional information about the biomolecule. The functional information may be obtained from a bioinformatics database such as Gene Ontology or KEGG Ontology. The display circuitry 30 directs the name and / or functional information to be displayed on or next to the display image 20. In other embodiments, any suitable information may be displayed as a mouse-over display.

[0081] As an example, at stage 74, the display circuitry 30 receives from memory 40 a two-dimensional image format, sometimes described as a gene map. The display circuitry 30 receives omics data 72, including transcriptome data, for a number of subjects. The display circuitry 30 generates a display image 20 of one of the subjects. Each mapped area in the display image 20, which corresponds to a gene, is colored based on the respective Z-score normalized gene expression value for those genes. The display circuitry 30 repeats this process for the other subjects, presenting the images of each subject side-by-side for comparison. The display circuitry 30 adds a mouse-over feature to show the individual gene names (e.g., KRAS, APS, etc.) and functional interpretations of the mapped genes, for example, using gene ontology or KEGG ontology.

[0082] In some embodiments, different hierarchical levels, such as functional levels in an organism, may be represented, such as by outlines in the display image 20. For example, a functional level in the ontology may be selected to identify a group of genes, and the display circuitry 30 may highlight the region representing the group of genes.

[0083] In the example shown in Figure 8, cursor 90 hovers over pixel region 92 representing the KRAS gene. Display circuitry 30 highlights border 94 of region 92. Display circuitry 30 displays a text box 96 containing text associated with the KRAS gene. In the illustrated example, the text is "KRAS:+1.32 Membrane Trafficking (hsa04131):+1.02." In other embodiments, any suitable text conveying any appropriate information may be displayed. Any suitable display method may be used.

[0084] The display image 20 may provide a visually distinguishable representation of the omics data, which may be thought of as providing a gestalt of the data. If the omics data is transcriptomics data for a cohort, the display image 20 may be examined to qualitatively identify differences between subjects. Regions of adjacent genes tend to show coherence, since genes that are plotted closely together are highly correlated. This means that regions that will be coherently under- or over-expressed across a cohort will appear together in the image. Thus, with practice, one may become familiar with the co-expression profile of a given gene in a dataset.

[0085] In some embodiments, display image 20 may be used as a thumbnail image for a dataset, such as a transcriptome dataset, such that different datasets may be visually distinguished efficiently.

[0086] A feature of the displayed image format is that it exhibits coherence, where neighboring pixels typically exhibit correlation, and typically also coherence with respect to biological function, e.g., neighboring pixels are more likely to belong to the same Gene Ontology (GO) category and KEGG biological pathway. To allow meaningful comparison of transcriptome images from different sources, the image format is specified in an absolute manner. The image format can be considered as a reproducible constant map format.

[0087] In some embodiments, display circuitry 30 performs stage 78 in addition to stage 76. In other embodiments, stage 78 is omitted. In further embodiments, stage 78 may be performed without performing stage 76.

[0088] At stage 78, the display circuitry 30 uses the display image 20 as an input to a deep learning algorithm, such as a convolutional neural network (CNN), based on a machine learning model. The deep learning algorithm may perform any suitable operation on the display image 20. For example, the deep learning algorithm may classify the deep learning image. The deep learning algorithm processes the display image to obtain an output 80, such as a classification.

[0089] In the method of Figure 3, unordered omics data is converted into a representation image 20 that is ordered in two-dimensional space. The conversion of the unordered data into ordered data may allow the resulting ordered representation image to be used as input to a machine learning method that operates on the image, such as a CNN. The machine learning method may perform any suitable task, which may include any suitable omics data processing.

[0090] In some embodiments, the display image 20 is augmented with a parallel atlas image to allow the network to exploit non-stationary properties. Typically, for omics data, it would be useful to use location, since biomolecules are represented at specific locations. Therefore, an atlas may be provided to the CNN along with the display image 20. The atlas may include or be derived from a two-dimensional image format. The atlas may provide a mapping from biomolecule names or other information to locations in the display image 20. The atlas may include a set of labeled x,y coordinates.

[0091] The method of FIG. 3 provides omics data displayed as an image in a manner that maintains the relationship between each biomolecule in the data set. Biomolecules will be adjacent to each other in the displayed image 20 if they have a strong correlation based on the data set or if they have some kind of functional relationship based on prior knowledge. The image provides a qualitative overview of the data and can be used as an adjunct to other bioinformatics methods used for data processing. The image can be used as an icon or preview of a file or UI element related to the omics data. The image allows the data to be visually inspected by a user who, with experience, will learn to recognize patterns in multiple images based on the same two-dimensional format.

[0092] The proximity of neighboring biomolecules is meaningful, and the images are suitable for a CNN-based ML model that can classify the images.

[0093] The two-dimensional image formats (e.g., genetic maps) may be obtained in a data-driven (using cohort data) or knowledge-driven (using biological database data) manner. In some circumstances, obtaining the two-dimensional image formats from external knowledge may result in more functionally interpretable image coherence and may be cohort independent.

[0094] In other embodiments, different devices may be used to perform different operations or different parts of the operations described below. For example, a first device may be used to perform distance calculations, a second device may be used to perform embedding, a third device may be used to perform image alignment, and a fourth device may be used to display the image. Any suitable combination of devices may be used.

[0095] In some embodiments, a first device is used to generate a two-dimensional image format, and one or more further devices then use the two-dimensional image format to generate display images from the omics data, which may only need to be generated once for a given set of biomolecules and can then be reused to display many data sets that include that set of biomolecules.

[0096] In some embodiments, one or more of the above-mentioned stages may be omitted or multiple stages may be combined.

[0097] When considering unordered data such as transcriptomic data, there may be various ways to impose an order that respects the proximity relatedness of both low-level and high-level features. Such an ordered space may perform better in CNN analysis than unordered data. Depending on the circumstances, such an ordered space may make the operation of DL algorithms more interpretable to a human observer. Such an ordered space may allow data from multiple modalities to be integrated in an interpretable way.

[0098] The embodiment described above with respect to Figures 3 to 8 provides a two-dimensional image format that provides an ordering of data, e.g., transcriptome data. Among the numerous methods for mapping transcriptome data onto an ordered space, the technique described below is very useful in CNN analysis to find a mapping method that performs better in CNN analysis.

[0099] Further methods for obtaining data ordering are described below in connection with Figures 9 and 10. In Figures 9 and 10, an encoder-decoder framework is used that allows for mapping from unordered input data to an ordered multidimensional space. The mapping is optimized directly in a data-driven manner using unannotated and unordered data, e.g., transcriptome data. The multidimensional space is optimized to preserve the data when it is processed using a pre-trained CNN, e.g., an autoencoder. This may force low-level and high-level spatial meaning into the multidimensional space. To accurately reconstruct the data, the data must be mapped into a space that can be represented by the CNN.

[0100] FIG. 9 shows an overview of a method for training a mapping that maps unordered data to an ordered multidimensional space. The stages shown in FIG. 9 are repeated multiple times to optimize the mapping. The mapping is trained using a pre-trained CNN 108, whose weights are locked during the mapping training process. The mapping is optimized to give an ordered representation of the previously unordered data. Now, by optimizing the ordering mapping method and inputting the data spatial distribution processed by the optimized mapping method to the CNN, it can also be processed by the CNN with high accuracy.

[0101] First, the training circuitry 32 obtains an unordered data set 100 from the training cohort. The unordered data set 100 is described as unordered because it does not have any meaningful order, such as an order in which degrees of proximity correspond to degrees of relatedness. In order to establish a mapping technique, the unordered data set 100 must be processed into a data set that has some order. For the purposes of this example, the data are stored in a data format that is not believed to provide any meaningful order, such as alphabetical order by gene name.

[0102] In the embodiment of Figure 9, the unordered data 100 is transcriptomic data that includes gene expression values ​​for a plurality of genes. The unordered data 100 includes a list of genes and corresponding gene expression values: gene1, gene2, gene3, .... The training cohort includes transcriptomic data for a plurality of subjects, e.g., hundreds or thousands of subjects.

[0103] In other embodiments, any suitable unordered data set 100 may be used, such as any set of omics data. The omics data may be any type of omics data including a plurality of values, each of which is associated with a distinct biomolecule of the biomolecules. The data may be, for example, transcriptome data, where the biomolecule is a gene and the associated value is the expression level of the gene. Or, the omics data may be another gene-related omics data, where the biomolecule is a germline DNA, SNP, or somatic mutation, and the associated value is, for example, a degree of mutation impact. Or, the omics data corresponds to a biomolecule, such as a protein or metabolite, and the associated value is the abundance of the protein or metabolite, respectively (i.e., the omics data is proteome or metabolome data).

[0104] 9, at stage 102, the training circuit 32 performs a first mapping of the unordered data set 100 into an N-dimensional space, which may be of any suitable number of dimensions, for example one, two or three dimensions.

[0105] The first mapping may be described as a first mapping layer. The first mapping may also be described as a data-driven mapping. In the first mapping, gene expression values ​​from an unordered dataset are mapped to positions in an N-dimensional space. In the embodiment of FIG. 9, the first mapping layer is a dense layer. A dense layer does not enforce a one-to-one mapping and allows weighting of gene expression values.

[0106] In other embodiments, the mapping is an unweighted one-to-one mapping. Such a mapping may use one or more algorithms similar to those used in image registration.

[0107] 9, stage 102 is represented as a number of lines that map gene expression values ​​of the unordered dataset to positions in N-dimensional space. The output of stage 102 is a mapped dataset 104 that includes data values ​​for a number of positions in the N-dimensional space.

[0108] In the embodiment of Figure 9, N = 2. For example, an input of 100 genes may be mapped to a two-dimensional space of 10 x 10. If the mapping is not one-to-one, the input of 100 genes can be mapped to a smaller space, for example, 5 x 5. Such a mapping results in a dimensionality reduction. In other embodiments, any suitable value for N may be used, for example, N = 3.

[0109] At stage 106, the training circuitry 32 inputs the mapped data 104 to a pre-trained CNN 108. The pre-trained CNN 108 is defined by parameters including a set of weights, sometimes described as processing weights or convolutional processing weights. The weights of the CNN 108 are locked during the method of Figure 9 and do not change their weighting. Training of the pre-trained CNN 108 is described below with reference to Figure 10.

[0110] 9, the pre-trained CNN 108 is an autoencoder. In other embodiments, any suitable pre-trained CNN 108 may be used. The pre-trained CNN 108 includes multiple layers that operate on low-level and high-level features of the mapped data.

[0111] The pre-trained CNN 108 encodes the mapped data 104 using high-level features to obtain a compressed representation of the mapped data 104, and then obtains the reconstructed data 112 from the compressed representation. The compressed representation may include an embedding having a lower dimensionality than the mapped data. The encoding may include feature compression into low, medium, and high level features.

[0112] At stage 110, the pre-trained CNN 108 outputs reconstructed data 112 into the further ordered N-dimensional space. The reconstructed data 112 includes a number of data values ​​having associated positions within the further ordered N-dimensional space.

[0113] At stage 114, the training circuit 32 performs a second mapping to map the reconstructed data from the further ordered N-dimensional space back to the original unordered space to obtain an output unordered data set 116. The second mapping may also be referred to as a second mapping layer. The second mapping may also be referred to as a data-driven mapping.

[0114] The unordered output data set 116 has the same data format as the unordered data set 100, which in the embodiment of Figure 6 includes a list of genes, gene1, gene2, gene3... and corresponding gene expression values. Data values ​​in the N-dimensional space are mapped to gene expression values ​​in the original unordered space. In Figure 6, stage 114 is represented as a number of lines that map locations in the N-dimensional space to data elements of the output data in the unordered space.

[0115] In some embodiments, the second mapping is restricted to be the inverse of the first mapping, while in other embodiments, the second mapping is not so restricted.

[0116] The method of Figure 9 is performed multiple times on unordered data sets from the training cohort to optimize parameters of the first and second mappings, for example using multiple batches of training cohort data.

[0117] The goal of optimizing the mapping method is to find a mapping technique that maps the ordered space into an N-dimensional space where the pre-trained CNN 108 can reconstruct the mapped data with minimal reconstruction loss. The representation of data in the N-dimensional space transformed by such an optimized mapping technique would be expected to have both high-level and low-level features of a type that can be processed by a CNN. The representation of data in the N-dimensional space is an ordered dataset in a form that can be processed by a CNN.

[0118] Optimizing the parameters of the first and second mappings includes optimizing with respect to a reconstruction loss. The reconstruction loss may include or be derived from a difference between the mapped data 104 and the reconstructed data 112. For example, the reconstruction loss may include a mean square error, a mean absolute error, or a mutual information error.

[0119] The optimization may continue until the reconstruction loss falls below a given threshold. Any suitable optimization method may be used. For example, the optimization method may include stochastic gradient descent or the Adam optimizer.

[0120] An accurate reconstruction of the unordered data may be thought of as meaning that a first mapping of the unordered data into an N-dimensional space results in an ordered representation that can be successfully processed by a CNN. Because the CNN weights are locked during the process of Figure 9 and are fixed, improvements in the reconstruction may be seen as a result of improvements in the mapping, and not a result of changing the weights of the CNN itself.

[0121] Figure 10 provides a more complete overview of how a mapping is trained and then used to provide input to a further CNN. The method of Figure 7 may be considered as three tasks: Task 1, Task 2, and Task 3. In some embodiments, Task 2 and Task 3 are performed together.

[0122] In the method of Figure 9 described above, a mapping is trained to generate a representation of ordered data using a pre-trained CNN 108.

[0123] Task 1 of the method of Figure 10 is a method of training a CNN. A CNN trained in task 1 may be described as a pre-trained CNN and may be used in a similar manner to that described in Figure 9. In the embodiment of Figure 10, the CNN trained in task 1 is a denoising autoencoder.

[0124] Task 2 of the method of Figure 10 is similar to the method of Figure 9 in that it involves training the mapping using a pre-trained CNN. In the embodiment of Figure 10, the pre-trained CNN is obtained as the output of Task 1 and is locked during the training of the mapping, which is performed in Task 2.

[0125] Task 3 of the method of Figure 10 uses the mapping from task 2 to convert the unordered data into ordered data in N-dimensional space, where the ordered data is suitable for use as input to a CNN. The ordered data is then input to a task-specific CNN to generate a specific task. Converting the unordered data into ordered data that can be used as input to a CNN allows the CNN to be used to process data that was originally unordered.

[0126] Task 3 involves training a task-specific CNN to perform a specific task. In the embodiment of Figure 10, the specific task is prediction. For example, the prediction may be predicting a state of a subject based on omics data or predicting a clinical state of a biological tissue obtained by a pathology test.

[0127] Returning again to task 1, task 1 involves training a CNN denoising autoencoder on a reconstruction of external data with spatial ordering. In other embodiments, the training instead or additionally involves training a CNN to classify or process external data with spatial ordering.

[0128] The external data may be non-omics data. In the embodiment of Fig. 10, the external data includes natural images. The natural images may be images of real objects captured by conventional imaging methods such as a camera.

[0129] In task 1, the training circuitry 32 receives or generates a CNN denoising autoencoder 122 to be trained.

[0130] The training circuitry 32 receives an image training cohort including a plurality of training image data sets, each of which represents a natural image. The training circuitry 32 adds noise to each of the training image data sets to obtain a corresponding plurality of input image data sets 120. Each input image data set 120 includes data representing a natural image to which noise has been added.

[0131] The training circuit 32 inputs an input image dataset 120 (denoted as “image+noise” in FIG. 10 ) to a CNN denoising autoencoder 122 .

[0132] The CNN denoising autoencoder 122 encodes the input image dataset using high-level features to obtain a compressed representation of the input image dataset 120. The CNN denoising autoencoder 122 then obtains reconstructed data from the compressed representation and outputs the reconstructed data as an output image dataset 124 representing the image.

[0133] Task 1 is repeated for many input datasets to train a CNN denoising autoencoder 122 to receive a natural image+noise and output the denoised natural image. Training includes updating the weights of the CNN denoising autoencoder. Training may include optimizing a loss function. The loss function used to train Task 1 may be based on the difference between the output image dataset 124 and the training image dataset used to create the corresponding input image dataset 120. Optimization may continue until the loss falls below a given threshold.

[0134] Task 1 may train the CNN denoising autoencoder 122 to deploy filters similar to Gabor filters. In some embodiments, Gabor filters may be manually added to the CNN denoising autoencoder. In some embodiments, training the CNN denoising autoencoder may include prioritizing certain convolutions. The certain convolutions may include Gabor filters.

[0135] After Task 1, the weights of the CNN denoising autoencoder 122 are locked. Task 1 pre-trained the CNN denoising autoencoder 122. The CNN denoising autoencoder 122 used in Task 2 and optionally Task 3 may be described as a pre-trained CNN.

[0136] In other embodiments, the training circuitry 32 may receive a CNN denoising autoencoder 122 that has already been pre-trained, for example in a separate device. In such embodiments, task 1 may be omitted. In further embodiments, any suitable method of training the CNN denoising autoencoder 122 may be used.

[0137] In a further embodiment, only some of the weights of the CNN denoising autoencoder are locked while performing task 2 and / or task 3. The CNN denoising autoencoder may continue to be trained during task 2 and / or task 3.

[0138] Although tasks 2 and 3 are described below as separate training tasks, in other embodiments, tasks 2 and 3 may be performed together. Tasks 2 and 3 may be performed alternately.

[0139] For task 2, the training circuit 32 receives or generates a first mapping 132 and a second mapping 134, each of which is to be trained. In some embodiments, the first mapping 132 and the second mapping 134 have a fixed relationship. For example, the second mapping 134 may be the inverse of the first mapping 132. In other embodiments, no such fixed relationship is defined. The first mapping 132 and the second mapping 134 may be similar to the first mapping and the second mapping described above with reference to FIG. 9.

[0140] The training circuit 32 receives a transcriptome training cohort that includes a plurality of transcriptome data sets each for a corresponding subject, where each transcriptome data set includes unordered data including gene expression values ​​for a plurality of genes.

[0141] The training circuit 32 adds noise to each transcriptome dataset to obtain a plurality of corresponding input transcriptome datasets 130. Each input transcriptome dataset 130 includes data representative of a noisy transcriptome dataset. In other embodiments, any suitable omics data may be used in place of the transcriptome datasets.

[0142] The training circuit 32 inputs the input transcriptome dataset into a first mapping 132. The first mapping 132 is a spatial mapping that transforms the input transcriptome dataset into an ordered N-dimensional space. The output of the first mapping 132 is a mapped dataset.

[0143] The first mapping 132 may include a dense layer, which may be parameterized as a fully dense layer. The first mapping 132 may include a fully connected layer, which provides the most flexibility in mapping. The first mapping 132 provides a one-to-one mapping between genes in the transcriptome and locations in the N-dimensional space. For example, the first mapping may be parameterized by a non-rigid space transformation.

[0144] The mapped dataset is input to a CNN denoising autoencoder 122 that is pre-trained on task 1. The weights of the CNN denoising autoencoder 122 are fixed during task 2, and we do not change the weight values.

[0145] The CNN denoising autoencoder 122 processes the mapped data, which includes encoding the mapped data to obtain a compressed representation of the mapped data, obtaining reconstructed data from the compressed representation, and outputting the reconstructed data into a further N-dimensional space. The pre-trained CNN denoising autoencoder 122 may be thought of as processing the mapped data as if it were an image.

[0146] The training circuit 32 inputs the reconstructed data output by the CNN denoising autoencoder 122 into a second mapping 134. The second mapping 134 is a spatial mapping that transforms the reconstructed data from a further N-dimensional space into an output transcriptome dataset 136.

[0147] The second mapping 134 may include a dense layer, which may be parameterized as a fully dense layer. The first mapping 132 may include a fully connected layer, which provides the most flexibility in mapping. The second mapping 134 provides a one-to-one mapping between locations in the further N-dimensional space and genes in the output transcriptome. For example, the second mapping may be parameterized by a non-rigid space transformation.

[0148] Task 2 is repeated on many input datasets to train a first mapping 132 that maps from transcriptomic data to data in an N-dimensional space, and to train a second mapping 134 that maps from data in a further N-dimensional space to transcriptomic data.

[0149] The loss function used in training task 2 may include a reconstruction loss. The reconstruction loss may be based on the input transcriptome and the output transcriptome, for example, the difference between the input transcriptome and the output transcriptome. Optimization of the mapping method may continue until the reconstruction loss falls below a given threshold.

[0150] In summary, in task 2, the CNN denoising autoencoder 122 is locked and the mapping layers 132, 134 are trained with the task of transcriptome denoising that enforces a mapping that allows the denoising autoencoder to denoise the transcriptomics data. In other embodiments, any suitable CNN may be used to train the mapping layers.

[0151] In task 3, the training circuitry 32 receives or generates a task-specific CNN to be trained. A task-specific CNN is a CNN that receives ordered input data and performs a specific task, such as classification, using the ordered input data. The task may include obtaining a prediction. The prediction may be, for example, of a medical condition or pathology. The prediction may be a prediction of treatment outcome, a survival prediction, or a recurrence prediction. The task may include, for example, predicting a type of cancer, whether a polyp will recur. The task may be described as a downstream task. The task may include classification, such as classification of a disease, a phenotype, or one or more disease features.

[0152] Training circuitry 32 uses a first mapping 132 that was trained on task 2. At least some parameters of first mapping 132 are locked when used on task 3. In the embodiment of FIG. 10, training circuitry 32 also uses a CNN denoising autoencoder 122 that was trained on task 1. CNN denoising autoencoder 122 is locked when used on task 3.

[0153] The training circuit 32 receives labeled training data including a plurality of transcriptome data sets each labeled with ground truth data for a task performed by the task-specific CNN. The ground truth data may include, for example, data regarding a condition or pathology. The ground truth data may include outcomes such as information about polyp recurrence. The ground truth data may include a type of cancer. In other embodiments, any suitable ground truth data may be used for any suitable task performed by the task-specific CNN.

[0154] Note that in the embodiment of Figure 10, the training data used to train the task-specific CNN for task 3 is labeled data, while the training data used for task 2 may be unlabeled. In other embodiments, both task 2 and task 3 are trained with unlabeled data. In further embodiments, both task 2 and task 3 are trained with labeled data.

[0155] The training circuit 32 inputs the transcriptome dataset 140 into a first mapping 132. The first mapping 132 transforms the transcriptome dataset 140 into an ordered N-dimensional space. The output of the first mapping 132 is a mapped dataset.

[0156] In the embodiment of Figure 10, the training circuit 32 inputs the mapped data to a CNN denoising autoencoder 122. The CNN denoising autoencoder 122 processes the mapped data, which includes encoding the mapped data to obtain a compressed representation of the mapped data, obtaining reconstructed data from the compressed representation, and outputting the reconstructed data into a further N-dimensional space.

[0157] The training circuitry 32 inputs the reconstructed data into a task-specific CNN 142. The task-specific CNN processes the reconstructed data and outputs a prediction 144. For example, the prediction may include an outcome or type of cancer.

[0158] In other embodiments, the CNN denoising autoencoder 122 is omitted from task 3 and the mapped data is input to the task-specific CNN 142.

[0159] Task 3 is repeated on many labeled transcriptome datasets to train a task-specific CNN 142 with the task of making predictions 144. The loss function used in training the task-specific CNN may include or be derived from the difference between the predictions 144 and ground truth data in the transcriptome dataset. The task-specific CNN is trained to generate correct predictions.

[0160] In some embodiments, multiple task-specific CNNs may be trained to perform different tasks. In some embodiments, one task-specific CNN may be trained to perform multiple tasks, e.g., multiple related tasks.

[0161] In the embodiment of FIG. 10, task 3 is trained after task 2, and the first mapping is not affected by the training of the task-specific CNN. In other embodiments, the first mapping may be trained using errors resulting from one or more tasks performed by the task-specific CNN, which may be described as one or more downstream tasks. The training using errors resulting from one or more tasks may be performed simultaneously with the training using the reconstruction loss described above with reference to task 2, or may be performed after the training using the reconstruction loss. Task 2 and task 3 may be trained together or alternately. The first mapping may be trained to suit the task performed by task 3 by training task 2 and task 3 together or alternately.

[0162] Once the task-specific CNN 142 is trained, predictions may be obtained by deploying the task-specific CNN 142 on new unlabeled data and performing a stage of task 3 using a locked version of the trained task-specific CNN. The display circuitry 30 may display the predictions to a user, for example on the display screen 16. The display circuitry 30 may additionally display the spatially ordered transcriptome data.

[0163] 11 shows an example of a display of spatially ordered transcriptomics data and predictions, provided on a user interface 150. The user interface 150 displays a subject ID 152, such as a patient identifier.

[0164] The display circuitry 30 receives the omics data set from the memory 40 or any suitable data storage. In this embodiment, the omics data set includes transcriptome data for a single patient. In other embodiments, any suitable omics data may be used. The omics data may be generated for multiple patients individually.

[0165] Display circuitry 30 performs task 3 of FIG. 10 using a locked version of the trained task-specific CNN and a locked version of the first mapping. In the embodiment of FIG. 11, three task-specific CNNs were trained: a first task-specific CNN predicts Non-Small Cell Lung Cancer (NSCLC) imaging biomarkers; a second task-specific CNN predicts NSCLC recurrence; and a third task-specific CNN performs frailty scoring. Task 3 is performed on one or more trained task-specific CNNs according to user selection.

[0166] A user selects one or more task-specific CNN models to use by interacting with a model selection box 154 on the user interface 150. The user checks one or more of the models listed in the model selection box 154. In this embodiment, NSCLC (non-small cell lung cancer) imaging biomarkers, NSCLC recurrence prediction, and frailty scoring are listed. The display circuit 30 receives input data representing the user selection. In other embodiments, any suitable task-specific model or models may be used, and any suitable type of user selection may be used, such as a checkbox, a drop-down menu, or free text input. In some embodiments, the user does not select the model(s) and the model(s) are automatically selected.

[0167] For each selected task-specific CNN model, the display circuit 30 inputs the omics dataset to a first mapping 132 that generates mapped data. The display circuit 30 inputs the mapped data to a CNN denoising autoencoder 122. The CNN denoising autoencoder 122 generates reconstructed data that is input by the display circuit 30 to a trained task-specific CNN 142. The trained task-specific CNN 142 uses the reconstructed data to perform transcriptome analysis. The trained task-specific CNN 142 outputs a prediction for the patient, such as an outcome or prediction of a type of cancer.

[0168] In other embodiments, the trained task-specific CNN 142 operates on the mapped data and omits the CNN denoising autoencoder 122 from the arrangement.

[0169] The training circuit 32 displays a two-dimensional image 156 representing a spatially ordered transcriptome data set. The spatially ordered transcriptome data set may be the mapped data generated by the first mapping or the reconstructed data generated by the CNN denoising autoencoder. Each pixel of the two-dimensional image may correspond to an individual biomolecule. Alternatively, multiple pixels may correspond to one biomolecule.

[0170] If the number of images, N, is greater than two, the training circuit 32 may represent the collection of images as, for example, a three-dimensional image and display the image representation generated by a three-dimensional image viewer.

[0171] The two-dimensional image 156 may be colored in any suitable manner, such as, for example, in a manner similar to that described above with reference to the two-dimensional image format generated by the method of FIG. 3. The two-dimensional image 156 may be colored according to gene expression values. For example, the color scale may color pixels representing genes that are overexpressed red, pixels representing genes that are underexpressed blue, and points representing genes with small changes gray. In other embodiments, any suitable color scale may be used to represent any suitable values ​​contained in the omics data.

[0172] Mapped or reconstructed data may provide a more human understandable visualization than unordered omics data. Images 156 may provide visually distinguishable representations of the transcriptome data. Regions of adjacent genes may show coherence.

[0173] The display circuitry 30 identifies regions 160, 162, and 164 of the spatially ordered transcriptome data image 156 with respect to the prediction of the task-specific CNN model. For example, attention-based techniques may be used to identify genes with high contributions to the prediction. In the example of FIG. 11, the display circuitry 30 highlights the regions 160, 162, and 164 with individual outlines. In other embodiments, any suitable method of highlighting one or more regions may be used. The display circuitry 30 also displays a model prediction box 166 in the user interface that lists the predictions made by each of the regions 160, 162, and 164. In the example of FIG. 11, the regions 160 and 162 include genes associated with high tumor recurrence risk. The region 164 includes genes associated with tumors with spicules.

[0174] Because the mapping and regions are consistent across all patients, data from different patients can be easily and meaningfully compared by comparing images generated with the optimized mapping method.

[0175] A user may use the visualization provided by user interface 150 to observe whether regions 160, 162, 164 have high or low gene expression values.

[0176] A user may mouse over a pixel of image 156. When a user mouses over a pixel of the image, display circuitry 30 displays pop-up text 170 containing the name of the gene that corresponds to that pixel. The mouse-over function may be similar to that described above with reference to FIG.

[0177] In further embodiments, any suitable display method may be used. In some embodiments, the visualization may be task specific. For example, proximity within the spatially ordered images may represent proximity with respect to a specific task.

[0178] In some embodiments, prediction details may be displayed without also displaying the spatially ordered transcriptome data.

[0179] Data from two or more subjects may be displayed, for example, using the methods described above with reference to FIG.

[0180] In other embodiments, the first mapping may be trained as described above and may be used to display omics data, e.g., transcriptomics data, without being used as input to a task-specific CNN. Images may be used in any suitable manner, e.g., as icons as described above.

[0181] In some embodiments, parallel atlas images may be used in combination with spatially ordered transcriptome data.

[0182] The method of FIG. 10 may produce a trained mapping that converts unordered transcriptome data into ordered data that can be processed by a CNN. This may enable the use of the trained CNN to make predictions based on the mapped data. The mapping may be obtained by a data-derived method based on the training data. The generation of the mapping using a trained autoencoder may produce an ordered data type suitable for processing by a CNN, allowing a task-specific CNN to then be applied to the mapped data.

[0183] CNNs can be designed to exploit hierarchical ordering and individual proximity. Mappings can be obtained with ordering that respects proximity relationships at both low and high level features. Such ordered spaces could allow CNNs to be better utilized on unordered data.

[0184] For example, the spatially ordered data obtained using the first mapping may enable traditional neural network processing techniques. In general, CNNs are less susceptible to overfitting than other models. Using CNNs may reduce the data load for deploying a model or may provide better performance with the same amount of data.

[0185] Such an ordered space would make the operation of deep learning algorithms more interpretable to a human observer. For example, one might use attention-based methods, or overlay intermediate activation maps. This would allow the user to see which regions of the data are used in the model's decision-making. If a task-specific CNN produces a score, they would know why that score was obtained.

[0186] The ordered space may allow for interpretable integration of multiple data modalities. The input may include data from multiple modalities, including, for example, clinical or pathology data in addition to omics data.

[0187] Unordered data will be compared and better visualized. Spatial refinement based on downstream tasks may enable informative data ordering in the clinician's perspective. Data may be ordered at both high and low levels. In some situations, training the mapping may allow a higher degree of representation than ordering by, for example, correlation. The method of FIG. 9 or FIG. 10 may produce a mapping that captures complex, high-dimensional features and relationships.

[0188] In some embodiments, the mapping is trained partially or entirely on unlabeled data. Often, only a limited amount of labeled data is available. Training on unlabeled data may allow more training data to be used, which may improve training. Training the mapping on unlabeled data may allow labeled data to be prepared for training the task-specific CNN, if necessary.

[0189] In some embodiments, correlation data and / or domain knowledge may be used to initialize the method of Figure 9 or Figure 10. The correlation data and / or domain knowledge may include data such as that described above with respect to the embodiment of Figure 3, such as data contained in bioinformatics database(s) 52 and / or study cohort data 56.

[0190] The method of FIG. 9 or FIG. 10 may be used to optimize or improve the initial ordering of data, such as transcriptome data. The initial ordering may be obtained using correlation or knowledge. For example, the method of FIG. 3 may be applied to unordered data to obtain initial ordered data. A mapping may be trained to convert the initial ordered data into mapped data that can be used to obtain task-specific predictions using a task-specific CNN.

[0191] The training of the mapping may be initialized using the two-dimensional image format obtained by the method of FIG. 3. In some situations, there may not be a unique solution to the task of ordering unordered data. The presence of local minima may break the optimization method. To obtain a more optimal mapping, it may be useful to initialize the training of the mapping with a non-random mapping, such as the two-dimensional image format obtained by the method of FIG. 3.

[0192] FIG. 12 shows examples of remapping different image data and how the data looks when a first mapping, a CNN denoising autoencoder, and a second mapping are applied to the data.

[0193] A first natural image 200 and a second natural image 210 are obtained. The natural images 200 and 210 are low resolution and are simple examples. The first natural image 200 and the second natural image 210 are different and represent different objects. In the example of FIG. 12, a frog is represented in the first natural image 200 and a truck is represented in the second natural image 210.

[0194] The first and second natural images 200, 210 are each randomly remapped using the same random remapping, and the resulting images 201, 211 may be considered to contain unordered data, since they have been randomly rearranged.

[0195] A first mapping, sometimes described as a dense remapping layer, is applied to the images 201, 211 to obtain mapped images 202, 212. The mapped images can be thought of as developing some structure from the unordered data before being passed through a CNN denoising autoencoder.

[0196] We apply the CNN denoising autoencoder to the mapped images 202, 212 to obtain output images 203, 213. It can be seen that the output of the CNN denoising autoencoder is different for different images and that the output of the CNN denoising autoencoder is likely to be spatially meaningful.

[0197] Images 204, 214 show the output of the CNN denoising autoencoder as log output.

[0198] A second mapping, sometimes described as a second dense remapping layer, is applied to the output of the CNN denoising autoencoder to produce images 205, 215. Images 205, 215 turn out to be sufficient to reconstruct the remapped images 201, 211.

[0199] In further embodiments, the methods described above may be performed using any suitable unordered data, such as, for example, any suitable omics data relating to any human or animal subject(s). Features of the different embodiments may be combined.

[0200] Any suitable device or combination of devices may be used, for example a first device may be used to train the mapping and a second, different device may be used to apply the mapping.

[0201] One embodiment provides a method for training a mapping layer that converts transcriptome data into mapping data suitable for a Convolutional Neural Network (CNN), the method comprising: Obtaining a CNN layer and training a first mapping layer coupled to an input layer of the CNN layers and a second mapping layer coupled to an output layer of the CNN layers; the first mapping layer maps the transcriptome data to n-dimensional data; the second mapping layer converts the output data of the CNN layer into data corresponding to the transcriptome data; Training the first mapping layer and the second mapping layer is based on determining parameters of the CNN layer to minimize a loss function that takes into account the transcriptome data and the data.

[0202] An embodiment provides a system comprising a processing circuit that maps unordered input data into an ordered space, performs a subsequent convolution process on the data using a Convolutional Neural Network (CNN), and then outputs data from the subsequent process that has been remapped back into the original unordered space, where the mapping(s) are derived in a data-driven manner by optimizing a reconstruction error between the input data and the output unordered data.

[0203] The mapping before the convolution may be the inverse of the mapping following the convolution.

[0204] The mapping may be parameterized as an all-dense layer.

[0205] The mapping may include, for example, a one-to-one mapping parameterized by a non-rigid spatial transformation.

[0206] The ordered space may be one, two, or three dimensional.

[0207] The mapping may be initialized in a manner that is derived from domain knowledge.

[0208] The convolution weights may be derived from a reconstruction of external data having a spatial ordering.

[0209] The convolution weights may originate from Gabor filters. At least some of the convolution weights may originate from training for classifying external data having spatial ordering. The convolution weights may originate from training for processing external data having spatial ordering. Some or all of the convolution weights may be locked during optimization of reconstruction of unordered data.

[0210] The space may be simultaneously or subsequently optimized based on errors resulting from a downstream task or tasks. The unordered data may comprise transcriptomic data. The reconstruction error may comprise mean squared error. The reconstruction error may comprise mean absolute error. The reconstruction error may comprise mutual information error. The processing circuitry may process the data to perform a classification task.

[0211] Although particular circuits are described herein, in alternative embodiments, the functionality of one or more of those circuits may be provided by a single processing resource or other component, or the functionality provided by a single circuit may be provided by a combination of two or more processing resources or other components. Reference to a single circuit encompasses multiple components that provide the functionality of that circuit, whether or not such components are separate from one another. Reference to multiple circuits encompasses single components that provide the functionality of those circuits.

[0212] Although several embodiments have been described, these embodiments are presented for illustrative purposes only and are not intended to limit the scope of the invention. In fact, the novel methods and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms and modifications that fall within the scope of the invention. [Explanation of symbols]

[0213] 10 equipment 12 Computing Equipment 16 Display Screens 18 Input Devices 20 Display images 22 Processing circuit 24 circuits 24 distance circuit 26 Circuit 28 Adjustment circuit 30 Display circuit 32 Training Circuit 40 Memory 50 Stages 52 Data 54 Stages 56 Study cohort data 58 Stages 60 distance 62 Stages 64 Stages 66 Stages 68 Stages 70 2D Image Formats 72 Omics Data 74 Stages 76 Stages 78 Stages 80 Output 90 Cursor 92 pixel region 92 areas 94 Boundary 96 Text Box 100 Data 102 Stages 104 Mapped Data 106 Stages 110 Stages 112 Reconstruction Data 114 Stages 116 Output Data Sets 120 input image dataset 122 Denoising Autoencoder 124 output image dataset 130 Input transcriptome dataset 132 First Mapping 134 Second Mapping 136 Output transcriptome dataset 140 Transcriptome Datasets 144 Predictions 150 User Interface 154 Model Selection Box 156 Transcriptome Data Images 160,162,164 areas 166 Model Prediction Box 170 Pop-up Text 200,210 Natural Images 201,211 Remapped Images 202,212 Mapped Images 203,213 Output images 204,214 images 205,215 images

Claims

1. A processing circuit is provided, the processing circuit comprising: Obtain a Convolutional Neural Network (CNN) layer; training a first mapping layer coupled to an input layer of the CNN layer and a second mapping layer coupled to an output layer of the CNN layer, the first mapping layer converting input omics data representing genetic information into first N-dimensional data, the second mapping layer receiving second N-dimensional data output by the CNN layer and converting the second N-dimensional data into output omics data representing genetic information; The training of the first mapping layer and the second mapping layer may include minimizing a loss function at given parameters of the CNN layer, the loss function being based on the input omics data and the output omics data. Medical information processing equipment.

2. the second mapping layer being the inverse of the first mapping layer; The medical information processing device according to claim 1 .

3. The loss function is the reconstruction loss between the input omics data and the output omics data. The medical information processing device according to claim 1 .

4. The CNN layer is an autoencoder layer. The medical information processing device according to claim 1 .

5. The first mapping layer is parameterized as a fully dense layer. The medical information processing device according to claim 1 .

6. the first mapping layer includes a one-to-one mapping; The medical information processing device according to claim 1 .

7. The processing circuitry further comprises training the CNN layer to reconstruct spatially ordered data prior to the training of the first mapping layer and the second mapping layer. The medical information processing device according to claim 1 .

8. The processing circuitry further trains a task-specific model to obtain a task-specific output from the ordered N-dimensional data obtained using the first mapping layer. The medical information processing device according to claim 1 .

9. The task-specific output comprises a classification or a prediction. The medical information processing device according to claim 8 .

10. the task-specific output comprises at least one of a disease classification, a phenotype classification, a classification of one or more disease features, a prediction of treatment response, a survival prediction, and a recurrence prediction; The medical information processing device according to claim 8 .

11. The first mapping layer is simultaneously or subsequently optimized based on errors resulting from the task performed by the task-specific model. The medical information processing device according to claim 1 .

12. The processing circuitry further comprises displaying the ordered N-dimensional data obtained using the first mapping layer. The medical information processing device according to claim 1 .

13. The processing circuitry further comprises: highlighting the ordered N-dimensional data region of data associated with one or more task-specific outputs. The medical information processing device according to claim 11.

14. the first mapping layer is initialized with domain knowledge; The medical information processing device according to claim 1 .

15. The training of the first mapping layer is initialized using a two-dimensional image format, the two-dimensional image format being: receiving omics data comprising a plurality of values, each of the plurality of values ​​being associated with a corresponding biomolecule of a plurality of biomolecules; calculating an individual distance for each biomolecule pair from the plurality of biomolecules; applying a manifold learning method to the distances to obtain a distinct position in two-dimensional space mapped to each of the plurality of biomolecules; adjusting the positions so that the positions are more evenly distributed in the two-dimensional space; and storing a two-dimensional image format of the display position of each of the plurality of biomolecules based on the adjusted positions; obtained by a method comprising: The medical information processing device according to claim 1 .

16. The input omics data and the output omics data include at least one of transcriptome data, proteome data, metabolome data, or genetic mutation data; The medical information processing device according to claim 1 .

17. obtaining a first mapping layer and a second mapping layer having updated parameters by being trained; The medical information processing device according to claim 1 .

18. Obtaining a Convolutional Neural Network (CNN) layer; training a first mapping layer coupled to an input layer of the CNN layers and a second mapping layer coupled to an output layer of the CNN layers, the first mapping layer converting input omics data representing genetic information into first N-dimensional data, the second mapping layer receiving second N-dimensional data output by the CNN layers and converting the second N-dimensional data into output omics data representing genetic information; The training of the first mapping layer and the second mapping layer may include minimizing a loss function at given parameters of the CNN layer, the loss function being based on the input omics data and the output omics data. Medical information processing method.

19. A processing circuit is provided, the processing circuit comprising: Obtain a trained first mapping layer that converts input omics data representing genetic information into first N-dimensional data, the first mapping layer being trained according to the medical information processing method of claim 18; converting an input omics dataset representing genetic information into a spatially ordered dataset using the trained first mapping layer; applying a task-specific model to the spatially ordered dataset to obtain a task-specific output; Medical information processing equipment.

20. The task-specific output comprises a classification or a prediction. The medical information processing device according to claim 19.

21. Obtaining a trained first mapping layer that converts input omics data representing genetic information into first N-dimensional data, the first mapping layer being trained according to the medical information processing method of claim 18; converting an input omics dataset representing genetic information into a spatially ordered dataset using the trained first mapping layer; applying a task-specific model to the spatially ordered dataset to obtain a task-specific output; A medical information processing method comprising:

Images