Evaluation method, visualization method, evaluation device, and visualization device

Abstract

One embodiment of the present invention provides an evaluation method and an evaluation device for evaluating biological robustness with respect to clusters and clusters of multiple omics data, and a visualization method and a visualization device for evaluation results. An evaluation method according to one embodiment of the present invention is a cluster evaluation method executed by an evaluation device provided with a processor, and the processor performs a process for acquiring cluster allocation information obtained by clustering implemented by an independent and arbitrary method for each group for multi-group data comprising two or more groups, and performing a process for performing cluster allocation on the cluster allocation information. The degree of matching between cluster allocations of different omics is calculated for each cluster on the basis of cluster allocation information, a first quantitative index indicating biological robustness of the cluster is calculated for each cluster on the basis of the degree of matching, and the calculated first quantitative index is output.

Description

Technical Field

[0001] The present invention relates to an evaluation method, a visualization method, an evaluation device and a visualization device, and in particular to a technology for evaluating clusters or clusters of omics data and a technology for visualizing evaluation results. Background Art

[0002] [Issues of clustering evaluation]

[0003] The clustering technique is a method of forming a collection (= called a cluster) of sample groups classified according to arbitrarily defined similarities, and is a method widely used as one of search methods based on data analysis.

[0004] Clustering is positioned as a type of unsupervised machine learning, and is usually performed without a teacher. Here, "with a teacher" means that the input data and correct answer information of "the input data of sample i is X, belonging to cluster K" are provided in advance. "Without a teacher", which mainly implements cluster analysis, is just the opposite, and becomes a problem setting of "classification is performed when only the input data X of sample i is provided, and it is unclear which cluster is correct." Below, in this article, it is set to implement clustering without a teacher.

[0005] The evaluation of the execution results is a problem in clustering without a teacher. Through evaluation, it is possible to verify whether the obtained clusters are appropriate results. Based on the evaluation results, the user implements (1) adjustment of setting values ​​(usually values ​​that need to be set by the user, such as hyperparameters in machine learning) or (2) selects an appropriate method from multiple clustering methods or algorithms.

[0006] In (1) above, it is known that even if the clustering algorithm is the same, the results will be very different if the setting values ​​are different. In (2), various clustering methods have been developed so far, and the method with good performance is usually different depending on the purpose or the scale and / or nature of the data. Therefore, evaluation is also important as a basis for appropriately selecting the method to be used and its setting values.

[0007] Evaluation in machine learning with a teacher is usually performed by calculating an index (called an external standard) based on the degree of match between the correct answer label and the predicted label. For example, for 5 samples, when the correct answer label {sample 1, sample 2, sample 3, sample 4, sample 5} = {A, B, A, C, C} is determined, the matching degree (correct answer rate) of the predicted label {A, B, B, A, C} becomes 3 / 5 = 0.6. However, since cluster analysis is set without a teacher, that is, no correct answer label is assigned, it is usually not easy to calculate the index based on the degree of match with the correct answer as described above.

[0008] Therefore, various evaluation indicators (called internal standards) for clustering results that do not use correct answer labels have been considered so far. As one example, an indicator based on the degree of agglomeration within the same cluster and the degree of divergence between different clusters can be cited. If applicable to the above-mentioned example, "the distance between sample 1 belonging to predicted cluster A and sample 4 belonging to the same predicted cluster A is small, and conversely, the distance to sample 2 belonging to predicted cluster B becomes larger" is considered to be a proper clustering. Here, the distance between samples is a value calculated based on the observed value.

[0009] Among these intrinsic criteria, there are reports that there are indicators that have achieved good correlation with external benchmarks through experiments based on artificially generated data. However, there are problems such as results in artificial data with limited conditions and the optimal intrinsic criteria differing depending on the object data, and there are still many issues in the evaluation of clustering results based on intrinsic criteria.

[0010] [Clustering of Omics Data]

[0011] First, the gene present in the living body is collectively referred to as the genome, and the data measuring the genome is called genome data. In addition, by establishing a corresponding association with a series of phenomena occurring in the living body, the concept can be expanded, and in addition to the genome, the epigenome, a general term for acquired control factors that are not accompanied by changes in gene sequences, the transcriptome, a general term for gene transcripts, and the proteome, a general term for proteins, are proposed. In recent years, there has been a view that these "general terms for living substances that exist at the same level in life phenomena" are summarized and called omics / omics data.

[0012] In the biomedical field, cluster analysis of various omics data is actively performed. For example, if the transcriptome data of cancer patients are clustered, the same cancer can be classified into subtypes with different properties, and attempts can be made to select high-precision treatments and / or medications suitable for each subtype. In addition, in recent years, the concept of multi-omics has emerged, and attempts are being made to model complex life phenomena more precisely by integrating and analyzing information obtained from multiple omics.

[0013] As a technique related to such clustering, for example, techniques described in Non-Patent Documents 1 to 3 are known.

[0014] [Technology of Non-Patent Document 1]

[0015] In Non-Patent Document 1, Lu et al. proposed “Robustness”, an evaluation index for the robustness of setting values ​​of a clustering algorithm. The “Robustness” mentioned here refers to the insensitivity of the results of the clustering algorithm to changes in the setting values.

[0016] In the “robustness”, clustering is performed with different settings (changing the number of clusters, etc.) for one algorithm, and the more common elements there are in the results, the higher the robustness. An example is given below.

[0017] (i) Clustering is performed on three samples using algorithm X under 10 different settings.

[0018] (ii) When focusing on sample a and sample b, the number of times a and b belong to the same cluster is set to 7. In this case, for (a, b), 7 / 10=0.7.

[0019] (iii) Similarly, in (b, c), it is set to 5 / 10=0.5, and in (c, a), it is set to 0.3.

[0020] (iv) At this time, the “robustness” of X becomes (0.7+0.5+0.3) / 3=0.5.

[0021] "Robustness" is also emphasized in the intrinsic criteria for algorithm selection for omics data cluster analysis. In particular, in the case of a data set without a teacher and with properties that are sensitive to set values, such as omics data cluster analysis, it is not easy to determine the appropriate set values. In this case, a method that is insensitive to set values ​​is preferred, which is achieved by selecting an algorithm with high "robustness".

[0022] In contrast, “robustness” is an indicator that assigns a score to an algorithm, and therefore cannot be used for the above-mentioned clustering issue “(1) Adjustment of set values”.

[0023] [Technology of Non-Patent Document 2]

[0024] Another problem in cluster analysis methods is cluster stability. Among clustering algorithms, there are algorithms that use random numbers in the generation of initial states. This algorithm does not guarantee that the cluster allocation obtained each time it is executed is equal. There is a high probability that there will not be a big difference in the evaluation value among the solutions obtained based on different initial states, so even if an appropriate evaluation index is used, it is not easy to determine which execution result is the best.

[0025] Therefore, there is a method called "consensus clustering" that extracts the common parts of multiple cluster assignments performed with randomly determined initial values ​​to derive the final cluster assignment. The above-mentioned non-patent document 2 is a representative method thereof, and is a method for obtaining reliable clusters.

[0026] [Formula 1]

[0027] The steps of “consensus clustering” are as follows:

[0028] 1. h = 1, 2, ..., h performs the following operations:

[0029] (a) From data d, D (h) Subsampling

[0030] (b) For D (h) Perform clustering and substitute M (h)

[0031] (c) Let M = M∪M (h)

[0032] 2. Calculate the consensus matrix M based on M:

[0033] (a) When the number of samples is N and (i, j) is the group of the i-th and j-th samples, the N×N dimensional matrix M storing the h-th clustering execution result (h) It is defined by the following formula (1).

[0034] [Formula 2]

[0035]

[0036] (b) In addition, the sub-sampled D in the storage step 1-(a) is defined by the following equation (2): (h) Whether the N×N dimension matrix I includes sample (i, j) (h) .

[0037] [Formula 3]

[0038]

[0039] (c) At this time, the consensus matrix M is expressed by the following formula (3).

[0040] [Formula 4]

[0041]

[0042] Finally, clustering is performed using the values ​​of the consensus matrix calculated in the above steps as the distances between samples to obtain

[0043] The final cluster allocation is obtained (where 1-M is converted).

[0044] [Technology of Non-Patent Document 3]

[0045] In Non-Patent Document 3, the "Consensus Clustering" of Non-Patent Document 2 is applied to multi-omics data.

[0046] First, clustering was performed in each omics (clustering algorithms were followed in previous studies) and the results were converted into binary vectors. Then, they were combined to generate a matrix of (sum of clusters in each omics) × number of samples.

[0047] Specifically, the case where the samples belonging to the first cluster of omics 1 are samples 1, 2, and 5 is expressed as "omics 1-c1 = [1, 1, 0, 0, 1]". Now, if there are three omics, the cluster assignments can be expressed as follows:

[0048] Omics 1-c1 = [1, 1, 0, 0, 1]

[0049] Omics 1-c2 = [0, 1, 0, 1, 0]

[0050] Omics 1-c3 = [1, 0, 0, 1, 0]

[0051] Omics 2-c1 = [1, 0, 1, 1, 1]

[0052] Omics2-c2=[0,1,1,0,0]

[0053] Omics 3-c1 = [1, 0, 0, 1, 1]

[0054] In the matrix combining these, each row is regarded as a "cluster feature". This matrix is ​​used as input to implement "Consensus Clustering" in Non-Patent Document 2. What is finally obtained is a "cluster of clusters" formed by merging similar objects in the clusters extracted from each omics.

[0055] [Clustering methods in the analysis of multi-omics data]

[0056] In addition to the above-mentioned non-patent documents 1 to 3, cluster analysis is also used in the analysis of multi-omics data. So far, a variety of clustering methods have been developed specifically for multi-omics data analysis. These are not techniques for intrinsic standards, but the effects obtained are equal in terms of obtaining more appropriate (especially from a biological point of view) clusters.

[0057] [Generalization of the concept of multi-omics]

[0058] In the field of machine learning, the concept of multi-omics is often referred to as "multi-view" or "multi-modality" (omics = view / modality). The characteristic of multi-view data, which includes multi-omics data, is that it has a data structure with signals composed of multiple different sources for one sample. For example, the group data of facial photos and personal information has two views, "image" and "text", and when the image and text are combined for processing, it becomes multi-view. Similarly, in omics data, there is data that measures gene expression data, methylation data, and protein expression data in each sample. In this case, various omics are equivalent to views, and when three are processed at the same time, it becomes multi-omics.

[0059] Previous technical literature

[0060] Non-patent literature

[0061] Non-patent document 1: “A robustness metric for biological data clustering algorithms”, Yuping Lu et al., [retrieved on September 9, 2022], Internet (https: / / pubmed.ncbi.nlm.nih.gov / 31874625 / )

[0062] Non-patent document 2: “Consensus Clustering: A Resampling-Based Method for Class Discovery and Visualization of Gene Expression Microarray Data”, Stefano Monti et al., [retrieved on September 9, 2022], Internet (https: / / link.springer.com / article / 10.1023 / A:1023949509487)

[0063] Non-patent document 3: "Multiplatform analysis of 12 cancer types reveals molecular classification within and across tissues of origin", Kathryn A Hoadley et al., [retrieved on September 9, 2022], Internet (https: / / pubmed.ncbi.nlm.nih.gov / 25109877 / ) Summary of the invention

[0064] Technical issues to be solved by the invention

[0065] [Problems of the prior art]

[0066] In the technology described in non-patent document 1, since the results of various settings are used for one algorithm, it is impossible to compare the setting values. For example, it cannot be used to determine the appropriate number of clusters (one of the setting values). In addition, in the technology described in non-patent document 1, it is impossible to evaluate the properties of the clusters. In order to evaluate the changes in multiple results, it is impossible to obtain information related to the obtained clusters. In addition, in the technology described in non-patent document 1, biological information is not considered. The information obtained in common for various settings has nothing to do with biological robustness.

[0067] Furthermore, in the technique described in non-patent document 2, multiple cluster assignments are output by repeated subsampling, so there are samples that are not used for clustering. Furthermore, in the technique described in non-patent document 2, since it is repeatedly performed on essentially the same sample population, the obtained cluster can only consider one omics. Therefore, in the case where there is a deviation in a specific omics, it will be affected by the deviation. Furthermore, in the technique described in non-patent document 2, biological information is not considered. Therefore, the information obtained from multiple subsamplings obtained from one omics is irrelevant to biological robustness.

[0068] Furthermore, in the technique described in Non-Patent Document 3, the results of clustering performed in each omics are stored as binary vectors and directly input into "Consensus Cluster", and in this method, no intrinsic criteria for the output clusters are provided. Therefore, although a score for "stability" proposed in "Consensus Clustering" can be obtained, this is a viewpoint on the non-determinism of the algorithm, and therefore there is no biological background.

[0069] [Issues in clustering of omics data]

[0070] In the cluster analysis of omics data, it is also performed with a non-teacher setting like the usual cluster analysis, and there are issues such as evaluation based on intrinsic benchmarks. In addition, as a characteristic of omics data, since it is a measurement of biological objects, it has the property of including noise from living body reactions (heterogeneity of cells, time changes based on dynamics) in addition to measurement-related noise. These methods that are not easily affected by noise or deviation and are biologically reasonable (hereinafter referred to as "biologically robust") have become issues in the clustering of omics data. However, the previous technologies shown in non-patent documents 1 to 3, etc. cannot fully cope with this issue.

[0071] The present invention is completed in view of the above situation. One embodiment of the present invention provides an evaluation method and an evaluation device capable of outputting an index representing the biological robustness of a cluster of multi-omics data. In addition, another embodiment of the present invention provides an evaluation method and an evaluation device capable of outputting an index representing the biological robustness of a clustering result of multi-omics data. In addition, another embodiment of the present invention provides a visualization method and a visualization device capable of visualizing the evaluation results based on these evaluation methods and evaluation devices.

[0072] Means for solving technical problems

[0073] The cluster evaluation method involved in the first embodiment of the present invention is a cluster evaluation method executed by an evaluation device equipped with a processor. In the evaluation method, the processor performs the following processing: for multi-omics data composed of more than two omics, cluster allocation information obtained by clustering implemented by an independent and arbitrary method for each omics is obtained, and the matching degree between the cluster allocations of different omics is calculated for each cluster based on the cluster allocation information; the first quantitative indicator representing the biological robustness of the cluster is calculated for each cluster based on the matching degree, and the calculated first quantitative indicator is output.

[0074] The evaluation method involved in the second embodiment is that in the first embodiment, the processor performs the following processing: in the calculation of the matching degree, for all pairs of samples belonging to each cluster, the number of times each sample pair belongs to the same cluster in the cluster allocation information of all clusters is counted for each omics pair; in all omics, the number of times counted for each omics pair is weighted by a specified weight; and the matching degree is calculated for each cluster based on the weighted result, and in the calculation of the first quantitative indicator, the first statistic for each cluster of the matching degree is calculated as the first quantitative indicator.

[0075] In the evaluation method according to the third aspect, in the second aspect, the processor performs processing for calculating one or more of a sum of the number of times weighting is performed, an average value, a median value, a minimum value, a maximum value, and a mode value as the first statistic.

[0076] In the evaluation method according to a fourth aspect, in the second or third aspect, the processor performs processing for weighting each omics by multiplying a designated constant by the counted number of times.

[0077] In the evaluation method according to the fifth aspect, in the fourth aspect, the processor performs a process of weighting by changing a constant multiplied by a part of the two or more omics and a constant multiplied by the remaining omics.

[0078] The evaluation method involved in the sixth aspect is any one of the second to fifth aspects, wherein the processor performs the following processing: weighting is performed when the counted results for any combination of omics are consistent with specified conditions and when the counted results for any combination of samples are consistent with specified conditions.

[0079] The evaluation method according to the seventh aspect is that in the first aspect, the processor performs the following processing: in calculating the matching degree, for each cluster of each omics, the similarity with all clusters belonging to different omics is calculated; and the first quantitative index is calculated using the second statistic for the similarity as the matching degree.

[0080] In the evaluation method according to the eighth aspect, in the seventh aspect, the processor performs processing for calculating one or more of a sum, an average value, a median value, a minimum value, a maximum value, and a mode value of similarities as the second statistic.

[0081] In the evaluation method according to the ninth aspect, in the seventh or eighth aspect, the processor performs processing for calculating any one of inner products, cosine similarity, Jaccard coefficient, Hamming distance, Dice coefficient, and correlation function between clusters as the similarity.

[0082] The evaluation method involved in the tenth embodiment is any one of the seventh to ninth embodiments, wherein the processor performs the following processing: in the calculation of similarity, when any combination of omics is consistent with the specified conditions and when any combination of samples is consistent with the specified conditions, each omics is weighted.

[0083] The evaluation method involved in the 11th method is an evaluation method for clustering results executed by an evaluation device equipped with a processor, wherein the processor performs the following processing: calculating a first quantitative indicator for each cluster through an evaluation method involved in any one of the 1st to 10th methods; calculating a third statistic of the first quantitative indicator; and outputting the third statistic as a second quantitative indicator representing the biological robustness of the clustering result.

[0084] In the evaluation method according to the twelfth aspect, in the eleventh aspect, the processor performs processing for calculating any one of a sum, an average value, a median value, a minimum value, a maximum value, and a mode value of the first quantitative index as the third statistic.

[0085] In addition, as an aspect of the present invention, there can be cited a program for causing a computer to execute the evaluation method according to any one of the first to twelfth aspects and a non-transitory tangible recording medium recording a computer-readable code of the program.

[0086] The visualization method involved in the 13th method is a visualization method of cluster evaluation performed by an evaluation device equipped with a processor, and the processor performs the following processing: for multi-omics data composed of more than two omics, obtaining cluster allocation information obtained from clustering implemented by an independent and arbitrary method for each omics; using a vector with samples belonging to each cluster as components to represent each cluster extracted from each omics; compressing the vector to a dimension less than three dimensions by an arbitrary dimensionality compression method; drawing each cluster on the coordinates in the compressed dimension; and associating the first quantitative indicator calculated by the evaluation method involved in any one of the first to 12th methods with each cluster and outputting it.

[0087] The visualization method involved in the 14th method is a visualization method of cluster evaluation performed by an evaluation device equipped with a processor, and the processor performs the following processing: based on the similarity calculated by the evaluation method involved in the 7th method, a graph structure with each cluster as a node and the similarity as an edge is constructed; and the constructed graph structure and the first quantitative indicator calculated by any one of the evaluation methods involved in the 1st to 10th methods are associated with each cluster and the first quantitative indicator in the graph structure and output.

[0088] In addition, as an aspect of the present invention, there can be cited a program for causing a computer to execute the visualization method according to the thirteenth or fourteenth aspect and a non-transitory tangible recording medium recording a computer-readable code of the program.

[0089] The evaluation device involved in the 15th method is a cluster evaluation device, which includes a processor, and the processor performs the following processing: for multi-omics data composed of more than two omics, obtain cluster allocation information obtained from clustering implemented by an independent and arbitrary method for each omics; calculate the matching degree between cluster allocations of different omics from the cluster allocation information; calculate the first quantitative indicator representing the biological robustness of the cluster for each cluster based on the matching degree; and output the calculated first quantitative indicator.

[0090] The evaluation device according to the fifteenth aspect may have a configuration for executing the same processing as the second to tenth aspects.

[0091] In the evaluation device involved in the 16th mode, the processor performs the following processing in the 15th mode: in the calculation of the matching degree, for all pairs of samples belonging to each cluster, the number of times each sample pair belongs to the same cluster in the cluster allocation information of all clusters is counted for each omics pair; in all omics, the number of times counted for each omics pair is weighted by a specified weight; and the matching degree is calculated for each cluster based on the weighted result, and in the calculation of the first quantitative indicator, the first statistic for each cluster of the matching degree is calculated as the first quantitative indicator.

[0092] In the evaluation device according to the 17th aspect, in the 15th aspect, the processor performs the following processing: in calculating the matching degree, for each cluster of each omics, the similarity with all clusters belonging to different omics is calculated; and the first quantitative index is calculated using the second statistic for the similarity as the matching degree.

[0093] The evaluation device involved in the 18th method is an evaluation device for clustering results, and the evaluation device includes a processor, and the processor performs the following processing: calculating the first quantitative indicator for each cluster through the evaluation device involved in any one of the 15th to 17th methods; calculating the third statistic of the first quantitative indicator; and outputting the third statistic as a second quantitative indicator representing the biological robustness of the clustering results.

[0094] The evaluation device according to the eighteenth aspect may have a configuration that executes the same processing as that of the twelfth aspect.

[0095] The visualization device involved in the 19th method is a visualization device for cluster evaluation, and the visualization device includes a processor, which performs the following processing: for multi-omics data composed of more than two omics, obtain cluster allocation information obtained from clustering implemented by an independent and arbitrary method for each omics; use a vector with samples belonging to each cluster as components to represent each cluster extracted from each omics; compress the vector into a dimension less than three dimensions by an arbitrary dimensionality compression method; draw each cluster on the coordinates in the compressed dimension; and associate the first quantitative indicator calculated by the evaluation device involved in any one of the 15th to 17th methods with each cluster and output it.

[0096] The visualization device involved in the 20th method is a visualization device for cluster evaluation, and the visualization device includes a processor, which performs the following processing: constructing a graph structure with each cluster as a node and the similarity as an edge based on the similarity calculated by the evaluation device involved in the 17th method; and associating the constructed graph structure and the first quantitative indicator calculated by any one of the evaluation devices involved in the 15th to 17th methods with each cluster and the first quantitative indicator in the graph structure and outputting them. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 It is a conceptual diagram showing the state of clustering of multi-omics data.

[0098] Figure 2 It is a diagram showing the configuration of an evaluation device according to the first embodiment.

[0099] Figure 3 It is a diagram showing the structure of a processing unit.

[0100] Figure 4 : is a flowchart showing the processing of the cluster evaluation method.

[0101] Figure 5 This is a diagram showing a state of calculating the first quantitative index.

[0102] Figure 6 This is a diagram showing a state in which the first quantitative index is calculated by weighting the counted number of times.

[0103] Figure 7 This is a diagram showing an example of index calculation taking into account the similarity between clusters.

[0104] Figure 8 Detailed description of the process of the clustering result evaluation method.

[0105] Fig. 9 It is a diagram showing the functional configuration of a processing unit according to the second embodiment.

[0106] Fig.10 is a flowchart showing the processing of the visualization method of cluster evaluation.

[0107] Fig.11 This is a diagram showing an example of visualization of cluster evaluation.

[0108] Fig.12 is another flowchart showing the processing of the visualization method of cluster evaluation.

[0109] Fig.13 This is a diagram showing another example of visualization of cluster evaluation. DETAILED DESCRIPTION

[0110] Hereinafter, embodiments of the evaluation method and evaluation device and the visualization method and visualization device according to the present invention will be described. In the description, the drawings will be referred to as needed.

[0111] [First embodiment]

[0112] [The concept of clustering of multi-omics data]

[0113] First, refer to Figure 1 A conceptual diagram of clustering of multi-omics data is shown in Figure 2. Figure 1 In the example of FIG. 1 , the sample population is composed of data on cells of a breast cancer patient, for example, and the state of the data is measured for three omics (refer to Figure 1 (Part (a)). Each omics data is high-dimensional data (for example, having thousands to hundreds of thousands of features for each sample).

[0114] exist Figure 1 In , it is assumed that a visualization based on dimensionality compression is applied on the basis of clustering (same as the visualization example described later). Figure 1The samples enclosed by circular marks in part (a) (for example, subtypes 600A to 600D) represent subtypes of breast cancer. Subtypes are used in subsequent steps, so it is preferable to appropriately determine them.

[0115] However, if, for example, one wants to determine subtypes based only on genome (omics 1), Figure 1 In the example, there are 4 subtypes, but when clustering is performed using different settings or methods, the subtypes may become 2 ( Figure 1 Subtypes 602A to 602B in part (b) of FIG.

[0116] Here, we focus on other omics. Even if they are different omics, since they are the measurement results of the same sample population 600, it is assumed that there are potentially common properties through omics. Therefore, it is expected that by specifying the clusters commonly obtained in different omics (omics 2: epigenome, omics 3: proteome), "biologically reasonable subtypes" can be defined.

[0117] [Structure of Evaluation Device]

[0118] Figure 2 1 is a diagram showing a schematic structure of an evaluation device 10 (evaluation device) according to a first embodiment of the present invention. Figure 2 As shown, the evaluation device 10 has a processing unit 100 (processor, computer), a storage unit 200, a display unit 300, and an operating unit 400, and these components are connected to each other to send and receive the required information. Regarding these components, various settings can be adopted. Each component can be set in one place (inside a frame, in a room, etc.), or it can be set in a separate position and connected via a network. In addition, the evaluation device 10 can be connected to an external server 500 and / or an external database 510 via a network NW such as the Internet, and can obtain multi-omics data, receive cluster allocation information, etc. as needed, and can store processing results, etc. in the external server 500 and / or the external database 510.

[0119] [Structure of the processing unit]

[0120] Figure 3 is a diagram showing the structure of the processing unit 100. Figure 3As shown, the processing unit 100 includes a processor 110 (processor, computer), a ROM 130 (ROM: Read Only Memory) and a RAM 140 (RAM: Random Access Memory). The processor 110 performs centralized control of the processing performed by each unit of the processing unit 100, and has the functions of an information acquisition unit 112, a matching degree calculation unit 114, a quantitative index calculation unit 116 and an output control unit 118. If these functions are briefly described, the information acquisition unit 112 (processor) acquires (receives or calculates) cluster allocation information, the matching degree calculation unit 114 (processor) calculates the matching degree between cluster allocations of different omics for each cluster based on the cluster allocation information, the quantitative index calculation unit 116 (processor) calculates the first quantitative index representing the biological robustness of the cluster for each cluster based on the matching degree, and the output control unit 118 (processor) outputs the calculated first quantitative index and the like to the storage unit 200 and / or the monitor 310.

[0121] The information acquisition unit 112 can receive multi-omics data or cluster allocation information from an external server 500 or an external database 510 and / or a recording medium such as a storage unit 200 via a network NW. The output control unit 118 can display the received or generated information or calculation results (first quantitative index, second quantitative index, etc.) on the monitor 310 and / or store them in the storage unit 200.

[0122] The functions of the processing unit 100 and the processor 110 can be implemented using various processors (processors) and recording media. Various processors include, for example, a general-purpose processor, i.e., a CPU (Central Processing Unit), which executes software (programs) to implement various functions. In addition, the various processors include a GPU (Graphics Processing Unit), which is a processor specifically used for image processing, and a programmable logic device (PLD), which is a processor whose circuit structure can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array). In the case of learning or recognizing an image, a structure using a GPU is effective. In addition, a dedicated circuit of a processor having a circuit structure specially designed for performing specific processing, such as an ASIC (Application Specific Integrated Circuit), is also included in the various processors mentioned above.

[0123] The functions of each part can be implemented by one processor, or by multiple processors of the same or different types (for example, multiple FPGAs, or a combination of CPU and FPGA, or a combination of CPU and GPU). In addition, multiple functions can also be implemented by one processor. As an example of multiple functions being composed of one processor, the first is the following method: as represented by a computer, one processor is composed of a combination of more than one CPU and software, and multiple functions are implemented by the processor. Second, there is the following method: as represented by a system on chip (System On Chip: SoC), a processor that implements the functions of the entire system is used by one IC (Integrated Circuit: Integrated Circuit) chip. In this way, for various functions, more than one of the above-mentioned various processors is used as a hardware structure. Moreover, more specifically, the hardware structure of these various processors is a circuit (circuitry) formed by combining circuit elements such as semiconductor elements. These circuits can also be circuits that use logical sum, logical AND, logical negation, exclusive OR and logical operations composed of their combinations to implement the above functions.

[0124] When the processor or circuit executes the software (program), the computer-readable code of the software to be executed (for example, various processors or circuits and / or their combination constituting the processing unit 100) is pre-stored in a non-temporary and tangible recording medium such as ROM130, and the software is referred to by the computer. The software stored in the non-temporary and tangible recording medium includes a program (evaluation program, visualization program) for executing the evaluation method and visualization method involved in the present invention and data used during execution (for example, multi-omics data, cluster allocation information, setting values ​​of hyperparameters required for executing the program, etc.). The code may also be recorded in various optical magnetic recording devices, semiconductor memories, and other recording media instead of recording the code in ROM130. When performing processing using software, for example, RAM140 is used as a temporary storage area, and data stored in recording media such as EEPROM (Electronically Erasable and Programmable Read Only Memory: Electrically Erasable and Programmable Read Only Memory) and flash memory, which are not shown, can also be referred to. The storage unit 200 may also be used as a "non-temporary and tangible recording medium".

[0125] The details of the processing performed by the processing unit 100 using the above-mentioned configuration will be described later.

[0126] [Structure of storage unit]

[0127] The storage unit 200 is composed of various storage devices such as an optical magnetic recording device, a hard disk, a semiconductor memory and their control units, and can store the above-mentioned multi-omics data, cluster allocation information, the first quantitative indicator, the second quantitative indicator, the matching degree between cluster allocations, the similarity between clusters, etc.

[0128] [Structure of display unit]

[0129] The display unit 300 includes a monitor 310 (display device) composed of a display such as a liquid crystal display, and can display input data, the execution results of the evaluation method (evaluation program) and the visualization method (visualization program), etc. The monitor 310 may also be composed of a touch panel display and receive user instruction input.

[0130] [Structure of the operation unit]

[0131] The operation unit 400 includes a keyboard 410 and a mouse 420, and the user can perform operations related to execution and result display of the evaluation method (evaluation program) and visualization method (visualization program) according to the present invention via the operation unit 400. The operation unit 400 may include other operation devices.

[0132] [Cluster evaluation method]

[0133] The cluster evaluation method executed by the evaluation device 10 (processor) will be described in detail.

[0134] [Get cluster allocation information]

[0135] Figure 4 is a flowchart showing the processing of the cluster evaluation method. Figure 5 As shown in part (a) of , the information acquisition unit 112 (processor) acquires cluster allocation information obtained from clustering performed independently and arbitrarily for each omics for multi-omics data composed of two or more omics (step S100: information acquisition process). In addition, in the present invention, the number of omics can be two or more, and there is no upper limit.

[0136] The “cluster assignment information” obtained in step S100 is the result of clustering the multi-omics data by an independent arbitrary method for each omics, such as Figure 5 As shown in part (b) of FIG. 1 , information on “in each omics, to which cluster each sample is assigned” is maintained. In addition, the information acquisition unit 112 may receive pre-calculated cluster assignment information from a recording device such as an external server 500 or an external database 510, or may receive multi-omics data from these recording devices and perform cluster assignment, thereby generating cluster assignment information.

[0137] [Calculation of matching degree]

[0138] The matching degree calculation unit 114 (processor) calculates the matching degree between the cluster assignments of different omics for each cluster based on the acquired cluster assignment information (step S110: matching degree calculation process). The matching degree calculation unit 114 counts the number of times all pairs of samples belonging to each sample belong to the same cluster in the cluster assignment information of all clusters, and weights the counted number of times according to a specified weight, and can calculate the matching degree for each cluster based on the weighted result.

[0139] [Formula 5]

[0140] The number of samples is N, the number of omics is H, and the number of clusters of omics h is K. (h) , the first cluster C of omics h i (h) The number of samples is set to L1 (h) For each group h = 1, 2, ..., H, the cluster allocation is set to the set of clusters represented by the following formula (4).

[0141] [Formula δ]

[0142]

[0143] Here, the first cluster is a collection of samples represented by the following formula (5).

[0144] [Formula 7]

[0145]

[0146] Among them, S 1，i (h) Number the sample

[0147] At this time, the matrix M indicating whether each sample belongs to the same cluster is defined by the following formula (6): (h) .

[0148] [Formula 8]

[0149]

[0150] When (i, j) is set as the group of i-th and j-th samples, the matrix M (h) It becomes an N×N dimensional matrix.

[0151] Thus, the matching degree calculation unit 114 can calculate the matching degree of cluster assignment in all omics by the following equation (7) (step S110 ).

[0152] [Formula 9]

[0153]

[0154] The matrix M(h) represents the number of times each sample belongs to the same cluster for each omics, and taking the sum (∑) of M(h) for h (=1 to H) means calculating the sum over all omics. In addition, in formula (7), the weight of each omics is set to be equal, that is, all are 1, but as described below, M(h) (the number of times counted) can be weighted by the specified weight.

[0155] Table 716 in part (c) of Fig. (5) is an example of expressing the matching degree (matrix M(h)). In addition, in table 716, the upper triangle and the lower triangle are symmetrical.

[0156] Furthermore, the quantitative index calculation unit 116 (processor) calculates the biological robustness R1 of each cluster by the following formula (8): (h) , a first quantitative index (an index representing the biological robustness of the cluster) (step S120: a first quantitative index calculation step). The biological robustness R1 (h) is the statistic of the matching degree for each cluster.

[0157] [Formula 10]

[0158]

[0159] Here, F(·) is a function that takes a set as an independent variable and returns a scalar. For the weighted order M(i, j), any statistics (first statistic) such as one or more sums, averages, medians, minimums, maximums, and most frequent values ​​can be used. Figure 5 Part (d) of and formula (8) show a calculation example using a sum function in F, but the quantitative index calculation unit 116 can also use other functions (other statistics) such as the average value of the matching degree to calculate the first quantitative index.

[0160] The quantitative index calculation unit 116 can determine which statistic to use to calculate the first quantitative index through the user's operation via the keyboard 410 and / or the mouse 420 (operation unit 400). The quantitative index calculation unit 116 may determine which statistic to use without depending on the user's operation.

[0161] The output control unit 118 (processor) outputs the calculated first quantitative index (step S130: index output process). The first quantitative index is an index representing the biological robustness of the cluster. The larger the value of the index, the higher the biological robustness of the cluster. The output control unit 118 can store the first quantitative index in the storage unit 200, or display it on the monitor 310 (display device). The display on the monitor 310 can be performed, for example, by using characters, numbers, graphics, symbols, or a combination thereof representing the index, and the color of the characters, numbers, graphics, symbols, etc. can also be changed according to the value of the index. In addition, the output control unit 118 preferably associates the cluster with the first quantitative index for the cluster and outputs it.

[0162] In the first embodiment, the biological robustness of the cluster of multi-omics data can be evaluated by using the first quantitative index. The same is true in the following embodiments.

[0163] [Formula 11]

[0164] [Count Weight]

[0165] In the above method, the weights of each group are set equal, that is, all are 1 (Formula

[0166] In (7), the weights of the matrix M(h) are all 1), but in the present invention,

[0167] The weight of the number of times can be changed according to the group and its combination. The following describes such a method.

[0168] Figure 6 : is a diagram showing a state where the first quantitative index is calculated by weighting the counted number of times. The information acquisition unit 112 (processor) acquires cluster allocation information of each omics (step S100: information acquisition step). Figure 6 Part (a) of FIG. 7 shows examples of cluster allocation information for omics 1 to 3 (Tables 720 to 724). In addition, in Tables 720 to 724 and Tables 726 and 728, the upper triangle is symmetrical to the lower triangle (with Figure 5 The same as Table 71δ).

[0169] [Weighting method 1]

[0170] In the above formula (7), the weights of the various groups are equal, that is, all are set to 1, but the weights of the various groups are changed in method 1. Specifically, the matching degree calculation unit 114 can perform weighting according to the following formula (9), for example.

[0171] [Formula 12]

[0172] M(i, j) = 3M (1) +M (2) +M (3) (9)

[0173] In formula (9), the matching degree calculation unit 114 performs weighting by multiplying the number of times a constant specified for each omics (3 for omics 1, 1 for omics 2 and 3) is counted. In this way, the matching degree calculation unit 114 can change the constant multiplied by a part of the omics among two or more omics and the constant multiplied by the remaining omics to perform weighting.

[0174] The weighted result of formula (9) is as follows: Figure 6 As shown in Table 726 shown in part (b1) of . Furthermore, if the weighting based on equation (9) is generalized, it can be expressed as the following equation (10). Here, w is an H-dimensional real number vector assigned to each omics (w1, wz, w3 are constants specified for each omics).

[0175] [Formula 13]

[0176]

[0177] The quantitative index calculation unit 116 can calculate the first quantitative index according to the weighted result through the above formula (8) (step S120: index calculation process), and the output control unit 118 can output the calculated first quantitative index (step S130: index output process). Regarding formula (8), as with the above content, for the statistic (first statistic) for the weighted result, one or more sums, averages, medians, minimum values, maximum values, and most frequent values ​​can also be used. Which statistic is used is equivalent to which function F is used in formula (8) (the same applies to the following methods).

[0178] [Weighting method 2]

[0179] In the method 2, as shown in the following formula (11), the matching degree calculation unit 114 multiplies the weight by 3 times only when the results of omics 1 and omics 2 are consistent.

[0180] [Formula 14]

[0181]

[0182] The condition "when the results of group 1 and group 2 are consistent" or the value "3 times" is an example of weighting, and other conditions and values ​​can also be used. The result of weighting by formula (11) is as follows Figure 6 Table 728 shown in part (b2) of . In the second embodiment, the calculation and output of the first quantitative index can be performed by equation (8) in the same manner as in the first embodiment.

[0183] [Other ways of weighting]

[0184] The above-mentioned method 1 and method 2 are examples of weighting. In addition to these, the matching degree calculation unit 114 can also perform weighting by "taking the matching of the results in more than 3 groups as a condition" (a method of "weighting when the counting results for any combination of groups match the specified conditions") or "weighting only when the results of a specific sample pair match in a specific group" (a method of "weighting when the counting results for any combination of samples match the specified conditions"). Moreover, the weighting method can also be "addition of weights" other than multiplication by weights. Taking the quotient and the difference can be achieved by setting a weight less than 1 or less than 0.

[0185] [Calculation method using the statistics of similarity between clusters as an indicator]

[0186] In the present invention, the matching degree calculation unit 114 (processor) calculates the similarity between each cluster of each group and all clusters belonging to different groups in the calculation of the matching degree, and the quantitative index calculation unit 116 (processor) can calculate the first quantitative index using the second statistic for the similarity as the matching degree. Figure 7 This is a diagram showing an example of index calculation taking into account the similarity between clusters.

[0187] The information acquisition unit 112 can communicate with Figure 4 Step S100 similarly obtains cluster allocation information ( Figure 7 The matching degree calculation unit 114 calculates the similarity between each cluster of each omics and all clusters belonging to different omics. The matching degree calculation unit 114 can calculate, for example, the inner product, cosine similarity, Jaccard coefficient, Hamming distance, Dice coefficient, and any related function between clusters as the similarity, but the similarity is not limited to these examples.

[0188] [Formula 15]

[0189] Figure 7 Part (c) shows the calculation of C3 (1) (Cluster 3 and C2 of Omics 1 (2) The result of the Jaccard coefficient (an example of similarity) of (Cluster 2 of Omics 2). In this case, similarity = 0.75. In addition, Figure 7 Part (e) is a table showing the results of calculating the Jaccard coefficient for omics 1 to 3 and clusters 1 to 3.

[0190] The quantitative index calculation unit 116 calculates the first quantitative index using the second statistic for the similarity calculated in this way as the degree of matching. Figure 7 In the example shown in part (f) of FIG. 1 , the maximum value of the similarity (an example of the second statistic) is set as the matching degree.(1) (Cluster 1 of Omics 1), 1.0, which is the maximum value (an example of the second statistic) of {0.0, 0.0, 1.0, 0.67, 0.0, 0.2}, is the matching degree, and similarly, for C1 (3) (For cluster 1 of omics 3, 0.67, which is the maximum value (an example of the second statistic) of {0.67, 0.0, 0.0, 0.0, 0.67}, is the matching degree. The quantitative index calculation unit 116 calculates the matching degree as the first quantitative index representing the biological robustness of the cluster.

[0191] A large value of the first quantitative index indicates that the biological robustness of the cluster is high. In the above example, the robustness of cluster 1 of omics 1 (1.0) is higher than the robustness of cluster 1 of omics 3 (0.67).

[0192] [Weighting of each group in similarity calculation]

[0193] The matching degree calculation unit 114 may weight each group of studies in the calculation of the similarity described above. Figure 7 As described in part (d) of Figure 7 In the example of , in the pair of omics 1 and omics 3), when common noise is known, overfitting to the noise can be prevented by multiplying by a weight less than 1. Figure 6 In the example described in part (b2), when it is determined that the match in a specific omics pair (in the example of this part, the pair of omics 1 and omics 2) increases reliability, a weight greater than 1 can be multiplied or added.

[0194] [Evaluation method of clustering results]

[0195] In the above cluster evaluation method, one index (first quantitative index) is assigned to each cluster. In the present invention, as described below, by calculating one or more statistics for the first quantitative index for each cluster, a quantitative index of biological robustness of the clustering result can be obtained.

[0196] Figure 8 This is a flowchart showing the processing of the clustering result evaluation method. The cluster evaluation method is similar to the above content, and each part of the processing unit 100 (processor) calculates the first quantitative index for each cluster (step S200: information acquisition process, matching degree calculation process, index calculation process, index output process). Figure 4 Steps S100 to S120 are the same as the above-mentioned processing, and detailed description of the processing of step S200 is omitted. Note that in the processing of step S200, any variation of the above-mentioned variations of steps S100 to S120 may be used.

[0197] The quantitative index calculation unit 116 (processor) calculates any one of the sum, average value, median value, minimum value, maximum value and mode value of the first quantitative index as the third statistic (step S210: statistic calculation process). The quantitative index calculation unit 116 can automatically determine which statistic to calculate based on the user operation via the operation unit 400 or independently of the user operation. The output control unit 118 (processor) outputs the third statistic as the second quantitative index representing the biological robustness of the clustering result (i.e., the clustering method or the clustering setting value group) (step S220: index output process).

[0198] [Second embodiment]

[0199] [Visualization of cluster evaluation (method 1)]

[0200] In the present invention, cluster evaluation (evaluation of biological robustness of clusters or cluster allocation) can be visualized using the index described above in the first embodiment. Fig. 9 1 is a diagram showing a configuration of a processing unit 100A (processor) in an evaluation device 10 (evaluation device) according to a second embodiment. Figure 3 ), the processing unit 100A further includes a visualization unit 120 (processor). The other structures of the evaluation device 10 according to the second embodiment are the same as those of the first embodiment.

[0201] Fig.10 is a flowchart showing the processing of the visualization method (method 1) for cluster evaluation. Fig.11 is a diagram showing the visualization of cluster evaluation. Fig.10 In the flowchart of , the calculation of the first quantitative index is the same as that of the first embodiment (refer to Figure 4 Steps S100 to S120), and the information acquisition unit 112 acquires cluster allocation information.

[0202] [Formula 16]

[0203] In the visualization of cluster evaluation, the visualization unit 120 (processor) represents each cluster extracted by each omics as a vector having samples belonging to the cluster as components (step S300: vectorization process). Fig.11 As shown in part (a) of (Table 720), C1 (1) The case of (cluster 1 of omics 1) = (1, 1, 1, 0, 0, 0, 0) indicates that samples A, B, and C are included in cluster 1. (1) In addition, if the arrangement is regarded as a vector, the matching degree calculation unit 114 (processor) can calculate the similarity between the two clusters based on the distance of the vector (cosine distance, Hamming distance, etc.).

[0204] The visualization unit 120 compresses the obtained vectors into three dimensions or less by any dimensional compression method (step S310: dimensional compression process), and draws each cluster on the coordinates in the compressed dimension (step S320: drawing process). The output control unit 118 also associates the first quantitative index calculated by the same method as the first embodiment with each drawn cluster and outputs it (step S330: output process).

[0205] Fig.11 Part (b) of is a diagram showing the state where the above vector is compressed into two dimensions by principal component analysis. Fig.11 It can be seen that each point represents a cluster, and similar clusters are plotted on approximate coordinates. Principal component analysis is a method of compressing multidimensional data (here, vectors) into a low dimension while preserving information as much as possible. In addition, in addition to principal component analysis, there are also dimensionality compression methods such as UMAP (UMAP: Uniform Manifold Approximation and Projection) or t-SNE (t-SNE: t-Distributed Stochastic Neighbor Embedding). The visualization unit 120 preferably selects a dimensionality compression method that is suitable for the nature of the data. In addition, the visualization unit 120 can select a dimensionality compression method based on the user's operation via the operating unit 400.

[0206] Fig.11 Part (c) of FIG. 7 is a table (Table 722) showing the first quantitative index for each cluster of each omics. Fig.11 In part (b), the first quantitative index included in the table is displayed near the drawing of each cluster (an example of a correlation output). The output control unit 118 can display the drawing result on the monitor 310 (display device) or store it in the storage unit 200. The drawing result can also be output through a printer (not shown). Through this visualization, the user can easily understand the biological robustness of the cluster.

[0207] [Visualization of cluster evaluation (method 2)]

[0208] Fig.12 is a flowchart showing the processing of the visualization method (method 2) for cluster evaluation. Fig.13 is a diagram showing the visualization of cluster evaluation. Fig.12 In the flowchart of the first embodiment, each unit of the processing unit 100 (processor) can calculate the first quantitative index (reference Figure 4 Steps S100 to S120; Step S410). And, Figure 7Similarly to the above-mentioned contents, each unit of the processing unit 100 (processor) can calculate the similarity (step S400).

[0209] The visualization unit 120 constructs a graph structure with each cluster as a node and the similarity as an edge based on the similarity calculated in step S400 (step S420: graph structure construction process), and the output control unit 118 associates the constructed graph structure and the first quantitative indicator with each cluster and the first quantitative indicator in the graph structure and outputs them (step S430: output process).

[0210] Fig.13 This is a diagram showing the state of visualization based on method 2. Fig.13 Part (a) of is a table showing similarity (Table 718; same as Figure 7 (same as part (e) of Fig.13 (b) is a diagram showing the graph structure. In this diagram, outputs are made near each cluster. Fig.13 The first quantitative indicator shown in part (c) of Fig.11 In addition, in the diagram structure ( Fig.13 In part (b) of FIG. 1 , the thickness of the edge indicates the magnitude of the similarity. The output control unit 118 may implement the following emphasis processing in the diagram of the chart structure: changing the size of the symbol (in this diagram, the ellipse) representing the cluster according to the size of the first quantitative indicator, changing the line type or color of the symbol, or changing the line type or color of the edge according to the magnitude of the similarity. Furthermore, the output control unit 118 may display the constructed diagram of the chart structure on the monitor 310 (display device), or may store it in the storage unit 200. The diagram may be output via a printer not shown in the figure. Through this visualization, the user can also easily grasp the biological robustness of the cluster.

[0211] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0212] Explanation of symbols

[0213] 10-evaluation device, 100-processing unit, 100A-processing unit, 110-processor, 112-information acquisition unit, 114-matching degree calculation unit, 116-quantitative index calculation unit, 118-output control unit, 120-visualization unit, 130-ROM, 140-RAM, 200-storage unit, 300-display unit, 310-monitor, 400-operation unit, 410-keyboard, 420-mouse, 500-external server, 510-external database, 600-sample population, 600A-subtype, 600B-subtype, 600C-subtype, 600D-subtype, 602A-subtype, 602B-subtype, NW-network, S100~S220-each step of the evaluation method, S300~S330-each step of the visualization method, S400~S430-each step of the visualization method.

Claims

1. A cluster evaluation method, the method being performed by an evaluation device having a processor, wherein: The processor performs the following processing: For multi-omics data consisting of two or more omics, obtaining cluster assignment information obtained by clustering each omics using an independent and arbitrary method; Calculating the matching degree between the cluster assignments of different omics for each cluster according to the cluster assignment information; Calculating a first quantitative indicator representing the biological robustness of the cluster for each cluster according to the matching degree; and The calculated first quantitative index is outputted.

2. The evaluation method according to claim 1, wherein: The processor performs the following processing: In the calculation of the matching degree, For all pairs of samples belonging to each cluster, count the number of times each sample pair belongs to the same cluster in the cluster assignment information of all clusters for each omics pair; Across all omics, weighting the number of times counted for each omics by the specified weight; and Calculating the matching degree for each cluster according to the weighted result, In the calculation of the first quantitative index, A first statistic for each cluster of the matching degree is calculated as the first quantitative indicator.

3. The evaluation method according to claim 2, wherein: The processor performs processing to calculate one or more of a sum, an average, a median, a minimum, a maximum, and a mode value of the weighted counts as the first statistic.

4. The evaluation method according to claim 2 or 3, wherein: The processor performs the following processing: weighting each group of omics by multiplying a designated constant by the counted number of times.

5. The evaluation method according to claim 4, wherein: The processor performs a process of changing a constant by which a part of the two or more omics is multiplied and a constant by which the remaining omics are multiplied to perform the weighting.

6. The evaluation method according to claim 2 or 3, wherein: The processor performs the following processing: The weighting is performed when the result of the counting for any combination of omics matches the specified condition and when the result of the counting for any combination of samples matches the specified condition.

7. The evaluation method according to claim 1, wherein: The processor performs the following processing: In the calculation of the matching degree, for each cluster of each omics, the similarity with all clusters belonging to different omics is calculated; and The first quantitative index is calculated using a second statistic for the similarity as the matching degree.

8. The evaluation method according to claim 7, wherein: The processor performs the following processing: calculating one or more of the sum, average value, median value, minimum value, maximum value, and mode value of the similarities as the second statistic.

9. The evaluation method according to claim 7 or 8, wherein: The processor performs the following processing: calculating any one of the inner product, cosine similarity, Jaccard coefficient, Hamming distance, Dyce coefficient and related functions between the clusters as the similarity.

10. The evaluation method according to claim 7 or 8, wherein: The processor performs the following processing: In the calculation of the similarity, When any combination of omics matches the specified conditions and any combination of samples matches the specified conditions, weights are applied to the omics.

11. An evaluation method, which is a clustering result evaluation method executed by an evaluation device having a processor, wherein: The processor performs the following processing: Calculating the first quantitative index for each cluster by the evaluation method according to any one of claims 1, 2, 3, 7 and 8; Calculating a third statistic of the first quantitative indicator; and The third statistic is output as a second quantitative indicator representing biological robustness of the clustering result.

12. The evaluation method according to claim 11, wherein: The processor performs the following processing: calculating any one of the sum, average value, median value, minimum value, maximum value, and mode value of the first quantitative index as the third statistic.

13. A visualization method, which is a visualization method of cluster evaluation performed by an evaluation device having a processor, wherein: The processor performs the following processing: For multi-omics data consisting of two or more omics, cluster assignment information obtained from clustering performed by an independent and arbitrary method for each omics is obtained; Each cluster extracted by each omics is represented by a vector with samples belonging to each cluster as components; Compressing the vector into a dimension less than three dimensions by any dimensionality compression method; plotting each cluster on the compressed coordinates in the dimension; and The first quantitative index calculated by the evaluation method according to any one of claims 1, 2, 3, 7 and 8 is associated with each cluster and output.

14. A visualization method, which is a visualization method of cluster evaluation performed by an evaluation device having a processor, wherein: The processor performs the following processing: Based on the similarity calculated by the evaluation method according to claim 7, constructing a graph structure with each cluster as a node and the similarity as an edge; and The constructed graph structure and the first quantitative index calculated by the evaluation method according to any one of claims 1, 2, 3, 7 and 8 are associated with each cluster in the graph structure and the first quantitative index, and are output.

15. A cluster evaluation device, the evaluation device comprising a processor, The processor performs the following processing: For multi-omics data consisting of two or more omics, obtaining cluster assignment information obtained from clustering performed by an independent and arbitrary method for each omics; Calculating the matching degree between the cluster assignments of different omics according to the cluster assignment information; Calculating a first quantitative indicator representing the biological robustness of the cluster for each cluster according to the matching degree; and The calculated first quantitative index is outputted.

16. The evaluation device according to claim 15, wherein: The processor performs the following processing: In the calculation of the matching degree, For all pairs of samples belonging to each cluster, count the number of times each sample pair belongs to the same cluster in the cluster assignment information of all clusters for each omics pair; Across all omics, weighting the number of times counted for each omics by the specified weight; and Calculating the matching degree for each cluster according to the weighted result, In the calculation of the first quantitative index, A first statistic for each cluster of the matching degree is calculated as the first quantitative indicator.

17. The evaluation device according to claim 15, wherein: The processor performs the following processing: In the calculation of the matching degree, for each cluster of each omics, the similarity with all clusters belonging to different omics is calculated; and The first quantitative index is calculated using a second statistic for the similarity as the matching degree.

18. An evaluation device, which is a clustering result evaluation device, comprising a processor, The processor performs the following processing: Calculating the first quantitative index for each cluster by the evaluation device according to any one of claims 15 to 17; Calculating a third statistic of the first quantitative indicator; and The third statistic is output as a second quantitative indicator representing biological robustness of the clustering result.

19. A visualization device, which is a visualization device for cluster evaluation, comprising a processor, The processor performs the following processing: For multi-omics data consisting of two or more omics, obtaining cluster assignment information obtained from clustering performed by an independent and arbitrary method for each omics; Each cluster extracted by each omics is represented by a vector with samples belonging to each cluster as components; Compressing the vector into a dimension less than three dimensions by any dimensionality compression method; plotting each cluster on the compressed coordinates in the dimension; and The first quantitative index calculated by the evaluation device according to any one of claims 15 to 17 is associated with each cluster and output.

20. A visualization device, which is a visualization device for cluster evaluation, comprising a processor, The processor performs the following processing: Based on the similarity calculated by the evaluation device according to claim 17, constructing a graph structure with each cluster as a node and the similarity as an edge; and The constructed graph structure and the first quantitative index calculated by the evaluation device according to any one of claims 15 to 17 are associated with each cluster in the graph structure and the first quantitative index, and are output.

Images