Method for constructing a clinical therapeutic drug recommendation model under a comorbid state

Abstract

The present invention provides a method for constructing a clinical treatment drug recommendation model in a comorbid state, including: obtaining a data set, where each data contains diagnostic symptom information, treatment drug information, and patient information; respectively extracting diagnostic symptom entities, treatment drug entities, and patient entities from each data; dividing each data in the data set into a training set and a test set; establishing a patient-variable matrix according to each data in the training set; using different recommendation strategies to respectively learn the patient-variable matrix to obtain an entity embedding matrix; constructing a treatment drug scoring model according to the entity embedding matrix; inputting the symptom diagnosis entities of each data in the test set into the treatment drug scoring model to obtain corresponding treatment drug recommendation results; selecting an optimal recommendation strategy according to the treatment drug information of each data in the test set and each treatment drug recommendation result; and determining a treatment drug recommendation model according to the optimal recommendation strategy. The present invention constructs a clinical treatment drug recommendation model in a comorbid state.

Description

Technical Field

[0001] The present invention relates to the field of comorbidity treatment, and particularly to a method for constructing a clinical treatment drug recommendation model in a comorbid state. Background Art

[0002] Comorbidity involves multiple drug uses in the process of clinical treatment, and multiple drug uses may cause harm to comorbid patients. Therefore, how to reasonably treat comorbid patients has become a medical and health problem that the whole society pays increasing attention to.

[0003] At present, many scholars have achieved multi-faceted research by constructing disease-related networks and combining network analysis techniques. Existing research on constructing disease-related networks includes constructing disease phenotype networks and constructing multi-layer disease-related networks using network analysis techniques.

[0004] However, when constructing the network, most of the research on routinely collecting health data constructs the network based on various complex comorbidities and does not incorporate various treatment drugs for analysis.

[0005] At present, the research on treatment drug recommendation models is relatively limited. Existing treatment drug recommendation systems include constructing treatment drug recommendation systems using a technical framework based on rules (defined by clinical guidelines and medical experts). The time cost and labor cost for creating, managing, updating, and maintaining the rules are relatively high, and the rule-based model may only be effective for general medical advice for specific diagnoses and has little effect on personalized recommendations for the diagnosis and treatment decisions of comorbid patients. Summary of the Invention

[0006] The first aspect of the present invention provides a method for constructing a clinical treatment drug recommendation model in a comorbid state, including the following steps: obtaining a data set, where the data set includes multiple pieces of data, and the data contains diagnostic symptom information, treatment drug information, and patient information, and the treatment drug information includes clinical actual medication information; respectively extracting diagnostic symptom entities, treatment drug entities, and patient entities from each piece of data; dividing each piece of data in the data set into a training set and a test set; establishing a patient-variable matrix according to the diagnostic symptom entities, treatment drug entities, and patient entities in each piece of data in the training set; using different recommendation strategies to respectively learn the patient-variable matrix to obtain entity embedding matrices corresponding to each recommendation strategy; constructing treatment drug scoring models corresponding to each recommendation strategy according to the entity embedding matrices corresponding to each recommendation strategy; inputting the symptoms and diagnostic entities of each piece of data in the test set into the treatment drug scoring models to obtain treatment drug recommendation results corresponding to each recommendation strategy; selecting the optimal recommendation strategy according to the clinical actual medication information of each piece of data in the test set and the treatment drug recommendation results obtained by the treatment drug scoring models corresponding to each recommendation strategy; determining the treatment drug recommendation model according to the optimal recommendation strategy.

[0007] The beneficial effects are as follows: In the present invention, each data in the dataset is divided into a training set and a test set. A patient-variable matrix is established based on each data in the training set. Different recommendation strategies are used to learn the patient-variable matrix to obtain an entity embedding matrix corresponding to each recommendation strategy. Then, the data in the test set are respectively imported into the entity embedding matrix corresponding to each recommendation strategy to obtain the treatment drug recommendation results corresponding to each recommendation strategy. The optimal recommendation strategy is selected according to the treatment drug recommendation results corresponding to each data in the test set and the clinical actual medication information in each data. Finally, a treatment drug recommendation model is formed according to the optimal recommendation strategy. In the prior art, when constructing a network, a network of complex comorbidities is constructed, and various treatment drugs are not incorporated for analysis. The treatment drug recommendation model constructed in the present invention, in the process of construction, establishes a patient-variable matrix based on the diagnostic symptom entities, treatment drug entities, and patient entities in each data in the training set, realizing the combination of network construction and treatment drugs. The present invention selects the optimal recommendation strategy according to the treatment drug recommendation results corresponding to each data in the test set and the clinical actual medication information in each data, and forms a treatment drug recommendation model according to the optimal recommendation strategy, further improving the accuracy of treatment drug recommendation.

[0008] Combined with the first aspect, in the first implementation manner of the first aspect, the recommendation strategy includes a first recommendation strategy. The entity embedding matrix corresponding to the first recommendation strategy is obtained through the following steps: The patient-variable matrix is constructed into a similarity coefficient matrix, and the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities; the similarity coefficient matrix is transformed into an edge list structure to form the first edge list data corresponding to the first comorbidity diagnosis and treatment network; the first edge list data is subjected to representation learning through a large-scale information network embedding method to obtain the entity embedding matrix corresponding to the first recommendation strategy.

[0009] The beneficial effects are as follows: The first recommendation strategy includes the relationships between diagnostic symptom entities and treatment drug entities, the relationships between the same-kind entities of diagnostic symptom entities, and the relationships between the same-kind entities of treatment drug entities. The first treatment drug recommendation model obtained through the first recommendation strategy fully reflects the relationships between diagnostic symptom entities, treatment drug entities, and each entity.

[0010] Combined with the first aspect, in the second implementation manner of the first aspect, the recommendation strategy includes a second recommendation strategy. The entity embedding matrix corresponding to the second recommendation strategy is obtained through the following steps: The patient-variable matrix is constructed into a similarity coefficient matrix, and the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities; the similarity coefficient matrix is transformed into an edge list structure, and all edges between the same-kind entities of diagnostic symptom entities and all edges between the same-kind entities of treatment drug entities are deleted to form the second edge list data corresponding to the second comorbidity diagnosis and treatment network; the second edge list data is subjected to representation learning through a large-scale information network embedding method to obtain the entity embedding matrix corresponding to the second recommendation strategy.

[0011] The beneficial effects are as follows: The second recommendation strategy only includes the relationship between the diagnostic symptom entity and the treatment drug entity. The second treatment drug recommendation model obtained through the second recommendation strategy fully demonstrates the relationship between the diagnostic symptom entity and the treatment drug entity.

[0012] Combined with the first aspect, in the third implementation manner of the first aspect, the recommendation strategy includes a third recommendation strategy. The entity embedding matrix corresponding to the third recommendation strategy is obtained through the following steps: Construct the patient-variable matrix into a similarity coefficient matrix, where the similarity coefficient matrix includes the diagnostic symptom entity and the treatment drug entity; Convert the similarity coefficient matrix into an edge list structure, and add the two-hop relationship data between the diagnostic symptom entity and the treatment drug entity mediated by the patient entity in the edge list structure to form the corresponding third edge list data of the third comorbidity diagnosis and treatment network; Perform representation learning on the third edge list data through the large-scale information network embedding method to obtain the entity embedding matrix corresponding to the third recommendation strategy.

[0013] The beneficial effects are as follows: The third recommendation strategy not only includes the relationship between the diagnostic symptom entity and the treatment drug entity, the relationship between the same-kind entities of the diagnostic symptom entity, and the relationship between the same-kind entities of the treatment drug entity, but also adds the two-hop relationship between the diagnostic symptom entity and the treatment drug entity mediated by the patient entity. The treatment drugs recommended by the obtained third treatment drug recommendation model are more comprehensive.

[0014] Combined with the first aspect or the second implementation manner of the first aspect, in the fourth implementation manner of the first aspect, the recommendation strategy includes a fourth recommendation strategy. The entity embedding matrix corresponding to the fourth recommendation strategy is obtained through the following steps: Construct the patient-variable matrix into a similarity coefficient matrix, where the similarity coefficient matrix includes the diagnostic symptom entity and the treatment drug entity; Convert the similarity coefficient matrix into an edge list structure, delete all the edges between the same-kind entities of the diagnostic symptom entity and all the edges between the same-kind entities of the treatment drug entity, and add the two-hop relationship data between the diagnostic symptom entity and the treatment drug entity mediated by the patient entity to form the corresponding fourth edge list data of the fourth comorbidity diagnosis and treatment network; Perform representation learning on the fourth edge list data through the large-scale information network embedding method to obtain the entity embedding matrix corresponding to the fourth recommendation strategy.

[0015] The beneficial effects are as follows: The fourth recommendation strategy includes the two-hop relationship between the diagnostic symptom entity and the treatment drug entity mediated by the patient entity. The obtained fourth treatment drug recommendation model focuses more on the relationship between the patient entity, the diagnostic symptom entity, and the treatment drug entity.

[0016] Combined with the fourth embodiment of the first aspect, in the fifth embodiment of the first aspect, the recommendation strategy includes a fifth recommendation strategy. The entity embedding matrix corresponding to the fifth recommendation strategy is obtained through the following steps: Construct the patient-variable matrix into a similarity coefficient matrix, where the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities; Convert the similarity coefficient matrix into an edge list structure, delete all edges between entities of the same type of diagnostic symptom entities and all edges between entities of the same type of treatment drug entities, add two-hop relationship data between diagnostic symptom entities and treatment drug entities mediated by patient entities, and delete the edges between treatment drug entities and diagnostic symptom entities to form the fifth edge list data corresponding to the fifth comorbidity diagnosis and treatment network; Perform representation learning on the fifth edge list data through a large-scale information network embedding method to obtain the entity embedding matrix corresponding to the fifth recommendation strategy.

[0017] The beneficial effect is that the fifth recommendation strategy includes the relationships between patient entities and treatment drug entities and diagnostic symptom entities respectively, and the obtained fifth treatment drug recommendation model focuses more on the relationships between patient entities and treatment drug entities and diagnostic symptom entities respectively.

[0018] Combined with the first aspect, in the sixth embodiment of the first aspect, the recommendation strategy includes a sixth recommendation strategy. The entity embedding matrix corresponding to the sixth recommendation strategy is obtained through the following steps: Perform representation learning on the patient-variable matrix through matrix factorization to obtain the entity embedding matrix corresponding to the sixth recommendation strategy.

[0019] The beneficial effect is that the matrix factorization method cannot meet the needs of complex representation learning tasks. When there are many entity types and the relationships between entities are complex, constructing a patient-variable matrix can be used as a technical means to represent entity types and complex relationships between entities.

[0020] Combined with the first aspect, in the seventh embodiment of the first aspect, according to the intersection and union of the treatment drug recommendation results corresponding to each data in the test set and the clinical actual medication information in each data, calculate the accuracy rate of the treatment drug recommendation results of each recommendation strategy respectively; According to the ranking of the treatment drugs in the clinical actual medication information in the treatment drug recommendation results, calculate the hit rate and the average ranking of treatment drug recommendations corresponding to each strategy respectively. The hit rate is the hit rate of each treatment drug in the clinical actual medication information in the treatment drug recommendation results, and the average ranking of treatment drug recommendations is the average value of the rankings of each treatment drug in the clinical actual medication information in the treatment drug recommendation results; Select the optimal strategy according to the accuracy rate, hit rate, and average ranking of treatment drug recommendations.

[0021] The beneficial effect is that according to the three evaluation indicators of accuracy rate, hit rate, and average ranking of treatment drug recommendations, the recommendation strategy with the best recommendation effect can be selected.

[0022] In a second aspect of the present invention, a method for recommending therapeutic drugs for patients with comorbidities is provided, including: obtaining patient information and diagnostic symptom information of a target patient; inputting the patient information and diagnostic symptom information into a pre-trained therapeutic drug recommendation model to obtain recommended therapeutic drug information, where the therapeutic drug recommendation model is constructed based on the method for constructing a clinical therapeutic drug recommendation model in the comorbid state according to any one of the first aspect and its optional embodiments.

[0023] Combined with the second aspect, in the first embodiment of the second aspect, the therapeutic drug information includes multiple therapeutic drugs, and the method further includes: determining a therapeutic drug recommendation index according to the total number of recommended therapeutic drugs and the ranking of the recommended therapeutic drugs.

[0024] The beneficial effect is that the obtained therapeutic drug information of the present invention includes multiple therapeutic drugs. However, for patients, not every drug is needed, and the therapeutic effects of each drug are also different for each patient. Therefore, according to the total number of recommended therapeutic drugs and the ranking of the recommended therapeutic drugs, the recommendation index of each therapeutic drug is determined to achieve personalized recommendation for patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention.

[0026] Figure 1 Shows a flowchart of a method for constructing a clinical therapeutic drug recommendation model in a comorbid state provided by an embodiment of the present invention;

[0027] Figure 2 Shows a schematic diagram of the design of the topological structure of a comorbid diagnosis and treatment network provided by an embodiment of the present invention;

[0028] Figure 3(a) shows a box plot of the relationship between the recommended effect evaluation index and the hidden space dimension of the first therapeutic drug recommendation model provided by an embodiment of the present invention;

[0029] Figure 3(b) shows a box plot of the relationship between the recommended effect evaluation index and the hidden space dimension of the second therapeutic drug recommendation model provided by an embodiment of the present invention;

[0030] Figure 3(c) shows a box plot of the relationship between the recommended effect evaluation index and the hidden space dimension of the third therapeutic drug recommendation model provided by an embodiment of the present invention;

[0031] Figure 3(d) shows a box plot of the relationship between the recommended effect evaluation index and the hidden space dimension of the fourth therapeutic drug recommendation model provided by an embodiment of the present invention;

[0032] Figure 3(e) shows a box plot of the relationship between the recommended effect evaluation index provided by the embodiment of the present invention and the hidden space dimension of the fifth therapeutic drug recommendation model;

[0033] Figure 3(f) shows a box plot of the relationship between the recommended effect evaluation index provided by the embodiment of the present invention and the hidden space dimension of the sixth therapeutic drug recommendation model;

[0034] Figure 4 shows the flow chart of the therapeutic drug recommendation method for co-morbid patients provided by the embodiment of the present invention. Detailed implementation manners

[0035] Next, the technical solutions of the present invention will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0036] In the description of the present invention, it should be noted that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0037] The embodiment of the present invention provides a method for constructing a clinical therapeutic drug recommendation model in a co-morbid state, as Figure 1 shown, including the following steps:

[0038] Step S001: Obtain a data set, which includes multiple pieces of data. The data contains diagnostic symptom information, therapeutic drug information, and patient information. The therapeutic drug information includes clinical actual medication information.

[0039] In an optional embodiment, the diagnostic symptom information includes the disease name, relatively important symptom-based diagnoses that have not been confirmed as related diseases, and symptom-based variables collected from the medical data intelligent platform.

[0040] In an optional embodiment, the preprocessing of the data in the embodiment of the present invention includes missing value and invalid variable processing, entity name normalization, graph data preprocessing, and other contents.

[0041] In an alternative embodiment, the handling of missing values and invalid variables in data preprocessing means that records with an excessive proportion of missing variables in clinical diagnosis and treatment data are excluded. Exemplarily, the visit records of patients with missing symptom and sign, treatment medication, or discharge condition data are excluded, and non-treatment-purpose clinical medications (including diagnostic medications, operation-assisting medications, etc.) are excluded. Previous studies have shown that the missing mechanism of unexamined diagnosis and treatment data of patients in real clinical practice is generally considered to be missing at random. In the embodiment of the present invention, the random forest algorithm is used to fill in the missing values of variables with a missing proportion of less than 30%. Exemplarily, in the embodiment of the present invention, the missForest package in R language is mainly used to fill in BMI (the missing proportion is about 19.08%).

[0042] In an alternative embodiment, entity name normalization in data preprocessing normalizes the diagnosis, symptoms, and treatment drugs in the dataset of the embodiment of the present invention, which mainly includes three parts: adjustment of symbols in the name, unification of expression methods, and correction of misspelled words in the name. Among them, the adjustment of symbols in the name includes: deleting invalid content or incorrect inputs such as spaces, "《》", ";", "*" in the entity name, and also changing the drug entity name "acetic acid * prednisone" to "acetic acid prednisone", etc. The unification of expression methods is to unify the entity names representing the same concept. Exemplarily, the symptom entity names "coughing white sputum", "coughing white sticky sputum", and "coughing white phlegm" are unified to "coughing white phlegm", and the drug name "double-k tablet" is adjusted to "hydrochlorothiazide"; the correction of misspelled words in the name is to correct the misspelled words in the entity name. Exemplarily, the drug names "tiotropium bromide" and "tiotropium amine" are unified and corrected to "tiotropium bromide".

[0043] In an alternative embodiment, graph data preprocessing in data preprocessing often requires pre-converting the data structure from the form of a two-dimensional table into an edge list, an adjacency list, and an adjacency matrix when constructing, describing, and analyzing a network (or called a graph). In the embodiment of the present invention, Python is used to complete the preprocessing work of the above data structure, and at the same time, the NetworkX package in Python is used to complete the mutual conversion between the edge list, the adjacency list, and the adjacency matrix when necessary.

[0044] In an alternative embodiment, the patient information includes a patient record identification code.

[0045] In an alternative embodiment, the dataset includes multiple pieces of data, that is, the dataset includes multiple clinical patients suffering from related diseases. Each piece of data contains the diagnostic symptom information, treatment drug information, and patient information of the patient. The treatment drug information includes the actual clinical medication information of the doctor during the clinical treatment process.

[0046] Step S002: Extract the diagnostic symptom entities, treatment drug entities, and patient entities from each piece of data respectively.

[0047] Step S003: Divide each piece of data in the dataset into a training set and a test set.

[0048] In an alternative embodiment, each piece of data in the dataset is divided into a training set and a test set according to a ratio of 9:1.

[0049] Step S004: Establish a patient-variable matrix based on the diagnostic symptom entities, treatment drug entities, and patient entities in each piece of data in the training set.

[0050] In an alternative embodiment, the patients in the patient-variable matrix are identified according to the patient record identifiers in the patient entities, and the variables in the patient-variable matrix are formed by the disease names in the diagnostic symptom entities, the symptom-based diagnoses that are relatively important but not confirmed as related diseases, the symptom-based variables collected from the medical data intelligent platform, and the treatment drug names in the treatment drug entities.

[0051] Step S005: Use different recommendation strategies to learn the patient-variable matrix respectively to obtain the entity embedding matrices corresponding to each recommendation strategy.

[0052] Step S006: Construct a treatment drug scoring model corresponding to each recommendation strategy according to the entity embedding matrix corresponding to each recommendation strategy.

[0053] In an alternative embodiment, the treatment drug scoring model recommends treatment drugs according to the symptoms and diagnosis entities in each piece of data and scores the recommended treatment drugs.

[0054] In an alternative embodiment, the score of treatment drug j corresponding to patient i in the treatment drug scoring model is defined as:

[0055]

[0056] where D is the set of all diagnoses and symptoms of patient i, and d and m are the embedding vectors of the symptom and diagnosis entity and the treatment drug entity respectively.

[0057] Step S007: Input the symptoms and diagnosis entities of each piece of data in the test set into the treatment drug scoring model to obtain the treatment drug recommendation results corresponding to each recommendation strategy.

[0058] In an alternative embodiment, each piece of data in the test set includes the diagnostic symptom information, treatment drug information, and patient information of this data. The treatment drug information includes the clinical medication information of doctors during the clinical treatment process. The treatment drug scoring model obtains the treatment drug recommendation results corresponding to each recommendation strategy according to the symptoms and diagnosis entities of each piece of data in the test set.

[0059] Step S008: Select the optimal recommendation strategy according to the clinical actual medication information of each data in the test set and the treatment drug recommendation results obtained by the treatment drug scoring models corresponding to each recommendation strategy.

[0060] In an alternative embodiment, the clinical actual medication information, that is, the clinical medication information of doctors during the clinical treatment process, is included in each data of the test set. Compare it with the treatment drug recommendation results obtained in the embodiments of the present invention, and select the recommendation strategy with the best comparison result.

[0061] Step S009: Determine the treatment drug recommendation model according to the optimal recommendation strategy.

[0062] In the embodiments of the present invention, each data in the data set is divided into a training set and a test set. A patient-variable matrix is established according to each data in the training set, and different recommendation strategies are used to learn the patient-variable matrix to obtain the entity embedding matrices corresponding to each recommendation strategy. Then, the data in the test set are respectively imported into the entity embedding matrices corresponding to each recommendation strategy to obtain the treatment drug recommendation results corresponding to each recommendation strategy. Select the optimal recommendation strategy according to the treatment drug recommendation results corresponding to each data in the test set and the clinical actual medication information in each data. Finally, form a treatment drug recommendation model according to the optimal recommendation strategy. In the prior art, when constructing a network, a network of complex comorbidities is constructed, and various treatment drugs are not included for analysis. In the treatment drug recommendation model constructed by the present invention, during the construction process, a patient-variable matrix is established according to the diagnostic symptom entities, treatment drug entities, and patient entities in each data in the training set, realizing the combination of network construction and treatment drugs. The present invention selects the optimal recommendation strategy according to the treatment drug recommendation results corresponding to each data in the test set and the clinical actual medication information in each data, and forms a treatment drug recommendation model according to the optimal recommendation strategy, further improving the accuracy of treatment drug recommendation.

[0063] In an alternative embodiment, for the method for constructing a clinical treatment drug recommendation model under the comorbid state provided in the embodiments of the present invention, the recommendation strategy includes a first recommendation strategy. The entity embedding matrix corresponding to the first recommendation strategy is obtained through the following steps:

[0064] First, construct the patient-variable matrix into a similarity coefficient matrix, and the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities.

[0065] Second, convert the similarity coefficient matrix into an edge list structure to form the first edge list data corresponding to the first comorbid diagnosis and treatment network.

[0066] Finally, perform representation learning on the first edge list data through a large-scale information network embedding method to obtain the entity embedding matrix corresponding to the first recommendation strategy.

[0067] In an alternative embodiment, the patient-variable matrix is constructed as a similarity coefficient matrix. In the similarity coefficient matrix, the correlation relationships between entities can be obtained. Such relationships are represented as edges between nodes in a graph, and the magnitude of the relationship is the magnitude of the edge weight.

[0068] In an alternative embodiment, the Scikit-Learn in Python is used to construct the patient-variable matrix as a similarity coefficient matrix.

[0069] In an alternative embodiment, the edge list structure is a basic data structure in the field of complex networks. After converting the similarity coefficient matrix into an edge list structure, it is more convenient to use NetworkX and Gephi to analyze and visualize the complex network.

[0070] In an alternative embodiment, the NetworkX package in Python is used to convert the similarity coefficient matrix into an edge list structure.

[0071] In an alternative embodiment, as Figure 2 shown, Figure 2 Type-1 in

[0072] In an alternative embodiment, the large-scale information network embedding method, namely the LINE algorithm, can combine the first-order proximity and second-order proximity between nodes to obtain richer topological information than its graph embedding technology. Exemplarily, in the embodiment of the present invention, the first edge list data is subjected to representation learning through the LINE algorithm to obtain the entity embedding matrix corresponding to the first recommendation strategy.

[0073] In an alternative embodiment, for the method for constructing a clinical treatment drug recommendation model in the co-morbidity state provided in the embodiment of the present invention, the recommendation strategy includes a second recommendation strategy. The entity embedding matrix corresponding to the second recommendation strategy is obtained through the following steps:

[0074] First, the patient-variable matrix is constructed as a similarity coefficient matrix, and the similarity coefficient matrix contains diagnostic symptom entities and treatment drug entities.

[0075] Second, the similarity coefficient matrix is converted into an edge list structure, and all edges between entities of the same type of diagnostic symptom entities and all edges between entities of the same type of treatment drug entities are deleted to form the second edge list data corresponding to the second co-morbidity diagnosis and treatment network.

[0076] Finally, perform representation learning on the second edge list data through a large-scale information network embedding method to obtain an entity embedding matrix corresponding to the second recommendation strategy.

[0077] In an alternative embodiment, as Figure 2 shown, Figure 2 Type-2 in

[0078] In an alternative embodiment, for the method for constructing a clinical treatment drug recommendation model in a comorbid state provided in the embodiments of the present invention, the recommendation strategy includes a third recommendation strategy. The entity embedding matrix corresponding to the third recommendation strategy is obtained through the following steps:

[0079] First, construct the patient-variable matrix into a similarity coefficient matrix, and the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities;

[0080] Secondly, convert the similarity coefficient matrix into an edge list structure, and add two-hop relationship data between the diagnostic symptom entities and the treatment drug entities with the patient entity as the intermediary in the edge list structure to form the third edge list data corresponding to the third comorbid diagnosis and treatment network;

[0081] Finally, perform representation learning on the third edge list data through a large-scale information network embedding method to obtain an entity embedding matrix corresponding to the third recommendation strategy.

[0082] In an alternative embodiment, as Figure 2 shown, Figure 2 Type-3 in

[0083] In an alternative embodiment, for the method for constructing a clinical treatment drug recommendation model in a comorbid state provided in the embodiments of the present invention, the recommendation strategy includes a fourth recommendation strategy. The entity embedding matrix corresponding to the fourth recommendation strategy is obtained through the following steps:

[0084] First, construct the patient-variable matrix into a similarity coefficient matrix, and the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities;

[0085] Secondly, convert the similarity coefficient matrix into an edge list structure, delete all edges between entities of the same type in the diagnostic symptom entities and all edges between entities of the same type in the treatment drug entities, and add two-hop relationship data between the diagnostic symptom entities and the treatment drug entities mediated by the patient entity to form the corresponding fifth edge list data of the fifth comorbidity diagnosis and treatment network;

[0086] Finally, perform representation learning on the fifth edge list data through the large-scale information network embedding method to obtain the entity embedding matrix corresponding to the fifth recommendation strategy.

[0087] In an alternative embodiment, as Figure 2 shown, Figure 2 Type-4 in is the fourth comorbidity diagnosis and treatment network, and the fourth comorbidity diagnosis and treatment network includes a two-hop relationship between the diagnostic symptom entity and the treatment drug entity mediated by the patient entity, as well as a direct relationship between the diagnostic symptom entity and the treatment drug entity.

[0088] In an alternative embodiment, for the method for constructing a clinical treatment drug recommendation model in a comorbid state provided in the embodiments of the present invention, the recommendation strategy includes a fifth recommendation strategy, and the entity embedding matrix corresponding to the fifth recommendation strategy is obtained through the following steps:

[0089] First, construct the patient-variable matrix into a similarity coefficient matrix, and the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities.

[0090] Secondly, convert the similarity coefficient matrix into an edge list structure, delete all edges between entities of the same type in the diagnostic symptom entities and all edges between entities of the same type in the treatment drug entities, add two-hop relationship data between the diagnostic symptom entities and the treatment drug entities mediated by the patient entity, and delete the edges between the treatment drug entities and the diagnostic symptom entities to form the corresponding fifth edge list data of the fifth comorbidity diagnosis and treatment network.

[0091] Finally, perform representation learning on the fifth edge list data through the large-scale information network embedding method to obtain the entity embedding matrix corresponding to the fifth recommendation strategy.

[0092] In an alternative embodiment, as Figure 2 shown, Figure 2 Type-5 in is the fifth comorbidity diagnosis and treatment network, and the fifth comorbidity diagnosis and treatment network includes the relationships between the patient entity and the treatment drug entity and the diagnostic symptom entity respectively.

[0093] In an alternative embodiment, constructing five comorbidity diagnosis and treatment networks is to traverse all entities and the relationships between entities in each data as comprehensively as possible, provide rich parameters for the representation learning of the LINE algorithm. At the same time, constructing five comorbidity diagnosis and treatment networks can also adapt to the treatment drug recommendation scenarios of different types of comorbid patients.

[0094] In an alternative embodiment, the treatment drug recommendation model in the prior art may only be effective for general medical advice for specific diagnoses and has little effect on personalized recommendations for the diagnosis and treatment decisions of patients with comorbidities. The five comorbidity diagnosis and treatment networks of the present invention are all constructed based on the numerous and complex comorbidity characteristics of each data. Therefore, the recommended treatment drugs obtained are also effective for patients with comorbidities.

[0095] In an alternative embodiment, for the method for constructing a clinical treatment drug recommendation model in the comorbid state provided in the embodiments of the present invention, the recommendation strategy includes a sixth recommendation strategy, and the entity embedding matrix corresponding to the sixth recommendation strategy is obtained through the following steps:

[0096] Perform representation learning on the patient-variable matrix through matrix factorization to obtain the entity embedding matrix corresponding to the sixth recommendation strategy.

[0097] In an alternative embodiment, through the matrix factorization method (Matrix Factorization, MF), the patient embedding matrix and variable embedding matrix corresponding to the sixth recommendation strategy are obtained.

[0098] In an alternative embodiment, the MF algorithm is a representation learning algorithm. However, when there are many entity types or the relationships between entities are complex, the MF algorithm often cannot meet the needs of complex representation learning tasks. Therefore, constructing complex networks of different topological types can be used as a means to represent the above-mentioned entity types and the relationships between entities.

[0099] In an alternative embodiment, the steps for obtaining the entity embedding matrix corresponding to the sixth recommendation strategy through the MF algorithm are as follows: Define the loss function:

[0100]

[0101] where, ‖·‖ F is the Frobenius norm, and the regularization parameters m, n, and d are the total number of patients, the total number of variables, and the dimension of the latent space respectively, represents the real number space of m×n, represents the real number space of m×d.

[0102] Use the Stochastic Gradient Descent (SGD) method to iteratively optimize the model parameters, and the update formula is:

[0103]

[0104] V = V 0 - 2η(-U T X + U TUV 0 +λV 0 ),

[0105] Among them, U is the patient embedding matrix, V is the variable embedding matrix, U 0 and V 0 are respectively the patient embedding matrix and the variable embedding matrix before update. The learning rate η = 0.003, and the set of latent space dimensions is set to {x|x = 2n, 1 ≤ n ≤ 20, n ∈ N}.

[0106] In an alternative embodiment, through the above-mentioned MF algorithm, the patient embedding matrix and the variable embedding matrix are learned, and the patient embedding matrix and the variable embedding matrix are the entity embedding matrices corresponding to the sixth recommendation strategy.

[0107] In an alternative embodiment, the method for constructing a clinical treatment drug recommendation model in the co-morbidity state provided in the embodiments of the present invention further includes:

[0108] According to the intersection and union of the treatment drug recommendation results corresponding to each data in the test set and the clinical actual medication information in each data, calculate the accuracy rate of the treatment drug recommendation results of each recommendation strategy respectively;

[0109] According to the ranking of the treatment drugs in the clinical actual medication information in the treatment drug recommendation results, calculate the hit rate and the average ranking of the treatment drug recommendation corresponding to each strategy respectively. The hit rate is the hit rate of each treatment drug in the clinical actual medication information in the treatment drug recommendation results, and the average ranking of the treatment drug recommendation is the average value of the rankings of each treatment drug in the clinical actual medication information in the treatment drug recommendation results;

[0110] Select the optimal strategy according to the accuracy rate, the hit rate, and the average ranking of the treatment drug recommendation.

[0111] In an alternative embodiment, the accuracy rate is defined as:

[0112]

[0113] Among them, M i is the set of recommended treatment drugs for patient i in the test set, is the set of treatment drugs in the actual clinical situation of patient i in the test set, and the threshold interval of Jaccard i is [0, 1]. The larger Jaccard i is, the more similar M i is to , the higher the prediction accuracy rate, that is, the better the recommendation effect. On the contrary, the lower the prediction accuracy rate, that is, the worse the recommendation effect.

[0114] In an alternative embodiment, the hit rate is defined as:

[0115]

[0116] Among them, Hits@10 i represents the hit rate of each therapeutic drug in the clinical actual drug use information in the top ten of the therapeutic drug recommendation results. is the set of therapeutic drugs in the actual clinical treatment of patient i in the test set, and rank j is the ranking of therapeutic drug j among all the proposed recommended therapeutic drugs in which is an indicator function. The threshold interval of Hits@10 is also [0, 1]. The larger it is, the higher the hit rate of in the top ten recommendations, that is, the better the recommendation effect. On the contrary, the lower the hit rate, the worse the recommendation effect.

[0117] In an alternative embodiment, the average rank of therapeutic drug recommendation is defined as:

[0118]

[0119] Among them, the average rank of therapeutic drug recommendation is the average value of the ranks of each therapeutic drug in the clinical actual drug use information in the therapeutic drug recommendation results. is the set of therapeutic drugs in the actual clinical treatment of patient i in the test set, and rank j is the ranking of therapeutic drug j among all the proposed recommended therapeutic drugs in. The smaller the Mean Rank, the more forward the average rank of all therapeutic drugs in is, and the better the recommendation effect. On the contrary, the more backward the average rank, the worse the recommendation effect.

[0120] In an alternative embodiment, by way of example, taking the comorbid patients with chronic obstructive pulmonary disease (COPD) as an example, the data set is COPD comorbid patients meeting the inclusion and exclusion criteria. According to the method for constructing a clinical therapeutic drug model in the comorbid state in any one of the above embodiments, a therapeutic drug recommendation model corresponding to each recommendation strategy is obtained. By calculating the accuracy rate, hit rate, and average rank of therapeutic drug recommendation for each data in the data set, a box plot showing the relationship between the recommendation effect evaluation index and the latent space dimension of each therapeutic drug recommendation model as Figure 3(a) - Figure 3(f) shown is obtained.

[0121] In an alternative embodiment, as Figure 3(a) - Figure 3(e)As shown, in the first therapeutic drug recommendation model and the second therapeutic drug recommendation model, the medians of the three indicators of Jaccard, Hits@10, and Mean Rank all reach their extreme values when the latent space dimension is 2. In the third therapeutic drug recommendation model, the fourth therapeutic drug recommendation model, and the fifth therapeutic drug recommendation model, the medians of the three indicators of Jaccard, Hits@10, and Mean Rank all reach their extreme values when the latent space dimension is 4.

[0122] Therefore, the optimal latent space dimension in the first therapeutic drug recommendation model is 2, the optimal latent space dimension in the second therapeutic drug recommendation model is 2, the optimal latent space dimension in the third therapeutic drug recommendation model is 4, the optimal latent space dimension in the fourth therapeutic drug recommendation model is 4, and the optimal latent space dimension in the fifth therapeutic drug recommendation model is 4.

[0123] In an alternative embodiment, as shown in Figure 3(f), the median of Hits@10 in the sixth therapeutic drug recommendation model reaches a maximum value of 0.320 when the latent space dimension is 22, the median of Mean Rank reaches a minimum value of 49.200 when the latent space dimension is 18, and the median of Jaccard is 0.333 when the latent space dimension ranges from 2 to 40.

[0124] Analyzing the three indicators of the sixth therapeutic drug recommendation model, the original significance level α is set to 0.050. After Bonferroni correction, the significance levels α are set to 0.002, 0.002, and 0.005 in sequence. Through Wilcoxon signed-rank test analysis, it is found that the medians of the differences between Jaccard and Hits@10 and the dimension of 40 at most latent space dimensions are not statistically significant. The medians of the differences between Mean Rank and the dimension of 40 when the dimension ranges from 24 to 38 are not statistically significant. Therefore, the optimal latent space dimension in the sixth therapeutic drug recommendation model is set to 40.

[0125] In an alternative embodiment, by way of example, taking the co - morbid patients with chronic obstructive pulmonary disease (COPD) as an example, the data set is COPD co - morbid patients meeting the inclusion and exclusion criteria. According to the co - morbidity state - based clinical treatment drug model construction method of any one of the above - mentioned embodiments, treatment drug recommendation models corresponding to each recommendation strategy are obtained, and three indicators, namely Jaccard, Hits@10, and Mean Rank, are analyzed. The original significance level α is set to 0.050, and the significance level α after Bonferroni correction is set to 0.012 respectively. As shown in Table 1, Table 2, and Table 3, through Wilcoxon signed - rank test analysis, for the three evaluation indicators, the median of the difference between the treatment drug recommendation models with other network topologies and the second treatment drug recommendation model is statistically significant. Among them, the medians of the differences corresponding to Jaccard and Hits@10 are both negative. Since the larger the values of Jaccard and Hits@10, the better the recommendation effect, the negative median of the difference between the treatment drug recommendation models with other network topologies and the second treatment drug recommendation model indicates that the Jaccard and Hits@10 of the second treatment drug recommendation model are the largest, so the recommendation effect of the second treatment drug recommendation model is good. The smaller the Mean Rank value, the better the recommendation effect. The corresponding difference of Mean Rank is positive, indicating that the Mean Rank value of the second treatment drug recommendation model is the smallest, so the recommendation effect of the second treatment drug recommendation model is good. It can be seen that when applying the treatment drug recommendation models of different topological structure diagnosis and treatment networks, the recommendation effect of the second treatment drug recommendation model is better than others. Therefore, the second treatment drug recommendation model is set as the representative recommendation model in the series of representation learning treatment drug recommendation models based on the LINE algorithm.

[0126] Table 1: Jaccard of the LINE algorithm in different topological structure diagnosis and treatment networks and its statistical analysis results

[0127]

[0128]

[0129] In an alternative embodiment, the indicator is expressed in the form of M(QL,QU); Wilcoxon signed - rank test analysis is used. For the * item, since the second treatment drug recommendation model is used as the reference benchmark, no test is performed.

[0130] Table 2: Hits@10 of the LINE algorithm in different topological structure diagnosis and treatment networks and its statistical analysis results

[0131]

[0132] In an alternative embodiment, the index is represented in the form of M(QL,QU); Wilcoxon signed-rank test analysis is adopted, and the * item is not tested because the second therapeutic drug recommendation model is used as the reference benchmark.

[0133] Table 3: Mean Rank of the LINE algorithm in different topological structure diagnosis and treatment networks and its statistical analysis results

[0134]

[0135] In an alternative embodiment, the index is represented in the form of M(QL,QU); Wilcoxon signed-rank test analysis is adopted, and the * item is not tested because the second therapeutic drug recommendation model is used as the reference benchmark.

[0136] In an alternative embodiment, as shown in Table 4, three indicators of Jaccard, Hits@10, and Mean Rank are analyzed. The original significance level α is set to 0.050, and the significance level α after Bonferroni correction is set to 0.016 respectively. Through Wilcoxon signed-rank test analysis, it can be obtained that for the three evaluation indicators, the median of the difference between the MF recommendation strategy and the LINE recommendation strategy is statistically significant. The larger the values of Jaccard and Hits@10, the better the recommendation effect. Among them, the medians of the corresponding differences in Jaccard and Hits@10 between the recommendation strategy under the MF algorithm and the recommendation strategy under the LINE algorithm are both positive, indicating that the Jaccard and Hits@10 of the recommendation strategy under the MF algorithm are the largest, so the recommendation effect of the therapeutic drug recommendation model of the recommendation strategy under the MF algorithm is good. The smaller the Mean Rank value, the better the recommendation effect. The difference between the Mean Rank corresponding to the recommendation strategy under the MF algorithm and the recommendation strategy under the LINE algorithm is positive, indicating that the Mean Rank value of the recommendation strategy under the MF algorithm is the smallest, so the recommendation effect of the therapeutic drug recommendation model of the recommendation strategy under the MF algorithm is good.

[0137] Judging from the three recommendation effect evaluation indicators, the recommendation effect of the therapeutic drug recommendation model of the MF recommendation strategy is better than that of the therapeutic drug recommendation model of the LINE recommendation strategy.

[0138] Table 4: Recommendation effect and its statistical analysis results under the optimal latent space dimension conditions of different representation learning algorithms

[0139]

[0140] In an alternative embodiment, the index is represented in the form of M(QL,QU); Wilcoxon signed-rank test analysis is adopted.

[0141] The embodiment of the present invention also provides a method for recommending therapeutic drugs for co-morbid patients, such asFigure 4 As shown, it includes:

[0142] Step S101: Obtain the patient information and diagnostic symptom information of the target patient.

[0143] Step S102: Input the patient information and diagnostic symptom information into a pre-trained treatment drug recommendation model to obtain recommended treatment drug information. The treatment drug recommendation model is constructed based on the co-morbidity state clinical treatment drug recommendation model construction method in any of the above method embodiments.

[0144] In an optional embodiment, for the co-morbidity patient treatment drug recommendation method provided by the embodiments of the present invention, the treatment drug information includes multiple treatment drugs, and the method further includes:

[0145] Determine the treatment drug recommendation index according to the total number of recommended treatment drugs and the ranking of the recommended treatment drugs.

[0146] In an optional embodiment, the formula for calculating the treatment drug recommendation index is:

[0147]

[0148] Where Index j represents the treatment drug recommendation index, Index j ∈{k|0<k≤100,k∈N}, n i represents the total number of recommended drugs for patient i, rank j represents the ranking of the recommended drug.

[0149] In an optional embodiment, Index j as a quantitative indicator of the relative magnitude of the recommendation degree of all drugs in the recommendation list, means that the larger the Index j of treatment drug j, the more recommended the drug is by the model under the current strategy.

[0150] The treatment drug information obtained by the embodiments of the present invention contains multiple treatment drugs. However, for patients, not every drug is needed, and the treatment effects of each drug are also different for each patient. Therefore, according to the total number of recommended treatment drugs and the ranking of the recommended treatment drugs, the recommendation index of each treatment drug is determined to achieve personalized recommendation for patients.

[0151] Obviously, the above embodiments are only examples clearly described and not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation manners here. And the obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A method for constructing a clinical treatment drug recommendation model in a comorbid state, characterized in that, it includes: Obtain a data set, which includes multiple pieces of data. The data contains diagnostic symptom information, treatment drug information, and patient information. The treatment drug information includes clinical actual medication information; Extract the diagnostic symptom entities, treatment drug entities, and patient entities from each piece of data respectively; Divide each piece of data in the data set into a training set and a test set; Establish a patient-variable matrix based on the diagnostic symptom entities, treatment drug entities, and patient entities of each piece of data in the training set; Use different recommendation strategies to learn the patient-variable matrix respectively to obtain the entity embedding matrix corresponding to each recommendation strategy; Construct a treatment drug scoring model corresponding to each recommendation strategy according to the entity embedding matrix corresponding to each recommendation strategy; Input the symptom diagnosis entities of each piece of data in the test set into the treatment drug scoring model to obtain the treatment drug recommendation results corresponding to each recommendation strategy; Select the optimal recommendation strategy through the clinical actual medication information of each piece of data in the test set and the treatment drug recommendation results obtained by the treatment drug scoring model corresponding to each recommendation strategy; Determine the treatment drug recommendation model according to the optimal recommendation strategy; Among them, according to the intersection and union of the treatment drug recommendation results corresponding to each piece of data in the test set and the clinical actual medication information in each piece of data, calculate the accuracy rate of the treatment drug recommendation results of each recommendation strategy respectively; According to the ranking of the treatment drugs in the clinical actual medication information in the treatment drug recommendation results, calculate the hit rate and the average ranking of the treatment drug recommendation corresponding to each strategy respectively. The hit rate is the hit rate of each treatment drug in the clinical actual medication information in the treatment drug recommendation results, and the average ranking of the treatment drug recommendation is the average value of the rankings of each treatment drug in the clinical actual medication information in the treatment drug recommendation results; Select the optimal strategy according to the accuracy rate, the hit rate, and the average ranking of the treatment drug recommendation.

2. The method for constructing a clinical treatment drug recommendation model in a comorbid state according to claim 1, characterized in that, The recommendation strategy includes a first recommendation strategy. The entity embedding matrix corresponding to the first recommendation strategy is obtained through the following steps: Construct the patient-variable matrix into a similarity coefficient matrix, which contains diagnostic symptom entities and treatment drug entities; Convert the similarity coefficient matrix into an edge list structure to form the first edge list data corresponding to the first comorbid diagnosis and treatment network; Perform representation learning on the first edge list data through a large-scale information network embedding method to obtain the entity embedding matrix corresponding to the first recommendation strategy.

3. The method for constructing a clinical treatment drug recommendation model in a comorbid state according to claim 1, characterized in that, The recommendation strategy includes a second recommendation strategy. The entity embedding matrix corresponding to the second recommendation strategy is obtained through the following steps: Construct the patient-variable matrix into a similarity coefficient matrix, which contains diagnostic symptom entities and treatment drug entities; Convert the similarity coefficient matrix into an edge list structure, delete all edges between entities of the same type of diagnostic symptom entities and all edges between entities of the same type of treatment drug entities, and form the corresponding second edge list data of the second comorbidity diagnosis and treatment network; Perform representation learning on the second edge list data through a large-scale information network embedding method to obtain the entity embedding matrix corresponding to the second recommendation strategy.

4. The method for constructing a clinical treatment drug recommendation model in a comorbid state according to claim 1, characterized in that, The recommendation strategy includes a third recommendation strategy, and the entity embedding matrix corresponding to the third recommendation strategy is obtained through the following steps: Construct the patient-variable matrix into a similarity coefficient matrix, where the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities; Convert the similarity coefficient matrix into an edge list structure, and add two-hop relationship data between the diagnostic symptom entity and the treatment drug entity mediated by the patient entity in the edge list structure to form the corresponding third edge list data of the third comorbidity diagnosis and treatment network; Perform representation learning on the third edge list data through a large-scale information network embedding method to obtain the entity embedding matrix corresponding to the third recommendation strategy.

5. The method for constructing a clinical treatment drug recommendation model in a comorbid state according to claim 1 or 3, characterized in that, The recommendation strategy includes a fourth recommendation strategy, and the entity embedding matrix corresponding to the fourth recommendation strategy is obtained through the following steps: Construct the patient-variable matrix into a similarity coefficient matrix, where the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities; Convert the similarity coefficient matrix into an edge list structure, delete all edges between entities of the same type of diagnostic symptom entities and all edges between entities of the same type of treatment drug entities, add two-hop relationship data between the diagnostic symptom entity and the treatment drug entity mediated by the patient entity, and form the corresponding fourth edge list data of the fourth comorbidity diagnosis and treatment network; Perform representation learning on the fourth edge list data through a large-scale information network embedding method to obtain the entity embedding matrix corresponding to the fourth recommendation strategy.

6. The method for constructing a clinical treatment drug recommendation model in a comorbid state according to claim 5, characterized in that, The recommendation strategy includes a fifth recommendation strategy, and the entity embedding matrix corresponding to the fifth recommendation strategy is obtained through the following steps: Construct the patient-variable matrix into a similarity coefficient matrix, where the similarity coefficient matrix includes diagnostic symptom entities and treatment drug entities; Convert the similarity coefficient matrix into an edge list structure, delete all edges between entities of the same type of diagnostic symptom entities and all edges between entities of the same type of treatment drug entities, add two-hop relationship data between the diagnostic symptom entity and the treatment drug entity mediated by the patient entity, and delete the edges between the treatment drug entity and the diagnostic symptom entity to form the corresponding fifth edge list data of the fifth comorbidity diagnosis and treatment network; Perform representation learning on the fifth edge list data through a large-scale information network embedding method to obtain the entity embedding matrix corresponding to the fifth recommendation strategy.

7. The method for constructing a clinical treatment drug recommendation model in a comorbid state according to claim 1, characterized in that, the recommendation strategy includes a sixth recommendation strategy, and the entity embedding matrix corresponding to the sixth recommendation strategy is obtained through the following steps: Performing representation learning on the patient-variable matrix through a matrix factorization method to obtain the entity embedding matrix corresponding to the sixth recommendation strategy.

8. A method for recommending treatment drugs for comorbid patients, characterized in that, it includes: Obtaining the patient information and diagnostic symptom information of the target patient; Inputting the patient information and diagnostic symptom information into a pre-trained treatment drug recommendation model to obtain recommended treatment drug information, and the treatment drug recommendation model is constructed based on the method for constructing a clinical treatment drug recommendation model in a comorbid state according to any one of claims 1-7.

9. The method for recommending treatment drugs for comorbid patients according to claim 8, characterized in that, the treatment drug information includes multiple treatment drugs, and the method further includes: Determining a treatment drug recommendation index according to the total number of recommended treatment drugs and the ranking of the recommended treatment drugs.

Images