Method and system for generating SQL query

Abstract

A computing system for generating an SQL query executable in an OMOP CDM, according to an embodiment, may comprise a memory and a processor that: acquires selection criteria content indicating subject selection criteria of a selected clinical trial; acquires one or more split selection criteria; acquires refined selection criteria by providing a generation model with a second prompt indicating refinement of the one or more selection criteria; acquires extracted content by providing the generation model with a third prompt indicating content extraction for each of clinical information types; maps a concept identifier to the extracted content by providing the generation model with a fourth prompt indicating mapping of the concept identifier corresponding to a clinical term to the extracted content; acquires a merged SQL query by merging individual SQL queries for the refined selection criteria, which are acquired by providing the generation model with a fifth prompt indicating generation of an SQL query for selecting a subject of a clinical trial on the basis of the mapped extracted content; and acquires evaluation information about the merged SQL query by providing the generation model with a sixth prompt indicating evaluation of the merged SQL query.

Description

SQL query generation method and system

[0001] Hereinafter, a technique is provided for generating SQL queries for selecting subjects for a clinical trial executable in OMOP CDM from selection criteria content indicating subject selection criteria for a clinical trial based on a machine learning-based Large Language Model (LLM).

[0002] Clinical trials are a crucial process in new drug development, and their success depends on appropriate subject selection. Subject selection criteria are established to select suitable candidates for the trial and include various factors such as the subject's health status, age, gender, medical history, and current medications. These selection criteria are typically provided in a document written in natural language, and identifying suitable subjects based on this document is crucial.

[0003] The current clinical trial subject selection process typically involves clinicians or researchers manually reviewing documented selection criteria and then extracting relevant data from hospital information systems or electronic medical records (EMRs) to select appropriate subjects. This method is time-consuming, requiring the manual selection of appropriate subjects from a large data set.

[0004] The Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) is a data model for storing clinical data in a standardized format. It integrates clinical data from various sources, including diseases, procedures, medications, and laboratory results, and helps clinicians and researchers more easily access and analyze the data. Therefore, using OMOP CDM allows for faster selection of clinical trial subjects. However, SQL (Structured Query Language) queries are typically used to access OMOP CDM. However, because SQL has a set of grammar rules that differ from natural language, utilizing it requires users to learn SQL separately.

[0005] Large Language Models (LLMs) are a useful tool for extracting meaningful information from documented text through natural language processing and transforming it into structured data. For example, LLMs can be used to generate queries (e.g., SQL queries) for accessing specific data models (e.g., OMOP CDM) from the content of complex documents, such as clinical trial subject selection criteria. In other words, by utilizing LLMs, users can access OMOP CDM without having to learn SQL separately, enabling them to quickly select clinical trial subjects.

[0006] However, despite the advancement of LLM technology, the technology to convert selection criteria described in natural language into executable SQL queries in a structured data model such as OMOP CDM by utilizing LLM in the clinical trial subject selection process is still inadequate.

[0007] The background technology described above is technology that the inventor possessed or acquired in the process of deriving the disclosure of the present application, and cannot necessarily be said to be publicly known technology disclosed to the general public prior to the present application.

[0008] A method for generating an executable Structured Query Language (SQL) query in an Observational Medical Outcomes Partnership Common Data Model (OMOP CDM), performed by a processor according to one embodiment, comprises the steps of: obtaining selection criteria content indicating subject selection criteria of a selected clinical trial; obtaining one or more selection criteria divided from the selection criteria content by providing a first prompt indicating segmentation of the selection criteria content to a machine learning-based generative model; obtaining refined selection criteria by providing a second prompt indicating refinement of the one or more selection criteria to the generative model; obtaining extracted content by providing a third prompt indicating content extraction by clinical information types for the refined selection criteria to the generative model; mapping concept identifiers corresponding to clinical terms to the extracted content by providing a fourth prompt indicating mapping of the concept identifiers corresponding to clinical terms to the extracted content for the refined selection criteria to the generative model; The method may further include: obtaining a merged SQL query by merging individual SQL queries for the refined selection criteria obtained by providing the generation model with a fifth prompt instructing generation of an SQL query for selecting subjects of a clinical trial based on the mapped extracted content for the refined selection criteria; and obtaining evaluation information for the merged SQL query by providing the generation model with a sixth prompt instructing evaluation of the merged SQL query based on evaluation criteria.

[0009] The second prompt may include content that maps the one or more screening criteria to one of the trial-specific categories for each clinical trial criteria and one of the non-trial-specific categories indicating criteria for assessing the baseline health status of the subject, and that directs deletion of the one or more of the mapped screening criteria that are mapped to the trial-specific categories.

[0010] The second prompt may include content that directs deletion of content that is redundant or irrelevant to subject selection for the clinical trial within the one or more selection criteria based on simplification criteria including a first simplification criterion directing maintenance of demographics and pregnant information in the OMOP CDM and a second simplification criterion directing maintenance of the full name when both the full name and an abbreviation of the full name are present.

[0011] The above clinical information types may include at least one of a clinical term type indicating a medical concept or condition, a concept code type, a value type indicating a numeric value or categorical measurement associated with the term, an attribute type indicating additional specifications that modify or specify the term, a time information type indicating time information of a condition or event, a negation expression type, and a domain type indicating a domain of the OMOP CDM.

[0012] The method may further include the step of, in response to the evaluation information satisfying an output condition required to output the merged SQL query, outputting the merged SQL query as an output SQL query, and in response to the evaluation information not satisfying the output condition, generating a modified merged SQL query based on the evaluation information and modified evaluation information corresponding to the modified merged SQL query.

[0013] The above evaluation criteria may include at least one of a first criterion for determining the accuracy of mapping of the concept identifier and a second criterion for determining the degree of inclusion of medical concepts defined in the one or more selection criteria.

[0014] A computing system for generating executable SQL queries in an OMOP CDM according to one embodiment comprises: memory; And obtaining screening criteria content indicating subject selection criteria of a selected clinical trial, and providing a first prompt instructing division of the screening criteria content to a machine learning-based generative model, thereby obtaining one or more screening criteria divided from the screening criteria content, obtaining refined screening criteria by providing a second prompt instructing refinement of the one or more screening criteria to the generative model, obtaining extracted content by providing a third prompt instructing content extraction by clinical information types for the refined screening criteria, and mapping the concept identifier to the extracted content by providing a fourth prompt instructing mapping of a concept identifier corresponding to a clinical term for the extracted content to the generative model, and providing a fifth prompt instructing generation of an SQL query for selecting subjects of a clinical trial based on the mapped extracted content for the refined screening criteria, thereby obtaining a merged SQL query by merging individual SQL queries for the obtained refined screening criteria, and providing a sixth prompt instructing evaluation of the merged SQL query based on evaluation criteria. A processor may be included that obtains evaluation information for the merged SQL query by providing a prompt to the generative model.

[0015] The second prompt may include content that maps the one or more screening criteria to one of the trial-specific categories for each clinical trial criteria and one of the non-trial-specific categories indicating criteria for assessing the baseline health status of the subject, and that directs deletion of the one or more of the mapped screening criteria that are mapped to the trial-specific categories.

[0016] The second prompt may include content that directs deletion of content that is redundant or irrelevant to subject selection for the clinical trial within the one or more selection criteria based on simplification criteria including a first simplification criterion directing maintenance of demographics and pregnant information in the OMOP CDM and a second simplification criterion directing maintenance of the full name when both the full name and an abbreviation of the full name are present.

[0017] The above clinical information types may include at least one of a clinical term type indicating a medical concept or condition, a concept code type, a value type indicating a numeric value or categorical measurement associated with the term, an attribute type indicating additional specifications that modify or specify the term, a time information type indicating time information of a condition or event, a negation expression type, and a domain type indicating a domain of the OMOP CDM.

[0018] The processor may output the merged SQL query as an output SQL query in response to the evaluation information satisfying an output condition required to output the merged SQL query, and may generate a modified merged SQL query based on the evaluation information and modified evaluation information corresponding to the modified merged SQL query in response to the evaluation information not satisfying the output condition.

[0019] The above evaluation criteria may include at least one of a first criterion for determining the accuracy of mapping of the concept identifier and a second criterion for determining the degree of inclusion of medical concepts defined in the one or more selection criteria.

[0020] Figure 1 is a schematic diagram illustrating a method for generating SQL queries according to one embodiment.

[0021] Figure 2 is a flowchart illustrating a method for generating SQL queries according to one embodiment.

[0022] FIG. 3 is a diagram illustrating acquisition and preprocessing of selection criteria content according to one embodiment.

[0023] Figure 4 is a diagram showing content extraction by clinical information types according to one embodiment.

[0024] FIG. 5 is a diagram illustrating concept identifier mapping and individual SQL query generation for extracted content according to one embodiment.

[0025] FIG. 6 is a diagram illustrating the merging of generated individual SQL queries according to one embodiment.

[0026] FIG. 7 is a diagram illustrating evaluation of a merge SQL query according to one embodiment.

[0027] FIG. 8 is a diagram illustrating execution of an output SQL query according to one embodiment.

[0028] FIG. 9 is a block diagram illustrating an SQL query generation system according to one embodiment.

[0029] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0030] Although terms such as "first" or "second" may be used to describe various components, these terms should be interpreted solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0031] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0032] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0033] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0034] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0035] Figure 1 is a schematic diagram illustrating a method for generating SQL queries according to one embodiment.

[0036] The SQL query generation method may be a method of generating an SQL query for selecting subjects for a clinical trial from content (e.g., selection criteria content) containing information regarding subject selection for a clinical trial. The SQL query generation method may include steps (110 to 170). Here, the content may include any form of media capable of conveying information, such as text, voice, images, and combinations thereof. In addition, the SQL query may include content (e.g., text) that indicates a query composed of SQL executable in the OMOP CDM.

[0037] As depicted within the dotted box in FIG. 1, steps (120 to 170) may utilize a generative model. The generative model may be an artificial intelligence model, including a machine learning-based neural network. For example, the generative model may be a pre-trained large language model (LLM), such as Generative Pre-trained Transformer 4 (GPT-4).

[0038] Additionally, the SQL query generation method may be performed by a processor. The processor may be a hardware device that processes data and executes commands. The processor may be included in a computing system (e.g., an SQL query generation system). A SQL query generation system including a processor is described in detail in FIG. 9 below.

[0039] In step (110), the processor may acquire selection criteria content. The selection criteria content may be content indicating subject selection criteria for a clinical trial (e.g., text indicating subject selection criteria for a clinical trial). Additionally, the selection criteria may be content indicating information necessary for selecting subjects for a clinical trial (e.g., text indicating information necessary for selecting subjects for a clinical trial). The acquisition of the selection criteria content is described in FIGS. 2 and 3 below.

[0040] In step (120), the processor can refine the obtained selection criteria content by preprocessing the obtained selection criteria content. The preprocessing of the selection criteria content is described in FIGS. 2 and 3 below.

[0041] In step (130), the processor may extract content by types (e.g., clinical information types) related to clinical information from the preprocessed selection criteria content (e.g., refined selection criteria). In this specification, the clinical information type may represent a type (e.g., a name of a type) referenced for searching and / or selecting subjects of a clinical trial among various types (e.g., fields) defined for classifying data in a database. Content extraction by clinical information types is described in FIGS. 2 and 4 below. As will be described later, the generated SQL query may include multiple clinical information types and content (e.g., values) assigned to each type.

[0042] In step (140), the processor can map concept identifiers corresponding to clinical terms included in the extracted content (e.g., extracted content) by type against the refined selection criteria. The mapping of concept identifiers is described in FIGS. 2 and 5 below.

[0043] In step (150), the processor can generate an SQL query for selecting clinical trial subjects from the extracted content to which concept identifiers are mapped, based on the refined selection criteria. The generation of the SQL query from the extracted content to which concept identifiers are mapped is described in FIGS. 2 and 5 below.

[0044] In step (160), the processor may merge the individual SQL queries generated for each refined selection criterion. In this specification, the SQL query generated by merging the individual SQL queries may be referred to as a merged SQL query. The merging of the individual SQL queries is described in FIGS. 2 and 6 below.

[0045] In step (170), the processor may evaluate the merged SQL query. For example, the processor may generate evaluation information for the merged SQL query. Based on the generated evaluation information, the processor may determine whether to re-perform steps (130 to 170), as indicated by the arrows in FIG. 1, or output the merged SQL query as is as output SQL. The evaluation of the merged SQL query is described in FIG. 2 and FIG. 7 below.

[0046] The processor can select a patient cohort by executing an output SQL query generated through a SQL query generation method in the OMOP CDM (180). The OMOP CDM (180) may be a data model for storing medical data (e.g., patient cohorts, etc.) in a standardized format. The execution of the output SQL query in the OMOP CDM (180) is described in FIG. 8 below.

[0047] Figure 2 is a flowchart illustrating a method for generating SQL queries according to one embodiment.

[0048] In step (110), the processor may obtain the selection criteria content. For example, the processor may obtain the selection criteria content by receiving the selection criteria content. The acquisition of the selection criteria content is described in detail in FIG. 3 below.

[0049] Step (120) may include steps (210) and (220). In step (210), the processor may obtain one or more selection criteria by segmenting the obtained selection criteria content. For example, the processor may obtain one or more selection criteria from the selection criteria content by providing a first prompt to the generation model. In step (220), the processor may obtain refined selection criteria by refining the one or more selection criteria. For example, the processor may obtain refined selection criteria from the one or more selection criteria by providing a second prompt to the generation model. The preprocessing of the selection criteria content according to steps (210) and (220) is described in detail in FIG. 3 below.

[0050] In step (130), the processor can obtain extracted content by extracting content by clinical information type based on the refined selection criteria. For example, the processor can obtain extracted content from the refined selection criteria by providing a third prompt to the generation model. Content extraction by clinical information type is described in detail in Figure 4 below.

[0051] In step (140), the processor can map concept identifiers corresponding to clinical terms to the extracted content based on the refined selection criteria. For example, the processor can obtain extracted content with mapped concept identifiers from the extracted content by providing a fourth prompt to the generation model. The mapping of concept identifiers to extracted content is described in detail in Figure 5 below.

[0052] In step (230), the processor may merge individual SQL queries generated from the mapped extracted content based on the refined selection criteria. Here, step (230) may include steps (150 and 160). For example, the processor may generate individual SQL queries from the mapped extracted content by providing a fifth prompt to the generation model. Furthermore, the processor may obtain a merged SQL query from the individual SQL queries by providing a prompt to the generation model that instructs the merging of the individual SQL queries. The generation of the individual SQL queries is described in detail in FIG. 5 below, and the merging of the individual SQL queries is described in detail in FIG. 6 below.

[0053] At step (170), the processor may evaluate the merged SQL query. For example, the processor may generate evaluation information from the merged SQL query by providing a sixth prompt to the generation model. Based on the generated evaluation information, the processor may determine whether to re-perform steps (130, 140, 230, and 170), as indicated by arrows in FIG. 2, or output the merged SQL query as an output SQL query. The evaluation of the merged SQL query is described in detail in FIG. 7 below.

[0054] FIG. 3 is a diagram illustrating acquisition and preprocessing of selection criteria content according to one embodiment.

[0055] A user (320) (e.g., a clinician, a researcher, etc.) may select a selection criteria content (330) from a database (310). Here, the database (310) may be a collection of structured data including one or more selection criteria content (330). As described below, the database (310) may be implemented by being stored on a server (e.g., a server of an SQL query generation system).

[0056] The processor can obtain the selection criteria content (330) by receiving the selection criteria content (330). For example, the user (320) can select the selection criteria content (330) from the database (310) and transmit the selected selection criteria content (330) to the processor. Furthermore, the selection criteria content (330) can be selected from various sources, not just the database (310).

[0057] In the aforementioned step (120), the processor may preprocess the selection criteria content (330). Here, the step (120) may include steps (210, 340, and 350).

[0058] In step (210), the processor may segment the selection criteria content (330). For example, the processor may obtain one or more selection criteria segmented from the selection criteria content (330) as output values ​​by providing a prompt (e.g., a first prompt) that instructs segmentation of the selection criteria content (330) as an input value to the generation model.

[0059] In steps (340 and 350), the processor may refine one or more selection criteria. For example, the processor may provide a prompt (e.g., a second prompt) as input to the generative model, which instructs the refinement of one or more selection criteria, thereby obtaining refined selection criteria (360) as output.

[0060] For example, in step (340), the processor may classify and filter the selection criteria. For example, the processor may classify the selection criteria into categories that fall into trial-specific categories or non-trial-specific categories, and may delete (i.e., filter) the selection criteria classified into the trial-specific categories. Here, the trial-specific category may be a set of categories for clinical trial criteria (e.g., criteria directly related to the design, execution, and ethical considerations of a clinical trial). Additionally, the non-trial-specific category may be a set of categories indicating criteria for assessing the underlying health status of a subject. The categories may, for example, encompass selection criteria having the same properties.

[0061] Additionally, for example, in step (350), the processor may simplify the selection criteria. Simplifying the selection criteria may include removing content from the selection criteria that is trivial, redundant, or irrelevant to subject selection for the clinical trial to prevent errors that may arise due to incomplete clinical data (e.g., clinical criteria content). For example, the processor may simplify the selection criteria based on simplification criteria, including sentence criteria that indicate criteria for simplification of content having sentence-level length, and syntax criteria that indicate criteria for simplification of content having phrase-level length.

[0062] In this way, the processor can obtain refined selection criteria (360) from the selection criteria content (330) by performing step (120) (i.e., preprocessing). That is, the processor can obtain one or more selection criteria from the selection criteria content (330) by performing step (210), and can obtain refined selection criteria (360) from the selection criteria by performing steps (340 and 350). Although steps (340 and 350) are illustrated in FIG. 2 as being performed in the order of step (210), step (340), and step (350), steps (210, 340, and 350) and parts thereof may be performed simultaneously or in any order. Each detailed step of the preprocessing will now be described in detail.

[0063] Since the selection criteria content (330) is typically written in free-text format, it is necessary to divide the selection criteria content (330) into selection criteria in advance to generate SQL queries for each selection criterion. Therefore, as described above, in step (210), the processor can obtain one or more selection criteria by dividing the selection criteria content (330). At this time, the division of the exemplary selection criteria content (330) can be represented as shown in Table 1 below.

[0064] Selection criteria contentOne or more selection criteriaInclusion Criteria:Older than 75 years of ageHypertensionExclusion Criteria:Any mechanical prosthetic heart valveStroke (ischemic or hemorrhagic){"inclusion_criteria": ["Older than 75 years of age","Hypertension"],"exclusion_criteria": ["Any mechanical prosthetic heart valve","Stroke (ischemic or hemorrhagic)"]}

[0065] As shown in Table 1 above, the selection criteria content (330) generally includes selection criteria divided into inclusion criteria and exclusion criteria. The processor can obtain one or more selection criteria by providing a first prompt to the generative model. The first prompt may include, for example, a prompt that instructs to first divide the selection criteria content (330) into inclusion criteria and exclusion criteria, and then further divide this into individual selection criteria included in the inclusion criteria and the selection criteria. The one or more selection criteria obtained through input of the first prompt described above are divided into inclusion criteria and exclusion criteria, as shown in Table 1 above, and each selection criterion is separated by quotation marks.

[0066] The selection criteria content (330) primarily includes information related to the clinical trial itself rather than the subjects (e.g., information corresponding to a trial-specific category). However, the information primarily related to the clinical trial itself is not necessary for generating SQL queries. Therefore, as described above, in step (340), the processor may classify one or more selection criteria into categories belonging to either a trial-specific category or a non-trial-specific category, and delete selection criteria classified into a trial-specific category. For example, the processor may provide a prompt as an input value to the generation model, which instructs the processor to associate one or more selection criteria with either a trial-specific category or a non-trial-specific category, and to delete criteria corresponding to a non-trial-specific category, thereby obtaining the classified and filtered selection criteria as an output value. The detailed categories included in the trial-specific category and the non-trial-specific category may be represented as shown in Table 2 below.

[0067] Category TypeCategoryNon-exam Specific CategoriesDemographic Eligibility, Health and Medical History, Behavioral and Lifestyle, Treatment and Medication History, Disease or Condition Specific

[0068] Additionally, each detailed category classified in Table 2 above can be defined as in Table 3 below.

[0069] Category Definition Example Demographic eligibility Criteria related to demographic factors such as the participant's age, sex, race, and ethnicity. Participants must be between 18 and 65 years old. Health and Medical History Criteria related to the participant's past and present medical conditions, surgeries, and overall health. Participants must have no history of cardiovascular disease. Behavioral and Lifestyle Criteria related to the participant's behavior, habits, and lifestyle choices (e.g., smoking, diet, physical activity, etc.). Participants must be non-smokers and engage in regular physical exercise. Treatment and Medication History Criteria related to the participant's past and present medication use and treatment history, and adherence to prescribed treatments. Participants must not be currently taking any immunosuppressive drugs. Disease or Condition Specific Criteria specific to the disease or condition being studied (e.g., symptoms, disease stage,Diagnosis criteria, etc.) Participants must have been diagnosed with type 2 diabetes within the last 5 years. Location and Accessibility: Criteria related to the geographic location of participants and their ability to access the study site or facilities. Participants must live within a 50-mile radius of the study center. Consent and Comprehension: Criteria to ensure that participants understand and are able to consent to the requirements of the study. Participants must be able to comprehend the study procedures and sign an informed consent form. Study Commitment: Criteria related to the willingness and ability of participants to adhere to the study protocol, attend study visits, and complete follow-up. Participants must be willing to attend monthly follow-up visits for a year. Participant Relationship: Criteria related to relationships with study participants or study staff. Participants must not be direct relatives of the study investigators. Regulatory and Ethical: Criteria to ensure that the study complies with regulatory and ethical standards, including the safety of participants and the confidentiality of data. Participants must meet all regulatory requirements as outlined by the institutional review board (IRB).

[0070] Content indicating definitions for each subcategory of the test-specific and non-test-specific categories according to Tables 2 and 3 above may be included in the second prompt. The second prompt may include, for example, prompts indicating refinement of one or more selection criteria.

[0071] An example of classification and filtering of selection criteria according to step (340) may be as shown in Table 4 below.

[0072] One or more selection criteriaClassified and filtered selection criteria{ "inclusion_criteria": ["Older than 75 years of age","Hypertension"],"exclusion_criteria": ["Any mechanical prosthetic heart valve","Stroke (ischemic or hemorrhagic)"]}{'inclusion_criteria': [{'criterion': 'Older than 75 years of age','category': 'Demographic Eligibility'},{'criterion': 'Hypertension','category': 'Disease or Condition Specific'}],'exclusion_criteria': [{'criterion': 'Any mechanical prosthetic heart valve','category': 'Health and Medical History'},{'criterion': 'Stroke (ischemic or hemorrhagic)','category': 'Health and Medical History'}]}

[0073] As shown in Table 4 above, each criterion in the classified and filtered selection criteria corresponds to a subcategory within the non-exam-specific category. Furthermore, selection criteria corresponding to the test-specific category are deleted, so all selection criteria included in the classified and filtered selection criteria correspond only to the non-exam-specific category.

[0074] The selection criteria content (330) or the selection criteria included therein may contain content that is redundant or irrelevant to the selection of clinical trial subjects (e.g., text that is redundant or irrelevant to the selection of clinical trial subjects). Such redundant or irrelevant content may cause errors in the process of generating and executing SQL queries from the selection criteria content (330), and thus, it is necessary to remove such content from the selection criteria. Accordingly, as described above, the processor may simplify the selection criteria based on simplification criteria (e.g., sentence criteria and syntax criteria). For example, the processor may obtain refined selection criteria (360) as an output value by providing a prompt, which instructs the refinement of the sorted and filtered selection criteria based on simplification criteria including sentence criteria and syntax criteria, as an input value to the generation model. Here, the simplification criteria may be represented as shown in Table 5 below.

[0075] Criteria for content sentences - Remove information regarding approval for research activities - Remove criteria requiring additional patient queries and investigator judgment - Remove criteria regarding ability to provide informed consent and willingness to follow protocols Phrase criteria - Always maintain phrases related to demographics and pregnancy - Always maintain full name when full name and abbreviation exist together - Remove phrases that refer to imaging results to exclude specific conditions - Remove phrases that refer to broad and unspecified concepts - Remove measurements that do not have values

[0076] In Table 5 above, measurement can represent the measurement table of OMOP CDM. In addition, Demographics can represent demographic information of OMOP CDM, and pregnant can represent pregnancy-related information of OMOP CDM. The first and second criteria of the syntax criteria in Table 5 above can be referred to as the first simplification criterion and the second simplification criterion, respectively. The first simplification criterion can be a criterion that indicates the maintenance of demographic information and pregnancy-related information of OMOP CDM. The second simplification criterion can be a criterion that indicates the maintenance of the full name when both the full name and its abbreviation exist. The first simplification criterion and the second simplification criterion described above can be included in the simplification criteria described above. Content indicating the contents of the simplification criteria in Table 5 above can be included in the second prompt. In addition, an example of the simplification of the selection criteria can be as in Table 6 below.

[0077] 분류 및 필터링된 선별 기준정제된 선별기준{'inclusion_criteria': [{'criterion': 'Older than 75 years of age','category': 'Demographic Eligibility'},{'criterion': 'Hypertension','category': 'Disease or Condition Specific'}],'exclusion_criteria': [{'criterion': 'Any mechanical prosthetic heart valve','category': 'Health and Medical History'},{'criterion': 'History of hypersensitivity or allergy to any of the study drugs, drugs of similar chemical classes, ACEIs, ARBs, or NEP inhibitors as well as known or suspected contraindications to the study drugs.','category': 'Treatment and Medication History'}]}{'inclusion_criteria': [{'criterion': 'Older than 75 years of age','category': 'Demographic Eligibility'},{'criterion': 'Hypertension','category': 'Disease or Condition Specific'}],'exclusion_criteria': [{'criterion': 'Any mechanical prosthetic heart valve','category': 'Health and Medical History'},{'criterion': 'History of hypersensitivity or allergy to any of the study drugs, drugs of similar chemical classes, ACEIs, ARBs, or NEP inhibitors.','category': 'Treatment and Medication History'}]}.

[0078] As shown in Table 6 above, in the classified and filtered selection criteria, the selection criterion classified as an exclusion criterion, "History of hypersensitivity or allergy to any of the study drugs, drugs of similar chemical classes, ACEIs, ARBs, or NEP inhibitors as well as known or suspected contraindications to the study drugs", may be refined to become "History of hypersensitivity or allergy to any of the study drugs, drugs of similar chemical classes, ACEIs, ARBs, or NEP inhibitors" in the refined selection criteria (360) by deleting the "as well as known or suspected contraindications to the study drugs" part, which is a phrase with overlapping meaning with the preceding part, through step (350).

[0079] Figure 4 is a diagram showing content extraction by clinical information types according to one embodiment.

[0080] By defining and extracting elements necessary for generating SQL queries from the refined selection criteria (360), more accurate SQL query generation may be possible. Accordingly, as described above, in step (130), the processor may extract content for each clinical information type from the refined selection criteria (360). For example, the processor may obtain content (e.g., extracted content (410)) extracted for each clinical information type as an output value by providing a third prompt as an input value to the generation model. The third prompt may include, for example, a prompt that instructs extraction of content for each clinical information type from the refined selection criteria (360). The extracted content (410) may include, for example, content (e.g., text) extracted for each clinical information type from the refined selection criteria (360). Exemplary clinical information types may be represented as in Table 7 below.

[0081] Type Definition Clinical Term The medical concept or condition being referenced Concept Code A unique identifier or code assigned to the medical concept within a standard terminology or coding system Value A numerical value or categorical measurement associated with the clinical term Attribute Additional specifications that modify or specify the clinical term Temporal Information Temporal information about the condition or event (e.g., duration, timing, frequency of occurrence) Negation Presence or absence of a negation Domain The domain of the OMOP CDM (e.g., condition, measurement, etc.)

[0082] In Table 7 above, a concept (or medical concept) can represent a standardized semantic unit (e.g., each clinical term) defined in OMOP CDM, and a condition (or medical condition) can represent a type of concept representing a patient's disease or diagnosed health state in OMOP CDM. In addition, an event can represent an incident that occurs at a specific point in time or during a specific period. The clinical information types can include at least one of a clinical term type, a concept code type, a value type, an attribute type, a temporal information type, a negation expression type, and a domain type, as defined in Table 7 above. As shown in Table 7 above, a clinical term type can represent a type indicating a medical concept or condition, and a concept code type can represent a type indicating a unique identifier or code assigned to a medical concept. An attribute type can represent additional specifications that modify or specify a term, and a temporal information type can represent a type indicating temporal information of a condition or event. A negation expression type can represent a type indicating the presence or absence of a negation expression, and a domain type can represent a type indicating a domain of the OMOP CDM. Content corresponding to the concept code type in Table 7 above may not be extracted in step (130). In this case, content corresponding to the concept code type may be added to the extracted content (410) through concept identifier mapping in step (140).

[0083] An example of content extraction by clinical information type can be shown in Table 8 below.

[0084] 정제된 선별 기준추출 컨텐츠{'inclusion_criteria': [{'criterion': 'Older than 75 years of age','category': 'Demographic Eligibility'},{'criterion': 'Hypertension','category': 'Disease or Condition Specific'}],'exclusion_criteria': [{'criterion': 'Any mechanical prosthetic heart valve','category': 'Health and Medical History'},{'criterion': 'Stroke (ischemic or hemorrhagic)','category': 'Health and Medical History'}]}{"inclusion_criteria": [{"Older than 75 years of age": [{"clinical_term": "Age","value": "75","attribute": "greater than","temporal": None,"negation": None,"domain": "Demographic"}]},{"Hypertension": [{"clinical_term": "Hypertension""value": None,"attribute": None,"temporal": None,"negation": None,"domain": "Condition"} ...

[0085] As shown in Table 8 above, each selection criterion of the refined selection criteria (360) may correspond to each clinical information type in the extracted content (410) through step (130). For example, the extracted content (410) may include "clinical_term," which is a field name for a clinical term type, and "Age," which is a content assigned to the clinical term type (e.g., a value in text form). Similarly, it may include "value," which is a field name for a value type, and "75," which is a content assigned to the value type (e.g., a value in text form).

[0086] Figure 5 is a diagram showing concept identifier mapping and individual SQL query generation for extracted content.

[0087] When generating an SQL query from the extracted content (410), in order to identify a medical concept included in the extracted content (410), it is necessary to map a concept identifier for identifying the clinical term (e.g., content corresponding to a clinical term type among clinical information types (e.g., text corresponding to a clinical term type)) of the extracted content (410). Here, the clinical term may include standardized clinical terms (e.g., diseases, symptoms, drugs, diagnostic procedures, etc. defined in OMOP CDM). In addition, the concept identifier may be a unique identifier or code for identifying the clinical term (e.g., concept_id of OMOP CDM, standard terminology code of SNOMED CT, etc.). To this end, as described above, the processor may map the concept identifier corresponding to the clinical term in the extracted content (410) in step (140). For example, the processor may obtain the extracted content (510) to which the concept identifier is mapped as an output value by providing the fourth prompt as an input value to the generation model. A fourth prompt may include, for example, a prompt that directs the mapping of concept identifiers corresponding to clinical terms in the extracted content (410) for the refined selection criteria. Here, the mapping may be performed on content (e.g., text) included in the clinical term type of the extracted content (410), as illustrated in step (140) of FIG. 5 .

[0088] Here, the concept identifier mapped to the extracted content (510) may be a concept identifier identifiable in OMOP CDM (e.g., concept_id of OMOP CDM). However, if the concept_id is directly mapped to the extracted content (410) in this way, the accuracy of the mapping may be relatively low. Therefore, an external identifier may first be mapped to the extracted content (410), and the concept_id may be identified using the concept code column when generating an SQL query. Here, the external identifier may include an identifier or code for a standardized term other than the concept identifier of OMOP CDM among the concept identifiers (e.g., a standard code of SNOMED CT, LOINC, or RxNorm). The concept code column may include data indicating a correspondence between the external identifier included in OMOP CDM and the concept_id.

[0089] As another example, the processor may map the concept_id directly in step (140). In this case, a mapping tool (e.g., USAGI of OHDSI) may be used for more accurate mapping. The mapping tool may be a software tool for generating a concept map that corresponds concept_id to clinical terms from the selection criteria contents. When using the mapping tool, after generating the concept map from the selection criteria contents, a prompt instructing the generation model to directly map the concept_id among the concept identifiers to the extracted contents (410) may be input as an input value, thereby obtaining the extracted contents (510) with the concept_id mapped as an output value. In this case, an individual SQL query (520) generated from the mapped extracted contents (510) can directly identify the concept_id (e.g., condition_concept_id = '42343007'). For example, as illustrated in step (140) of FIG. 5, the processor can obtain extracted content (510) mapped to an output value by providing an external standard code for the extracted content (410) as an input value to the fourth prompt generation model.

[0090] In step (150), the processor may obtain an individual SQL query (520) as an output value by providing a fifth prompt as an input value to the generation model. The fifth prompt may include, for example, a prompt that instructs the generation of an SQL query for selecting subjects for a clinical trial based on the mapped extracted content (510) for the refined selection criteria. Here, the individual SQL query (520) may be an SQL query generated from each refined selection criterion. An example of concept identifier mapping for the extracted content (410) may be shown in Table 9 below.

[0091] Extracted ContentMapped Extracted Content{"inclusion_criteria": [{"Older than 75 years of age": [{"clinical_term": "Age","value": "75","attribute": "greater than","temporal": None,"negation": None,"domain": "Demographic"}]},{"Hypertension": [{"clinical_term": "Hypertension""value": None,"attribute": None,"temporal": None,"negation": None,"domain": "Condition"} ...{"inclusion_criteria": [{"Older than 75 years of age": [{"clinical_term": "Age","concept_code": "","value": "75","attribute": "greater than","temporal": None,"negation": None,"domain": "Demographic"}]},{"Hypertension": [{"clinical_term": "Hypertension""concept_code": "38341003(SNOMED CT)"value": None,"attribute": None,"temporal": None,"negation": None,"domain": "Condition"} ...

[0092] Additionally, an example of generating an individual SQL query (520) can be represented as shown in Table 10 below.

[0093] 매핑된 추출 컨텐츠개별 SQL 쿼리{"inclusion_criteria": [{"Older than 75 years of age": [{"clinical_term": "Age","concept_code": "","value": "75","attribute": "greater than","temporal": None,"negation": None,"domain": "Demographic"}]},{"Hypertension": [{"clinical_term": "Hypertension""concept_code": "38341003(SNOMED CT)"value": None,"attribute": None,"temporal": None,"negation": None,"domain": "Condition"} ...{"inclusion_criteria": [{"simplified_criterion": "Older than 75 years of age","sql_query": "SELECT * FROM person WHERE year_of_birth <= YEAR(CURRENT_DATE) - 75;"},{"simplified_criterion": "Hypertension","sql_query": "SELECT * FROM condition_occurrence WHERE condition_concept_id IN (SELECT concept_id FROM concept WHERE vocabulary_id = 'SNOMED' AND concept_code = '38341003');"} ...

[0094] At the bottom of the generated individual SQL query (520) of Table 10 above, the "condition_concept_id IN (SELECT concept_id FROM concept WHERE vocabulary_id = 'SNOMED' AND concept_code = '38341003')" part may indicate that the concept_id is identified using the concept code column from the external identifier ('38341003' of the standard vocabulary code SNOMED). In this way, the processor can generate an individual SQL query (520) that can relatively accurately identify the concept_id by mapping the external identifier to the extracted content (410) in step (140) and generating the individual SQL query (520) from the mapped extracted content (510). As shown in Table 10 above, by mapping the concept identifier to the extracted content (410), the mapped extracted content (510) may have a concept code type ("concept _code") added for each refined selection criterion. As shown in Table 10 above, the contents of the mapped extracted content (510) can be converted into an SQL query by generating an individual SQL query (520) from the mapped extracted content (510).

[0095] FIG. 6 is a diagram illustrating the merging of generated individual SQL queries according to one embodiment.

[0096] In step (160), the processor can obtain a merged SQL query (620) as an output value by providing a prompt as an input value to the generation model that instructs the merging of the individual SQL queries (610) generated for each refined selection criterion. Here, the merged SQL query (620) can include a merged SQL query (620) generated by merging the individual SQL queries (610) for each refined selection criterion. In this way, the selection criteria that were divided from the selection criterion content in step (240) can be merged back into the merged SQL query (620) through step (160). By dividing the selection criterion content by each selection criterion, generating individual SQL queries (610) for each selection criterion, and then merging them back together, more accurate SQL query generation for each selection criterion can be possible.

[0097] FIG. 7 is a diagram illustrating evaluation of a merge SQL query according to one embodiment.

[0098] In step (160), the processor may evaluate the merged SQL query based on evaluation criteria. For example, the processor may generate evaluation information for the merged SQL query by providing a sixth prompt to the generation model. The sixth prompt may include a prompt that directs evaluation of the merged SQL query based on evaluation criteria, including, for example, a first criterion, a second criterion, a third criterion, a fourth criterion, and a fifth criterion. The evaluation criteria may be represented as shown in Table 11 below.

[0099] Evaluation Criteria Description Criteria 1 - Evaluate the accuracy of concept_id mapping in the SQL query - Ensure that each concept in one or more selection criteria is correctly mapped to the appropriate concept_id in the OMOP CDM, and that no concepts are missing - Example question: Is the concept ID mapping done correctly? Criteria 2 - Evaluate whether the SQL query includes all necessary concepts defined in one or more selection criteria - Ensure that the relevant concept_id in the OMOP CDM is correctly identified and included in the SQL query - Example question: Are there any missing concepts? Criteria 3 - Evaluate whether the SQL query accurately reflects the intent and context of the selection criteria content - Ensure that the SQL query properly captures the nuances and conditions described in the selection criteria content - Example question: Is the SQL query contextually correct? Criteria 4 - Evaluate whether the SQL query follows the syntax rules of SQL - Ensure that the query can be executed without syntax errors by using SQL keywords, clauses, and structures correctly - Example question: Is the SQL query syntactically correct? Criteria 5 - Evaluate whether the schema connection with the OMOP CDM in the SQL query is correct - Does the SQL query correctly map to the OMOP CDM? Ensure that tables and fields within the CDM's schema are properly referenced and joined to accurately represent one or more selection criteria - Example question: Is the schema linking done correctly?

[0100] In Table 11 above, a join can represent an operation that combines two or more tables in SQL to retrieve data. Furthermore, the first criterion may represent a criterion for determining the accuracy of the mapping of concept identifiers, the second criterion may represent a criterion for determining the degree of inclusion of medical concepts defined in one or more selection criteria, the third criterion may represent a criterion for determining the degree of contextual reflection of the selection criteria content, and the fourth criterion may represent a criterion for determining the degree of compliance with SQL syntax rules. The fifth criterion may represent a criterion for determining the accuracy of schema connection. Furthermore, a table represents a basic unit for storing data having rows and columns, and a field may represent a column representing the attributes of individual data items within a table. The sixth prompt may include a description of one or more of the evaluation criteria in Table 11 above. Evaluation according to the evaluation criteria in Table 11 above may be performed, for example, by determining an evaluation score, which is a natural number from 1 to 5. In this case, the evaluation information may include an evaluation score, which is an integer from 1 to 5, for each evaluation criterion. In this case, the output condition may be set to obtain a score of 4 or higher for all evaluation criteria, for example. The output condition may include conditions required for the processor to output the evaluated merged SQL query as an output SQL query. In addition, the evaluation information may include various forms of content (e.g., text), and in this case, the sixth prompt input to the generation model may be configured to direct the generation of evaluation information in various forms. In addition, the output condition may be set in various ways depending on the form of the evaluation information (e.g., evaluation scores as natural numbers from 1 to 5, etc.).

[0101] In step (160), the processor may determine, based on the generated evaluation information, whether to re-perform the SQL query generation method from step (130), as illustrated by the arrows in FIG. 7, or to output the merged SQL query as an output SQL query as is. For example, the processor may output the merged SQL query as an output SQL query in response to the evaluation information of the merged SQL query satisfying the output condition. Furthermore, the processor may re-perform steps (130 to 170) in response to the evaluation information of the merged SQL query failing to satisfy the output condition. In this case, the processor may generate a modified merged SQL query and modified evaluation information corresponding to the modified merged SQL query based on the evaluation information. For example, the processor may update the prompts (e.g., the third to sixth prompts) based on the evaluation information. Furthermore, for example, the processor may update the prompts by adding to the prompts a merged SQL query that fails to satisfy the output condition and an evaluation report content for the merged SQL query. Here, the evaluation report content may, for example, include an explanation of the reason for failing to satisfy the output condition. The evaluation report content may be included in the evaluation information. The processor may generate a modified merged SQL query and modified evaluation information corresponding to the modified merged SQL query by performing steps (130 to 170) based on the updated prompts as described above. By modifying the merged SQL query and the evaluation information based on the evaluation information as described above, the processor may generate a more accurate merged SQL query (i.e., the modified merged SQL query). Although only steps (130, 150, and 170) are illustrated in FIG. 7, iterations based on the evaluation information may be performed for steps (130 to 170).

[0102] The number of repetitions of steps (130 to 170) based on the evaluation information may be limited. For example, the upper limit of the number of repetitions of steps (130 to 170) may be set to 3. In this case, if steps (130 to 170) have already been performed 3 times because the output condition is not met, the merged SQL query may be output as an output SQL query even if the evaluation information generated for the merged SQL query does not meet the output condition.

[0103] FIG. 8 is a diagram illustrating execution of an output SQL query according to one embodiment.

[0104] The processor can obtain a patient cohort (820) by executing an output SQL query (810) in the OMOP CDM (180). The patient cohort (820) can include a group of subjects for a clinical trial selected from the output SQL query (810). The processor can store the patient cohort (820) obtained in this manner together with the corresponding output SQL query (810) in a database (310).

[0105] The processor may evaluate the output SQL query (810) obtained in this manner based on the patient cohort (820) obtained by executing the corresponding query. For example, the processor may evaluate the output SQL query (810) based on the scoring of Table 12 below for the evaluation criteria described above in FIG. 7 (e.g., the first to fifth criteria) and the cohort evaluation criteria (e.g., the sixth to ninth criteria) for evaluating the patient cohort (820) obtained by executing the output SQL query (810). Exemplary scoring for each evaluation criterion may be represented as in Table 12 below.

[0106] Evaluation CriteriaScoring Criteria 1: All concept_ids are incorrectly mapped 2: Most concept_ids are incorrectly mapped 3: A few concept_ids are incorrectly mapped 4: Almost all concept_ids are correctly mapped 5: All concept_ids are perfectly mapped 2: Most of the required concepts are missing 3: A few required concepts are missing 4: Almost all of the required concepts are included 5: All of the required concepts are included perfectly 3 Criteria 1: The SQL query does not reflect the context at all 2: The SQL query partially reflects the context, but misses key conditions 3: The SQL query reflects most of the context, but some conditions are incorrect 4: The SQL query accurately reflects almost all of the context 5: The SQL query perfectly reflects the context 4 Criteria 1: The SQL query is syntactically inaccurate and cannot be executed 2: The SQL query contains numerous syntax errors 3: The SQL query contains several syntax errors that can be corrected 4: Almost all of the syntax is correct, but has minor errors 5: The SQL query is Syntactically Perfect Criterion 5: Schema connections are mostly incorrect. 2: Schema connections contain numerous errors. 3: Schema connections contain a few errors. 4: Schema connections are mostly correct. 5: Schema connections are perfect.

[0107] Additionally, the cohort evaluation criteria can be represented as shown in Table 13 below.

[0108] Criteria Description 6 Compare the number of patients in the patient cohort obtained by executing the Criteria Output SQL query with the number of patients in the patient cohort selected by experts (using concept_id) 7 Compare the gender distribution (ratio of male and female patients) of the patient cohort obtained by executing the Criteria Output SQL query with the gender distribution of the patient cohort selected by experts 8 Compare the age distribution of the patient cohort obtained by executing the Criteria Output SQL query with the age distribution of the patient cohort selected by experts 9 Compare the disease distribution (e.g., proportion of patients with a specific disease) of the patient cohort obtained by executing the Criteria Output SQL query with the disease distribution of the patient cohort selected by experts

[0109] The processor can evaluate the output SQL query (810) based on the above evaluation criteria and cohort evaluation criteria. For example, the evaluation information for the output SQL query (810) can be obtained by providing a prompt to the generation model that instructs the generation of evaluation information for the output SQL query (810), which is content (e.g., text) that indicates whether the output SQL query (810) is properly written and functions according to the evaluation criteria and cohort evaluation criteria (i.e., includes a scoring of the evaluation criteria of Table 11 above and the evaluation criteria of Table 12 above, and a description of the cohort evaluation criteria of Table 13 above). The user can, for example, update the prompts (e.g., the first to sixth prompts) based on the evaluation information for the output SQL query (810), if necessary. For example, a user can update the prompts by modifying the prompts so that the difference between the patient cohort (820) selected by the expert according to the cohort evaluation criteria and the patient cohort (820) obtained by executing the output SQL query (810) is reduced, based on the evaluation information for the output SQL query (810) so that the score according to the evaluation criteria is high.

[0110] FIG. 9 is a block diagram illustrating an SQL query generation system according to one embodiment.

[0111] The SQL query generation system may be a computing system including a user terminal (910) and a server (920). Here, the computing system may be a system including one or more computing devices. In addition, the computing devices may include electronic devices (e.g., processors, memories) that perform logical operations and / or storage. The SQL query generation system may perform a SQL query generation method (e.g., steps (110 to 170) described above). More specifically, a processor included in the SQL query generation system (e.g., server (920)) may perform the SQL query generation method. In this case, not only a single processor but also multiple processors may perform the SQL query generation method together.

[0112] A user terminal (910) may include one or more computing devices (e.g., a desktop computer, a laptop, a mobile phone, and a tablet PC) that input or output data. A user may input or output data through the user terminal (910). The user terminal (910) may transmit data received from the user to the server (920), and conversely, may output data received from the server (920).

[0113] The server (920) may include an API (Application Programming Interface) server (930) and a generation server (940). The server (920) may include one or more computing devices. The API server (930) may be a server (920) (e.g., a server (920) built on Flask) that exchanges data with a user terminal (910) and another server (920) (e.g., a generation server (940)) through an API. An API may be an interface that provides a set of rules and defined methods that enable software applications to communicate with each other. The generation server (940) may include one or more computing devices that store a generation model (e.g., GPT-4). When the generation server (940) receives a prompt (e.g., at least one of the first to sixth prompts) from the API server (930), the generation server (940) may generate an output value using the generation model and transmit the generated output value back to the API server (930). Additionally, the server (920) may include one or more computing devices storing the aforementioned database.

[0114] Data transmission and reception between the user terminal (910) and the server (920) and transmission and reception between each server (920) (e.g., the API server (930) and the generation server (940)) may be performed via a network. The network may include communication devices and transmission lines (e.g., an Internet line) used to interconnect devices that are located apart from each other so that data can be exchanged between these devices (e.g., the user terminal (910), the API server (930), the generation server (940)). The network may be used by one or more computing devices included in the user terminal (910) and the server (920) to transmit and receive data to and from other computing devices.

[0115] A user can select a selection criterion content by inputting the selection criterion content to be selected from a database through a user terminal (910). For example, a user can input a selection criterion content selected from among the selection criterion contents received from a database of a server (920) through the user terminal (910). The user terminal (910) can transmit data regarding the input selection criterion content to the API server (930) of the server (920). The API server (930) can perform a SQL query generation method based on the received selection criterion content. For example, the processor of the API server (930) can input the prompt used in each step of the SQL query generation method into a generation model by transmitting it as an input value to the generation server (940), and can perform the SQL query generation method by receiving an output value from the generation server (940). The API server (930) can transmit an output SQL query generated through the SQL query generation method to the user terminal (910). The user terminal (910) can output an output SQL query received from the API server (930). The user can check the output SQL query output through the user terminal (910) and input the modified output SQL query into the user terminal (910). In this case, the user terminal (910) can transmit the modified output SQL query to the API server (930). The API server (930) can obtain a patient cohort by executing the output SQL query or the modified SQL query in the OMOP CDM. The user terminal (910) can provide the patient cohort to the user by outputting the patient cohort received from the API server (930).

[0116] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0117] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0118] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0119] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0120] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0121] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A method for generating an executable SQL (Structured Query Language) query in OMOP CDM (Observational Medical Outcomes Partnership Common Data Model) performed by a processor, A step of obtaining selection criteria content indicating subject selection criteria for a selected clinical trial; A step of obtaining one or more selection criteria segmented from the selection criteria content by providing a first prompt instructing a machine learning-based generative model to segment the selection criteria content; A step of obtaining refined selection criteria by providing a second prompt to the generative model, which instructs refining of the one or more selection criteria; A step of obtaining extracted content by providing a third prompt to the generation model, which instructs content extraction by clinical information types, for the above refined selection criteria; A step of mapping the concept identifier to the extracted content by providing the generative model with a fourth prompt that instructs mapping of the concept identifier corresponding to the clinical term to the extracted content for the refined selection criteria; A step of obtaining a merged SQL query by merging individual SQL queries for the refined selection criteria obtained by providing the generation model with a fifth prompt that instructs the generation of an SQL query for selecting subjects of a clinical trial based on the mapped extracted content for the refined selection criteria; and A step of obtaining evaluation information for the merged SQL query by providing the generation model with a sixth prompt that instructs the evaluation of the merged SQL query based on the evaluation criteria. Including method.

2. In paragraph 1, The second prompt includes content that maps the one or more screening criteria to one of the trial-specific categories for clinical trial criteria and one of the non-trial-specific categories indicating criteria for assessing the baseline health status of the subject, and that instructs deletion of the screening criteria mapped to the trial-specific categories among the one or more mapped screening criteria. method.

3. In paragraph 1, The second prompt includes content that directs deletion of content that is redundant or irrelevant to subject selection of the clinical trial within the one or more selection criteria based on simplification criteria including a first simplification criterion directing maintenance of demographics and pregnant information of the OMOP CDM and a second simplification criterion directing maintenance of the full name when both the full name and an abbreviation of the full name exist. method.

4. In paragraph 1, The above clinical information types include at least one of a clinical term type indicating a medical concept or condition, a concept code type, a value type indicating a numeric value or categorical measurement associated with the term, an attribute type indicating an additional specification that modifies or concretizes the term, a time information type indicating time information of a condition or event, a negation expression type, and a domain type indicating a domain of the OMOP CDM. method.

5. In paragraph 1, In response to the evaluation information satisfying an output condition required to output the merged SQL query, outputting the merged SQL query as an output SQL query, and in response to the evaluation information not satisfying the output condition, generating a modified merged SQL query based on the evaluation information and modified evaluation information corresponding to the modified merged SQL query. Including more method.

6. In paragraph 1, The above evaluation criteria include at least one of a first criterion for determining the accuracy of mapping of the concept identifier and a second criterion for determining the degree of inclusion of medical concepts defined in the one or more selection criteria. method.

7. A computer-readable recording medium having recorded thereon a program for executing the method of paragraph 1.

8. In a computing system that generates executable SQL queries in OMOP CDM, memory; and By obtaining screening criteria content indicating subject selection criteria of a selected clinical trial, and providing a first prompt indicating division of the screening criteria content to a machine learning-based generative model, one or more screening criteria divided from the screening criteria content are obtained, and by providing a second prompt indicating refinement of the one or more screening criteria to the generative model, refined screening criteria are obtained, and by providing a third prompt indicating content extraction by clinical information types for the refined screening criteria, extracted content is obtained, and by providing a fourth prompt indicating mapping of concept identifiers corresponding to clinical terms for the extracted content to the generative model, for the refined screening criteria, the concept identifiers are mapped to the extracted content, and by providing a fifth prompt indicating generation of an SQL query for selecting subjects of a clinical trial based on the mapped extracted content for the refined screening criteria, a merged SQL query is obtained by merging individual SQL queries for the obtained refined screening criteria, and by providing a sixth prompt indicating evaluation of the merged SQL query based on evaluation criteria. A processor that obtains evaluation information for the merged SQL query by providing it to the above generation model. Computing system.

9. In paragraph 8, The second prompt includes content that maps the one or more screening criteria to one of the trial-specific categories for clinical trial criteria and one of the non-trial-specific categories indicating criteria for assessing the baseline health status of the subject, and that instructs deletion of the screening criteria mapped to the trial-specific categories among the one or more mapped screening criteria. Computing system.

10. In paragraph 8, The second prompt includes content that directs deletion of content that is redundant or irrelevant to subject selection of the clinical trial within the one or more selection criteria based on simplification criteria including a first simplification criterion directing maintenance of demographics and pregnant information of the OMOP CDM and a second simplification criterion directing maintenance of the full name when both the full name and an abbreviation of the full name exist. Computing system.

11. In paragraph 8, The above clinical information types include at least one of a clinical term type indicating a medical concept or condition, a concept code type, a value type indicating a numeric value or categorical measurement associated with the term, an attribute type indicating an additional specification that modifies or concretizes the term, a time information type indicating time information of a condition or event, a negation expression type, and a domain type indicating a domain of the OMOP CDM. Computing system.

12. In paragraph 8, The processor outputs the merged SQL query as an output SQL query in response to the evaluation information satisfying an output condition required to output the merged SQL query, and generates a modified merged SQL query based on the evaluation information and modified evaluation information corresponding to the modified merged SQL query in response to the evaluation information not satisfying the output condition. Computing system.

13. In paragraph 8, The above evaluation criteria include at least one of a first criterion for determining the accuracy of mapping of the concept identifier and a second criterion for determining the degree of inclusion of medical concepts defined in the one or more selection criteria. Computing system.

Images