Methods, systems, and computer programs for determining the reliability of data types.

Abstract

To provide a method, system and computer program for determining data shape confidence.SOLUTION: A system determines a shape of incoming data, and determines, based on the shape, whether the data needs to be passed on to support answering a query. If the shape of the data suggests that the data is insufficient or may even be misleading, the system can gate the data or require more data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a method, a system, and a computer program for determining the reliability of data forms.

Background Art

[0002] An entity (e.g., a company) attempting to perform data analysis often sends a "query" to a data source (e.g., a server or a device). The data source responds to the query using data that the entity can use to perform the analysis of the query.

[0003] For example, in the field of healthcare, a company may attempt to determine a patient's adherence to a medication plan (e.g., whether / with what certainty the patient is taking the prescribed medication according to the schedule of the plan). Some drug delivery devices can currently record data regarding their use. For example, an inhaler can be equipped to record when it was used, how many doses the user received, etc. This data can be synchronized to the user's mobile device.

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to provide a method, a system, and a computer program for determining the reliability of data forms.

Means for Solving the Problems

[0005] Some embodiments of the present disclosure can be illustrated as methods. The method includes receiving a query from a query source. The method further includes receiving a dataset. The method further includes obtaining an expected data shape. The method further includes determining the data shape of the dataset. The method further includes comparing the data shape with the expected data shape. The method further includes determining a confidence factor based on the above comparison of the data shape with the expected data shape. The method further includes detecting whether the confidence factor exceeds a confidence threshold. The method further includes sending a response to the query to the query source (in response to the detection), wherein the response is based on the dataset.

[0006] Some embodiments of the present disclosure can also be exemplified as a computer program product comprising a computer-readable storage medium having program instructions embodied thereby, wherein the computer-executable program instructions cause the computer to perform the methods described above.

[0007] Some embodiments of this disclosure can be illustrated as a system, which may comprise memory and a central processing unit (CPU). The CPU may be configured to execute instructions for performing the methods described above.

[0008] The above summary is not intended to describe any of the exemplary embodiments or implementations of this disclosure.

[0009] The drawings included in this application are incorporated herein and form part thereof. These drawings illustrate embodiments of the disclosure and, together with the description, are useful in illustrating the principles of the disclosure. The drawings are illustrative of specific embodiments only and do not limit the disclosure. The features and advantages of various embodiments of the subject matter described in the claims will become apparent as the detailed description below progresses and as reference is made to the drawings. Hereinafter, similar figures indicate similar parts. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows a data-shape-based high-level query response method consistent with several embodiments of this disclosure. [Figure 2] Figure 2 shows a detailed data form model generation and comparison method consistent with several embodiments of this disclosure. [Figure 3] Figure 3 shows a confidence evaluation method based on high-level data format, consistent with several embodiments of this disclosure. [Figure 4] Figure 4 is a block diagram showing exemplary predicted data forms consistent with several embodiments of this disclosure. [Figure 5] Figure 5 illustrates a high-level block diagram of an exemplary computer system that may be used to implement embodiments of the present disclosure. [Modes for carrying out the invention]

[0011] The present invention is capable of various modifications and alternative forms, the details of which are shown in the drawings as examples and will be described in detail below. However, it should be understood that the intention is not to limit the present invention to the specific embodiments described. Rather, the intention is to encompass all modifications, equivalents, and alternatives that fall within the spirit and scope of the invention.

[0012] This disclosure relates to methods, systems, and computer programs for determining the reliability of data format. More specifically, it relates to systems, etc., for receiving data, determining the data format, determining the required data format in a query context, and determining the confidence rating of the received data.

[0013] Throughout this disclosure, one or more “queries” are referenced. As used herein, “query” can mean a request for information. Queries may be received from external sources, for example, via a network. For example, in some embodiments, adherence queries may be received that request data on a patient’s adherence to a medication plan.

[0014] Systems and methods consistent with this disclosure may receive datasets from any number of sources and determine a response to the above query based on one or more of the received datasets. To determine the response, an exemplary system may determine a predicted data shape based on the query, the received datasets, industry standards, or a combination thereof.

[0015] As used herein, the “shape” of a dataset refers to the values ​​of the dataset, as well as metadata such as the type of data the data describes ("data fields"), the number of columns or rows of data or a combination thereof, the completeness of the data, the number of supporting references, etc. In other words, the data shape refers to several attributes of the dataset, such as data fields (e.g., “Patient Name”, “Drug Name”, “Drug Administration Time”, etc.), the values ​​contained within those data fields (e.g., “John Doe”, “Acetaminophen”, “10:26 AM”, etc.), the size of the dataset, etc. An exemplary dataset is shown in Figure 4. The values ​​of the received dataset are classified by data fields.

[0016] When responding to a query, multiple data sources may provide data in various states using various different methods. Data sources can include, for example, applications running on a user's mobile device, external databases, etc. For example, a first external data source may provide a first dataset depending on its internet connection (for example, an application running on a user's mobile device may attempt to synchronize daily, but may not be able to synchronize for a long time if the mobile device is disconnected; upon reconnection, the application may send a dataset, for example, a dataset containing all previously unsent data). Furthermore, the first data source may provide data only for a limited number of data fields. On the other hand, a second data source may periodically provide a second source dataset, where the second source dataset contains data on different data fields. Overlap is also possible; a third data source may periodically provide its own dataset. The third source dataset may include some data fields covered by the first source dataset and some data fields covered by the second source dataset, and so on.

[0017] The shape of one or more received datasets can be compared to a predicted data shape to determine whether the datasets would help to respond to a query or would otherwise be useful in responding to a query. In other words, comparing the shape of the received datasets to a predicted data shape used for a complete response to a hypothetical query allows the system to determine whether the received datasets make relevant responses to the query. For example, if a query requests a number of items sold in a store, and a received dataset describing the store's expenses (salaries, rent / mortgage, utilities, etc.) has a different data shape from the predicted data shape of the response, the system can determine that the received datasets do not make useful responses to the query.

[0018] Figure 1 shows a data-based high-level query response method 100 consistent with several embodiments of the present disclosure. Method 100 includes receiving a query in operation 102. Operation 102 may include, for example, receiving a request for data on a topic, such as patient adherence to a medication plan. This information may be received from an external source, such as a client-server.

[0019] Method 100 further includes, in operation 104, acquiring data. Operation 104 may include, for example, sending a request to a device and receiving data in response. An example of operation 104 is receiving a dataset from an application running on a user's mobile device (e.g., a medication tracker application). In some embodiments, the data in the dataset may be tagged to identify the data fields represented in the dataset. For example, the dataset may indicate that it contains information that identifies a patient, administered medication, one or more times of administration, etc. In some embodiments, operation 104 may include identifying the data fields in the dataset (e.g., if the dataset is not explicitly tagged). In some embodiments, the data may be received from the same source as the query (e.g., a client may send a query to a system performing Method 100 with respect to its own data).

[0020] Method 100 further includes determining the predicted data form in operation 106. As used herein, “data form” means properties of the data, such as the data type (e.g., the data format or file type), a list of data fields contained in the data (e.g., user ID, medication ID, management events, etc.), the size of the dataset (e.g., number of rows / columns, file size, etc.), or some values ​​of the data (e.g., if the medication ID is “acetaminophen” or a numeric code corresponding to acetaminophen), or a combination thereof. Operation 106 may include predicting the predicted data form of a dataset that would be an appropriate response to a query, for example, a query about the adherence of a patient named John Doe to a medication plan for acetaminophen. The predicted form of the data in response to such a query may indicate that the data should include fields describing a patient identifier, a medication identifier, and the number of doses. The form may further indicate that the value of the Patient ID field should be (or refer to) "John Doe" (because data on medication management for different patients may be irrelevant at best, and at worst actively misleading) and that the value of the Drug ID field should be "Acetaminophen".

[0021] The predicted form of the data can be determined based on the received query, the received data, existing industry standards, or a combination thereof, as described in more detail with reference to Figure 2 below. In some embodiments, operation 106 may include receiving the predicted data form from an external source (which may be the source of the query, the source of the data, or various other sources) or loading the predicted data form from storage. An example of the predicted data form is shown in Figure 4.

[0022] Method 100 further includes, at operation 108, determining a confidence level. Operation 108 can include, for example, comparing the form of the received data to the predicted data form. Generally, the confidence level indicates whether the received data set includes a data field and a value indicating that the predicted data form should be so. In other words, the confidence level represents the degree of confidence in the responsiveness of the data (e.g., the extent to which the data is sufficient as a complete answer to a query). For example, a high confidence level indicates that the data is likely to be sufficient as a response to a query, while a low confidence level indicates that the data may be incomplete or that more data is needed.

[0023] The specific method by which the confidence level is determined can vary depending on the embodiment and the use case. In some embodiments, the confidence level can be a simple ratio of the data fields of the received data to the data fields of the predicted data form that makes up the predicted data form. In some embodiments, the confidence level can further account for included data that is not necessary (e.g., if a portion of the received data is irrelevant, this can result in a penalty to the confidence level).

[0024] Method 100 further includes, in operation 110, sending a response to the query. Operation 110 may include, for example, sending the data received in operation 104 to the source of the query received in operation 102 (or, in some use cases, to another source identified by the query). In some embodiments, operation 110 may also include sending a confidence level. This can be particularly advantageous for some query sources (e.g., customers) in that it enables some query sources (e.g., customers) to better utilize the data. For example, a customer may benefit from knowing that the data included in a particular response has a high confidence rating even if the customer has relatively low confidence level requirements (included in the query). In some embodiments, operation 110 may send only data whose form matches the predicted data form. For example, data sent in response to a medication adherence query may be predicted to include data fields such as drug identification data, patient identification data, and drug administration data. If the received data includes the predicted data fields but also includes irrelevant data fields such as the patient's doctor visit date, operation 110 may include removing the irrelevant data before sending the relevant data in response to the query.

[0025] FIG. 2 is a more detailed method 200 of determining and comparing predicted and received data forms that is consistent with some embodiments of the present disclosure. Method 200 includes receiving a query in operation 202. Operation 202 may include, for example, receiving a request for data regarding a topic such as a patient's adherence to a medication schedule. The information may be received from an external source such as a client-server.

[0026] Method 200 further includes, in operation 204, determining the expected data fields of the response to the query. Operation 204 may include, for example, parsing the query and selecting one or more data fields from a predetermined generalized list. The data fields may be selected such that a dataset containing each of the selected data fields can form a coherent response to the query. As an example, the response to a “medication adherence” query may require a patient ID (identifier) ​​field (to allow the client to verify that the data relates to the correct patient), a drug ID (identifier) ​​field (to allow the client to verify that the patient was administered the correct drug), and a dosage field (to allow the client to verify that the drug was administered in the appropriate dosage).

[0027] In some embodiments, data fields may be selected based on pre-programmed behavior. For example, operation 204 may include parsing a query and determining which fields should be included in the response based on the query. In some embodiments, data fields may be based on historical data. For example, operation 204 may include comparing the query to past queries and selecting fields based on the response to the most similar previous query. In some embodiments, data fields may be selected via manual user review. For example, operation 204 may include a user of the system performing method 200, which involves reviewing the query and entering a list of data fields that should be included in the response to the query (via one or more user input devices). Combinations of the above are also considered (for example, operation 204 may include both manual user review and automated parsing of the query).

[0028] In some embodiments, the system may receive a query, identify the general industry of the query (e.g., healthcare), select an existing generalized data form based on the industry, and then refine the generalized data form into a predicted data form.

[0029] Method 200 further includes, in operation 206, determining the expected data values ​​for the response to the query. Operation 206 may, for example, parse the query and determine (and, if so, predict) whether any of the data fields determined in operation 204 can be predicted (and predict the values). For example, if the query relates to John Doe's medication plan adherence and operation 204 identifies at least the “Patient ID” field, operation 206 may include determining whether the value of the “Patient ID” field should contain (or decode, etc.) “John Doe”. Similarly, if the query indicates that the medication plan is for the administration of acetaminophen, operation 206 may include determining that the value of the “Drug ID” field should contain “Acetaminophen”. However, not all data fields have predicted values; for example, the “Administration Timestamp” data field may not have a specific predicted value (unless the query indicates that the administration should be administered within a certain time window), but the field itself may still be predicted to contain regardless of its value. In some use cases, the data field may not have the expected value.

[0030] The predicted fields and values ​​determined in operations 204 and 206, respectively, may form the “predicted data form” of the response to the query. Method 200 further includes in operation 208 comparing the form of the received data to the predicted data form of the data. Operation 208 may include, for example, determining whether the received dataset is missing any predicted data fields or whether the value of any predicted data field is “wrong” (different from the predicted value).

[0031] Operation 208 may further include determining whether any included data field is “irrelevant” (for example, a patient’s height cannot be relevant to the patient’s adherence to a medication plan). Including irrelevant data may result in unnecessary privacy risks, resource consumption (e.g., bandwidth), or even obscuration of the requested data. Information about the first patient, if transmitted in response to a query about the second patient (even if accompanied by appropriate data about the second patient), could be misinterpreted by the client as relating to the second patient.

[0032] Method 200 further includes determining confidence in operation 210. Operation 210 may include, for example, compiling the information resulting from the above comparison in operation 208 into a numerical value (e.g., a value between 0 and 1). The confidence may represent confidence that the received data forms a complete response to the query. In other words, a high confidence may indicate that the received data contains most, if not all, of the predicted data fields, and that the values ​​of those data fields are their predicted values.

[0033] When determining confidence, different predicted fields and values ​​may have different "weights." For example, a predicted patient value (e.g., patient name, e.g., "John Doe") may have a significant weight, and therefore an incorrect patient ID will result in a large penalty to confidence (because the received data relates to a completely different patient and is likely to constitute an inadequate response to the query). However, in the case of a medication adherence plan query, the reliability of the data's effectiveness as a response to the query is not affected, so a dosing timing that deviates from the predicted value may have a relatively small weight. In other words, patients who do not adhere to the plan do not affect the confidence rating; the confidence rating describes how accurately the data answers the query, not the query itself.

[0034] The confidence level can be used to make various decisions about what to do with the received data (for example, whether to look for additional data or send a response), as further explained below with reference to Figure 3. Depending on the embodiment / query, the confidence level may be included in the response to the query.

[0035] Figure 3 shows a confidence evaluation method 300 based on a high level of data form, consistent with several embodiments of the present disclosure. Method 300 includes receiving a confidence score in operation 302. Operation 302 may include, for example, determining a confidence score that represents the confidence that the received dataset will answer a query received (for example, via Method 200 in Figure 2).

[0036] Method 300 further includes in operation 304 determining whether the confidence level exceeds a threshold. Operation 304 may include, for example, comparing the confidence level to a pre-configured threshold (e.g., 0.8, 0.9). The threshold may be determined based on the query, the query may explicitly specify the threshold, or a default threshold may be used. In some embodiments, the query (or configuration) may mean that a threshold comparison is unnecessary, in which case operation 304 may be skipped (for example, the client may want to collect all query responses as long as the client is aware of which responses may be insufficient, which may further allow the client to make its own decisions about which data to trust). Generally, operation 304 determines whether the received dataset forms a sufficient response to the query.

[0037] If the confidence level exceeds a threshold (304 "Yes"), or if 304 is skipped, method 300 further includes sending a response to the query in operation 312. Operation 312 may include, for example, forwarding the raw (unmodified) dataset to the client or other external entity. In some embodiments, operation 312 may include modifying the received data to conform to a specific format or file type or combination thereof. In some embodiments, the confidence level may be included in the response.

[0038] If the confidence level does not exceed the threshold (304 "No"), method 300 further includes attempting to obtain additional data in action 306. Action 306 may include, for example, waiting for the user device to synchronize and sending a request for additional data to an external entity (e.g., a clinic or a database).

[0039] In some embodiments, operation 306 may include selecting a supplemental data retrieval approach based on data or queries. For example, the received query may point to a “backup” data source (e.g., a patient provider’s server) for use by the system running method 300. Operation 306 may also depend on missing data. For example, if a missing data field may be received via periodic device synchronization (e.g., a medication management event field), operation 306 may then include waiting for device synchronization. As a further example, if the missing data field is not likely to be received via periodic device synchronization, operation 306 may send the request to a secondary source instead of waiting for synchronization. In some embodiments, operation 306 may include sending a request or reminder to the user device (to prompt the user of the user device to either synchronize the device or take another action, e.g., administer medication).

[0040] Method 300 further includes receiving additional data in operation 308. Operation 308 may include receiving a supplementary dataset, for example, via device synchronization or in response to a request made as part of operation 306. The additional data may be checked to determine whether it includes previously missing data fields or values ​​(e.g., those identified via operation 208) or combinations thereof. In some embodiments, the additional data may be combined with previously acquired data to form a new dataset, which is then checked for completeness (e.g., the form of the new dataset may be compared to the expected data form of the response).

[0041] Method 300 further includes modifying the confidence level in operation 310. Operation 310 may be performed in substantially the same manner as operation 210 of Method 200, as illustrated with reference to Figure 2. Operation 310 may include, for example, compiling the received data and additional data into a numerical value (e.g., a value between 0 and 1) that represents confidence that they form a complete response to the query. In other words, a high confidence level may indicate that the data contains most, if not all, of the predicted data fields, and that the values ​​of those data fields are their predicted values.

[0042] Using the modified confidence, method 300 may loop back to operation 304, comparing the modified confidence to a threshold. Operations 304–310 may be repeated until the confidence reaches a threshold (at which point the response is sent via operation 312). In some embodiments, an upper limit on the number of loops may be implemented, set by the user of the system running method 300, or outlined in the query.

[0043] Figure 4 is a block diagram showing an exemplary predicted data form 401 consistent with several embodiments of the present disclosure. Figure 4 shows a “generalized” data form 400 which includes several different data fields. The generalized form 400 may be an industry standard or may evolve over time based on past queries or responses to those past queries or a combination thereof.

[0044] The predicted data form 401 includes data fields or values, or combinations thereof, that are determined to be particularly relevant to the query. Note that in some cases, the predicted data form 401 may be identical to the generalized form 400. However, the predicted data form 401 may also include data fields or values, or combinations thereof, that are not present in the generalized form 400. Similarly, the predicted data form 401 may not include any data fields or values ​​from the generalized form 400. Overall, the predicted data form 401 is typically a subset of the generalized form 400.

[0045] In the example shown in Figure 4, the generalized form 400 is a data form related to "healthcare" (including common healthcare data fields, such as patient ID 402, weight field 418, etc.). The exemplary predicted data form 401 shown in Figure 4 includes data fields or values, or combinations thereof, that are determined to be relevant to the response to a query regarding patient medication plan adherence.

[0046] The predicted data type 401 includes a patient field 402. The patient field 402 value may include a patient ID, e.g., a name (e.g., "John Doe"), a patient number, etc. The predicted data type 401 also includes a drug field 404. The drug field 404 value may include one or more drug IDs. For example, the value of drug 1 405 could be "acetaminophen", the value of drug 2 406 could be "ibuprofen", and so on.

[0047] The predicted data form 401 further includes a drug administration field 408. The drug administration field 408 may include a value that identifies when the drug is administered to the patient identified in the patient field 402. The drug administration field 408 includes its own data field for each administration (dose 1 409 and dose 2 410). Dosage 1 409 and dose 2 410 may include specific data, such as the amount of drug administered / the amount of drug estimated to be absorbed or the amount of drug estimated to be bioavailable.

[0048] The generalized form 400 includes all the fields of the predicted data form 401. The generalized form 400 further includes a provider field 412, whose value may identify the provider (e.g., a physician) associated with the patient identified in the patient field 402. The generalized form 400 further includes a prescription field 414, which may contain a value identifying the prescription associated with the patient. The prescription field 414 is not necessarily relevant to adherence queries because the drug in question may be an over-the-counter drug or because queries may describe their own parameters outside of prescriptions.

[0049] The generalized form 400 also includes a blood pressure field 416 (whose value may identify the patient's blood pressure), a weight field 418 (whose value may identify the patient's weight), and a height field 420 (whose value may identify the patient's height). As will be understood by those skilled in the art, these may not be relevant to medication adherence queries (except in rare special cases), in which case they may be omitted from the predicted data form 401.

[0050] Referring here to Figure 5, a high-level block diagram of an exemplary computer system 500 that may be configured to perform various aspects of the present disclosure, such as various aspects of the present disclosure encompassing methods 100, 200, and 300. The exemplary computer system 500 may be used to implement one or more methods or modules described herein, and any associated functions or operations, according to embodiments of the present disclosure (for example, using one or more processor circuits or computer processors of a computer). In some embodiments, the main components of the computer system 500 may comprise one or more CPUs 502, a memory subsystem 508, a terminal interface 516, a storage interface 518, an I / O (input / output) device interface 520, and a network interface 522, all of which may be communicated directly or indirectly for intercomponent communication via a memory bus 506, an I / O bus 514, and an I / O bus interface unit 512.

[0051] The computer system 500 may include one or more general-purpose programmable central processing units (CPUs) 502, some or all of which may include one or more cores 504A, 504B, 504C, and 504D, collectively referred to herein as CPU 502. In some embodiments, the computer system 500 may include multiple processors, which is typical for relatively large systems. However, in other embodiments, the computer system 500 may alternatively be a single CPU system. Each CPU 502 may execute instructions stored in a memory subsystem 508 in a CPU core 504 and may have one or more levels of onboard cache.

[0052] In some embodiments, the memory subsystem 508 may comprise random-access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. In some embodiments, the memory subsystem 508 may represent the entire virtual memory of computer system 500 and may also include the virtual memory of other computer systems connected to or via a network to computer system 500. Conceptually, the memory subsystem 508 may be a single monolithic entity; however, in some embodiments, the memory subsystem 508 may be a more complex arrangement, such as a hierarchy of caches and other memory devices. For example, memory may exist in multiple levels of caches, and these caches may be further divided by function, so that one cache holds instructions, while another holds non-instruction data used by one or more processors. As is known in various so-called non-uniform memory access (NUMA) computer architectures, memory may be further distributed and associated with different CPUs or sets of CPUs. In some embodiments, the main memory or memory subsystem 804 may include elements for controlling and managing the flow of memory used by the CPU 502. This may include a memory controller 510.

[0053] Although the memory bus 506 is shown in Figure 5, as a single bus structure providing a direct communication path between the CPU 502, the memory subsystem 508, and the I / O bus interface unit 512, in some embodiments the memory bus 506 may include multiple different buses or communication paths, which can be arranged in various forms, such as point-to-point links in a hierarchical, star, or web configuration, multiple hierarchical buses, parallel paths, and redundant paths, or any other suitable type of configuration. Furthermore, although the I / O bus interface unit 512 and the I / O bus 514 are shown as a single individual unit, the computer system 500 may, in some embodiments, comprise multiple I / O bus interface units 512, multiple I / O buses 514, or both. In addition, although multiple I / O interface units are shown to isolate the I / O bus 514 from various communication paths leading to various I / O devices, in other embodiments some or all of the I / O devices may be directly connected to one or more system I / O buses.

[0054] In some embodiments, the computer system 500 may be a multi-user mainframe computer system, a single-user system, or a server computer, or a similar device that has little to no direct user interface but receives requests from other computer systems (clients). Furthermore, in some embodiments, the computer system 500 may be implemented as a desktop computer, a portable computer, a laptop computer or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, a mobile device, or any other suitable type of electronic device.

[0055] It should be noted that Figure 5 is intended to show typical main components of an exemplary computer system 500. However, in some embodiments, individual components may be more or less complex than those shown in Figure 5, there may be other or additional components besides those shown in Figure 5, and the number, type, and configuration of such components may vary.

[0056] The present invention may be a system, method, or computer program product or a combination thereof at any possible level of technical detail. The computer program product may encompass one or more computer-readable storage media having computer-readable program instructions for causing a processor to execute the aspects of the present invention.

[0057] The computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer-readable storage medium includes: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved structures on which instructions are recorded, or any suitable combination thereof. As used herein, a computer-readable storage medium should not be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted via wires.

[0058] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to individual computing / processing devices, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may consist of copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and transfers them for storage in a computer-readable storage medium within the individual computing / processing device.

[0059] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, such as object-oriented programming languages, object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.), or procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by personalizing the electronic circuit by utilizing state information of computer-readable program instructions in order to implement the aspects of the present invention.

[0060] The aspects of the present invention are described herein with reference to flowcharts or block diagrams or combinations thereof of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It will be understood that each block in such flowcharts or block diagrams or combinations thereof, and combinations of multiple blocks in such flowcharts or block diagrams or combinations thereof, can be implemented by computer-readable program instructions.

[0061] These computer-readable program instructions can be provided to a computer processor or other programmable data processing device to generate means for implementing functions / operations specified in one or more blocks of the flowchart or block diagram or combination thereof, thereby creating a machine. These computer-readable program instructions can also be stored in a computer-readable storage medium that can instruct a computer-programmable data processing device or other device or combination thereof to function in a particular manner, such that the stored instructions include a product containing instructions that implement the functional / operational aspects specified in one or more blocks of the flowchart or block diagram or combination thereof.

[0062] The computer-readable program instructions may also be loaded onto the computer, other programmable data processing device, or other device such that the instructions executed on the computer, other programmable data processing device, or other device implement the functions / operations specified in one or more blocks of the flowchart or block diagram or combination thereof, thereby causing a series of operational steps on the computer, other programmable device, or other device to generate a computer-implemented process.

[0063] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or part thereof of instructions, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively shown blocks may actually be achieved as a single step executed simultaneously, substantially simultaneously, partially or entirely in a temporally overlapping manner, depending on the functions involved, or the blocks may be executed in reverse order. Note that each block in the block diagram or flowchart or a combination thereof, and any combination of blocks in the block diagram or flowchart or a combination thereof, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations, or by a combination of special-purpose hardware and computer instructions.

[0064] The descriptions of various embodiments in this disclosure are provided for illustrative purposes only and are not intended to be exhaustive or limitful to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to describe the principles of the embodiments, their practical application to or improvement over technologies seen in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer receives queries from a query source. The computer receives a dataset from a data source, wherein the data form of the dataset includes a data field and the value of the data field. The computer determines a predicted data shape, the determination including parsing the query to determine the predicted data fields of the response to the query, predicting the values ​​of the determined predicted data fields, the determination including the determined predicted data fields and the predicted values ​​forming the predicted data shape. The computer compares the data form of the received dataset with the predicted data form, wherein the comparison includes comparing each of the data fields and values ​​of the data fields of the received dataset with each of the data fields and values ​​of the data fields of the predicted data form to determine the degree of agreement. The computer determines the confidence level based on the degree of agreement obtained by the comparison. The computer detects whether the determined confidence level exceeds or does not exceed the confidence threshold, and The computer transmits a response to the query to the query source in accordance with whether the determined confidence level exceeds the confidence threshold, wherein the response includes only the data fields and values ​​of the received dataset that match the predicted data form. A method that includes this.

2. The method according to claim 1, wherein determining the predicted data shape includes determining one or more predicted data fields based on the query, wherein the predicted data shape includes one or more predicted data fields.

3. Determining the predicted data shape is Based on the aforementioned query, identify the industry, and Selecting a generalized data form based on the aforementioned industry, wherein the predicted data form is the generalized data form. The method according to claim 1, including the method described in claim 1.

4. Determining the predicted data shape is Compare the aforementioned query with past queries, Selecting a data field from the response to the most similar past query, wherein the predicted data shape includes the selected data field. The method according to claim 1, including the method described in claim 1.

5. The computer receives a second query from a second query source. The computer receives an additional dataset from the data source or a second data source, wherein the data form of the additional dataset includes a data field and the value of the data field. The computer determines a second predicted data form, the determination of the second predicted data form includes parsing the second query to determine a second predicted data field in the second response to the second query, and predicting the value of the determined second predicted data field, wherein the determined second predicted data field and the predicted value form the second predicted data form. The computer compares the data form of the received additional dataset with the second predicted data form, wherein the comparison includes comparing each of the data fields and values ​​of the data fields of the received additional dataset with each of the data fields and values ​​of the data fields of the second predicted data form to determine the degree of agreement. The computer determines a second confidence level based on the degree of agreement obtained by comparison. The computer detects whether the determined second confidence level exceeds or does not exceed the second confidence threshold, and The computer acquires further data sets depending on whether the determined second confidence level does not exceed the second confidence threshold. The method according to any one of claims 1 to 4, further comprising:

6. The computer determines a third confidence level based on the additional dataset and the further dataset. The computer detects whether the third confidence level exceeds or does not exceed the second confidence threshold, and The computer transmits a second response to the second query to the second query source in response to the third confidence level exceeding the second confidence threshold, wherein the second response includes only the data fields and values ​​of the additional dataset that match the second predicted data form, and only the data fields and values ​​of the further dataset that match the second predicted data form. The method according to claim 5, further comprising:

7. It is a system, Memory and The central processing unit (CPU) connected to the aforementioned memory and It is equipped with, and the CPU, Receive the query from the query source, A dataset is received from a data source, where the data form of the dataset includes a data field and the value of the data field. Determine the predicted data shape, wherein determining includes parsing the query to determine the predicted data fields of the response to the query, predicting the values ​​of the determined predicted data fields, and the determined predicted data fields and the predicted values ​​form the predicted data shape. The data form of the received dataset is compared with the predicted data form, wherein the comparison includes comparing each of the data fields and values ​​of the data fields of the received dataset with each of the data fields and values ​​of the data fields of the predicted data form to determine the degree of agreement. Based on the degree of agreement obtained by the comparison, the confidence level is determined, and Depending on whether the determined confidence level exceeds the confidence threshold, a response to the query is sent to the query source, wherein the response includes only the data fields and values ​​of the received dataset that match the predicted data form. It is structured in such a way. The aforementioned system.

8. The system according to claim 7, wherein determining the predicted data shape includes determining one or more predicted data fields based on the query, wherein the predicted data shape includes one or more predicted data fields.

9. Determining the predicted data shape is Based on the aforementioned query, identify the industry, and Selecting a generalized data form based on the aforementioned industry, wherein the predicted data form is the generalized data form. The system according to claim 7, including the system described in claim 7.

10. Determining the predicted data shape is Compare the aforementioned query with past queries, Selecting a data field from the response to the most similar past query, wherein the predicted data shape includes the selected data field. The system according to claim 7, including the system described in claim 7.

11. The aforementioned CPU, Receive the second query from the second query source, An additional dataset is received from the aforementioned data source or a second data source, wherein the data form of the additional dataset includes a data field and the value of the data field. Determine a second predicted data form, wherein determining the second predicted data form includes parsing the second query to determine a second predicted data field in the second response to the second query, and predicting the value of the determined second predicted data field, wherein the determined second predicted data field and the predicted value form the second predicted data form. The data form of the received additional dataset is compared with the second predicted data form, wherein the comparison includes comparing each of the data fields and values ​​of the data fields of the received additional dataset with each of the data fields and values ​​of the data fields of the second predicted data form to determine the degree of agreement. Based on the degree of agreement obtained by the comparison, a second confidence level is determined. The system detects whether the second confidence level determined above exceeds or does not exceed the second confidence threshold, and The CPU acquires further data sets depending on whether the determined second confidence level does not exceed the second confidence threshold. The system according to any one of claims 7 to 10, further configured as follows.

12. The aforementioned CPU Based on the aforementioned additional dataset and the further dataset, a third confidence level is determined. It is detected whether the third confidence level exceeds or does not exceed the second confidence threshold, and Depending on whether the third confidence level exceeds the second confidence threshold, a second response to the second query is sent to the second query source, wherein the second response includes only the data fields and values ​​of the additional dataset that match the second predicted data form, and only the data fields and values ​​of the further dataset that match the second predicted data form. The system according to claim 11, further configured as follows.

13. The aforementioned CPU, Receive the second query from the second query source, An additional dataset is received from the aforementioned data source or a second data source, wherein the data form of the additional dataset includes a data field and the value of the data field. Determine a second predicted data form, wherein determining the second predicted data form includes parsing the second query to determine a second predicted data field in the second response to the second query, and predicting the value of the determined second predicted data field, wherein the determined second predicted data field and the predicted value form the second predicted data form. The data form of the received additional dataset is compared with the second predicted data form, wherein the comparison includes comparing each of the data fields and values ​​of the data fields of the received additional dataset with each of the data fields and values ​​of the data fields of the second predicted data form to determine the degree of agreement. Based on the degree of agreement obtained by the comparison, a second confidence level is determined. The system detects whether the second confidence level determined above exceeds or does not exceed the second confidence threshold, and In accordance with whether the determined second confidence level exceeds the second confidence threshold, a second response to the second query is sent to the second query source, wherein the second response includes only the data fields and values ​​of the additional dataset that match the second predicted data form. The system according to any one of claims 7 to 10, further configured as follows.

14. On the computer, Receiving queries from a query source, Receiving a dataset from a data source, wherein the data form of the dataset includes a data field and the value of the data field. Determining a predicted data shape, wherein determining includes parsing the query to determine the predicted data fields of the response to the query, predicting the values ​​of the determined predicted data fields, and the determined predicted data fields and the predicted values ​​form the predicted data shape. Comparing the data form of the received dataset with the predicted data form, wherein the comparison includes comparing each of the data fields and values ​​of the data fields of the received dataset with each of the data fields and values ​​of the data fields of the predicted data form to determine the degree of agreement. Based on the degree of agreement obtained through the comparison, the confidence level is determined. To detect whether the determined confidence level exceeds or does not exceed the confidence threshold, and Depending on whether the determined confidence level exceeds the confidence threshold, send a response to the query to the query source, wherein the response includes only the data fields and values ​​of the received dataset that match the predicted data form. A computer program that executes something.

15. The computer program according to claim 14, wherein determining the predicted data shape includes determining one or more predicted data fields based on the query, wherein the predicted data shape includes one or more predicted data fields.

16. Determining the predicted data shape is Based on the aforementioned query, identify the industry, and Selecting a generalized data form based on the aforementioned industry, wherein the predicted data form is the generalized data form. The computer program according to claim 14, including the following:

17. Determining the predicted data shape is Compare the aforementioned query with past queries, Selecting a data field from the response to the most similar past query, wherein the predicted data shape includes the selected data field. The computer program according to claim 14, including the following:

18. To the aforementioned computer, Receiving a second query from a second query source, Receiving an additional dataset from the aforementioned data source or a second data source, wherein the data form of the additional dataset includes a data field and the value of the data field. Determining a second predicted data form, wherein determining the second predicted data form includes parsing the second query to determine a second predicted data field in the second response to the second query, and predicting the value of the determined second predicted data field, wherein the determined second predicted data field and the predicted value form the second predicted data form. Comparing the data form of the received additional dataset with the second predicted data form, wherein the comparison includes comparing each of the data fields and values ​​of the data fields of the received additional dataset with each of the data fields and values ​​of the data fields of the second predicted data form to determine the degree of agreement. Based on the degree of agreement obtained through the comparison, a second confidence level is determined. To detect whether the second confidence level determined above exceeds or does not exceed the second confidence threshold, and Depending on whether the second confidence level determined above exceeds the second confidence threshold, further datasets may be obtained. A computer program according to any one of claims 14 to 17, which further causes the execution of the above.

19. To the aforementioned computer, Based on the aforementioned additional dataset and the further dataset, a third confidence level is determined. To detect whether the third confidence level exceeds or does not exceed the second confidence threshold, and Sending a second response to the second query to the second query source in response to the third confidence level exceeding the second confidence threshold, wherein the second response includes only the data fields and values ​​of the additional dataset that match the second predicted data form, and only the data fields and values ​​of the further dataset that match the second predicted data form. The computer program according to claim 18, which further causes the execution of the above.

20. To the aforementioned computer, Receiving a second query from a second query source, Receiving an additional dataset from the aforementioned data source or a second data source, wherein the data form of the additional dataset includes a data field and the value of the data field. Determining a second predicted data form, wherein determining the second predicted data form includes parsing the second query to determine a second predicted data field in the second response to the second query, and predicting the value of the determined second predicted data field, wherein the determined second predicted data field and the predicted value form the second predicted data form. Comparing the data form of the received additional dataset with the second predicted data form, wherein the comparison includes comparing each of the data fields and values ​​of the data fields of the received additional dataset with each of the data fields and values ​​of the data fields of the second predicted data form to determine the degree of agreement. Based on the degree of agreement obtained through the comparison, a second confidence level is determined. To detect whether the second confidence level determined above exceeds or does not exceed the second confidence threshold, and Depending on whether the determined second confidence level exceeds the second confidence threshold, send a second response to the second query to the second query source, wherein the second response includes only the data fields and values ​​of the additional dataset that match the second predicted data form. A computer program according to any one of claims 14 to 17, which further causes the execution of the above.

Images