Use of metadata to drive batch and real-time processing in data processing systems

Abstract

A technique is described for causing a data processing system to perform real-time decisions by generating data records (e.g., dynamic data records) based on requests for real-time decisions, where the data records include batch data and real-time data retrieved from one or more operational systems in response to the receipt of a request, and real-time refers to the time when the request is received by the data processing system.

Description

Technical Field

[0001] (Claim of Priority) This application claims priority to U.S. Patent Application No. 63 / 490,895, filed on March 17, 2023, and to U.S. Patent Application No. 18 / 491,601, filed on October 20, 2023, under 35 U.S.C. § 119(e), the entire contents of both of which are incorporated herein by reference.

Background Art

[0002] The present invention relates to a data processing system.

[0003] Modern data processing systems manage vast amounts of data within an enterprise. For example, large organizations may have millions of data sets. These data sets can support multiple aspects of a company's operations. Complex data processing systems typically process data in multiple stages, with the results generated by one stage being fed into the next. The overall flow of data through such a system can be described in terms of a directed data flow graph, where the nodes or vertices in the graph represent components (either data files or processes), and the links or "edges" in the graph indicate the flow of data between components. A system for performing such graph-based computations is described in U.S. Patent No. 5,966,072, titled "Executing Computations Expressed as Graphs," which is incorporated herein by reference.

[0004] Graphs can also be used to directly invoke computations. Graphs created according to this system provide methods for introducing data and / or information into individual processes represented by graph components, for retrieving information from individual processes, for moving data and / or information between processes, and for defining the execution order for processes. Systems that invoke these graphs include algorithms for selecting inter-process communication methods and algorithms for scheduling process execution and for monitoring the execution of the graph.

[0005] To support a wide range of functionalities, data processing systems can run applications, whether they implement routine processes or extract insights from datasets. These applications can be programmed to access data stores to read and write data. [Overview of the project]

[0006] In a general embodiment 1, a method implemented by a data processing system for performing a real-time decision by generating a data record (e.g., a dynamic data record) based on a request for a real-time decision, wherein the data record includes batch data and real-time data retrieved from one or more operating systems in response to the receipt of a request, where real-time refers to the time when the request is received by the data processing system, and includes receiving a request for a data record associated with a given key, and in response to receiving the request, retrieving from the repository of the data processing system one or more first values ​​of one or more first fields identified as batch fields retrieved in batch from one or more operating systems prior to the request and stored in the repository of the data processing system, wherein the one or more retrieved first values ​​are associated with a given key, and from one or more operating systems one or more real-time fields stored by one or more operating systems and retrieved in real time from one or more operating systems. Requesting one or more second values ​​for one or more second fields identified as such, wherein the requested one or more second values ​​are associated with a given key; receiving the requested one or more second values ​​associated with the given key from one or more operating systems; generating a requested data record for a given key, the data record comprising one or more first values ​​for one or more first fields identified as one or more batch fields, and one or more second values ​​for one or more second fields identified as one or more real-time fields; and applying one or more rules to the generated data record, which includes determining whether one or more conditions of one or more rules are met, if one or more conditions of one or more rules are met, for a given key,A method is described that includes generating instructions to perform one or more actions specified by one or more rules.

[0007] Aspect 2 of aspect 1 includes reading a table from memory that has cells that specify fields as batch fields and fields as real-time fields, and sending one or more signals that specify a request to one or more operating systems in response to the read.

[0008] Aspect 3, according to any one of aspects 1 to 2, further includes determining from the metadata repository that one or more first fields are identified as one or more batch fields.

[0009] In Embodiment 4, which is one of Embodiments 1 to 3, the batch field is a field that is likely to change at a first frequency, and the real-time field is a field that is likely to change at a second frequency that is higher than the first frequency.

[0010] In embodiment 5, which is one of embodiments 1 to 4, the batch field is a field that is updated intermittently and / or at predetermined intervals.

[0011] In embodiment 6, which is one of embodiments 1 to 5, the real-time field is a field that is updated in real time.

[0012] In embodiment 7, which is one of embodiments 1 to 6, the data record is generated only in response to the receipt of a request for a data record.

[0013] In embodiment 8, which is one of embodiments 1 to 7, the requested one or more second values ​​are received from the operating system only in response to the receipt of a request for a data record.

[0014] In embodiment 9, which is any one of embodiments 1 to 8, the only data requested and / or received from the operating system to generate a requested data record for a given key is one or more requested second values ​​associated with the given key.

[0015] In embodiment 10, which is any one of embodiments 1 to 9, the data record contains only the value associated with a given key.

[0016] In aspect 11, according to any one of aspects 1 to 10, the further includes determining, in response to a request, that one or more second fields from the metadata repository are identified as one or more real-time fields.

[0017] Embodiment 12, according to any one of embodiments 1 to 11, further comprises retrieving a plurality of values ​​of one or more first fields identified as one or more batch fields from one or more operating systems, wherein the plurality of values ​​are associated with a plurality of keys, the plurality of values ​​include one or more first values, and the plurality of keys include a given key.

[0018] Embodiment 13, according to any one of embodiments 1 to 12, further includes storing multiple values ​​associated with multiple keys in a repository of the data processing system.

[0019] In a general embodiment 14, a method implemented by a data processing system for performing a real-time decision by generating a dynamic data record based on a request for a real-time decision, wherein the dynamic data record includes batch data and real-time data retrieved from one or more operating systems in response to the receipt of a request, where real-time refers to when the request is received by the data processing system, and comprises: receiving a request for a dynamic data record associated with a given key; retrieving from the repository of the data processing system one or more first values ​​of one or more first fields identified as batch fields retrieved in batch from one or more operating systems, wherein the one or more retrieved first values ​​are associated with a given key; and requesting from one or more operating systems in response to the receipt of the request one or more second values ​​of one or more second fields identified as real-time fields retrieved in real-time from one or more operating systems, wherein the one or more requested second values ​​are associated with a given key and The process involves: being associated with; receiving from one or more operating systems one or more requested second values ​​associated with a given key; generating a requested dynamic data record for a given key, the dynamic data record comprising one or more first values ​​of one or more first fields identified as one or more batch fields, and one or more second values ​​of one or more second fields identified as one or more real-time fields; applying one or more rules to the generated dynamic data record, which includes determining whether one or more conditions of one or more rules are met based at least in part on one or more second values ​​of one or more second fields identified as one or more real-time fields and one or more first values; determining whether to instruct an external system to perform one or more actions specified by one or more rules for a given key based on the application of one or more rules; writing the generated dynamic data record to memory; and using data indicating whether an instruction has been sent.A method is described that includes modifying one or more data systems stored in memory.

[0020] In a general embodiment 15, a data processing system for performing a real-time decision by generating a data record based on a request for a real-time decision, the data record comprising batch data and real-time data retrieved from one or more operating systems in response to the receipt of a request, where real-time refers to the time when the request is received by the data processing system, and comprising one or more processing devices and one or more machine-readable hardware storage devices that store instructions executable by one or more processing devices to perform an operation according to any one of embodiments 1 to 14.

[0021] In a general embodiment 16, one or more machine-readable hardware storage devices for performing a real-time decision by generating a data record based on a request for a real-time decision, the data record comprising batch data and real-time data retrieved from one or more operating systems in response to the receipt of a request, where real-time is as of the time the request is received by a data processing system, and the one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform an operation according to any one of embodiments 1 to 14.

[0022] According to a preferred embodiment, the data processing system can generate records (e.g., wide records) in which all data is joined together based on relationships from a schema definition. A user can annotate locations within the schema for special roll-ups. Based on which area of the schema is desired (i.e., customer, transaction,...), many different records (e.g., wide records) can be generated. Batch fields are more likely to be updated and / or changed less frequently than real-time fields. Batch fields can be updated and / or changed intermittently and / or at a predetermined time. Real-time fields can be updated and / or changed in real-time. Batch fields may be stored in persistent storage. Real-time fields can be stored in volatile memory.

[0023] Aspects of the techniques described herein reduce latency and / or memory resources in providing the required records (e.g., data records), while improving the computational efficiency in doing so. This is because the techniques described herein only update records on demand, e.g., in response to a request. Thus, the techniques described herein do not waste resources updating records that are never used.

[0024] Details of one or more embodiments of the invention are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the invention will be apparent from the specification and drawings, and from the claims.

Brief Description of the Drawings

[0025] [Figure 1A] FIG. 1A is a block diagram of a system including a data processing system including a real-time integrator. [Figure 1A-01] FIG. 1B is a legend for FIG. 1A. [Figure 1B] FIG. 1C is a block diagram of the system of FIG. 1A showing records (e.g., wide records). [Figure 2]Block diagram of the system of FIG. 1A showing customer records (e.g., batch and real-time records) and business records (e.g., batch and real-time records). [Figure 3A] Block diagram of the system of FIG. 1A showing real-world examples. [Figure 3B] Block diagram of the system of FIG. 1A showing real-world examples. [Figure 3C] Block diagram of the system of FIG. 1A showing real-world examples. [Figure 4A] Block diagram of the system of FIG. 1A showing another real-world example without batch updates. [Figure 4B] Block diagram of the system of FIG. 1A showing another real-world example without batch updates. [Figure 4C] Block diagram of the system of FIG. 1A showing another real-world example without batch updates. [Figure 4D] Block diagram of the system of FIG. 1A showing another real-world example without batch updates. [Figure 5] Graph showing the timeline of component execution in the system of FIG. 1A. [Figure 5A] Legend of FIG. 5. [Figure 6] Flow diagram showing the flow of operations for performing real-time decisions by generating data records (e.g., dynamic data records) based on requests for real-time decisions. [Figure 7] Block diagram of an example of a computing device or computer or data processing system or client or server.

Mode for Carrying Out the Invention

[0026] Referring to FIG. 1A-1, the legend of FIG. 1A is shown. In the flow of FIG. 1A-1, the data flow is represented by a dotted line, the batch flow is represented by a dashed line, and the real-time flow is represented by a solid line.

[0027] Next, referring to Figure 1A, System 10 is shown. System 10 includes a client system 12 (e.g., a customer system) and a data processing system 20. The client system 12 includes a data store 14 for storing records (e.g., transaction records), an operations system 16 that communicates with the data processing system 20, and a client device 17 that communicates with a computing system 18, the computing system 18 which communicates with a rule execution engine 34 in the data processing system 20.

[0028] The data processing system 20 includes a real-time integrator 22. The real-time integrator 22 includes a metadata manager 24 and a metadata repository 25. The metadata manager 24 communicates with a batch search engine 26, and the metadata repository 25 communicates with a real-time search engine 28. The batch search engine 26 sends batch results (for example, every night) of records fetched in batches from the operational system 16 to the batch repository 30, and the real-time search engine 28 sends real-time results (for example, records fetched in real time from the operational system 16) to the on-demand record generator 32. The batch repository 30 and the on-demand record generator 32 communicate with each other, and the on-demand record generator 32 also communicates with the rule execution engine 34.

[0029] Next, referring to Figure 1B, a scaled-down version of Figure 1A is shown, with the on-demand record generator 32 highlighted. If necessary, refer to Figure 1A to identify where the on-demand record generator 32 is positioned relative to the other elements in Figure 1A.

[0030] Figure 1B also shows a record 50 (e.g., a wide record) generated by the on-demand record generator 32. The record 50 (e.g., a wide record) contains a number of fields, including, for example, fields 50a, 50b...50e. Some of the fields 50a, 50b...50e may be vectors of fields and / or vectors of vectors. In one example, the record (e.g., a wide record) contains a data structure structured with fields, where each field stores or is associated with a data value.

[0031] In this example, record 50 (for example, a wide record) has 2146 fields, including a field 50a that stores a key represented by 5 characters, customer information (batch field) 50b represented by multiple fields, a vector of vector fields (batch field) 50c represented by multiple fields, a credit (real-time field) 50d, and a transaction field (real-time field) 50e.

[0032] Referring now to Figure 2, a scaled-down version of Figure 1A (without showing the client device 17, computing system 18, and rule execution engine 34 of Figure 1A) is shown with the operational system 16 and metadata repository 25 highlighted. Refer to Figure 1A as needed to identify where the operational system 16 and metadata repository 25 are located relative to the other components of Figure 1A. The operational system 16 has customer records 60 from the data store 14 (Figure 1A) (four types of records are shown for each of several keys, with three keys being an example, but more types of records may exist for more keys). In this example, customer records 60 include customer records 60a to 60d. Customer records 60 from the operational system 16 have field names that are different from the field names in the real-time integrator 22. The metadata repository 25 stores in record 62 the business name 62a, type 62b, e.g., batch or real-time type, and source name 62c corresponding to the field names in customer records 60.

[0033] Figures 3A to 3C show examples from the real world.

[0034] Next, referring to Figure 3A, a scaled-down version of Figure 1A is shown, with the operating system 16 and metadata manager 24 highlighted in particular. If necessary, refer to Figure 1A to identify where the operating system 16 and metadata manager 24 are located relative to the other components of Figure 1A.

[0035] At the time for batch searching, the batch search engine 26 sends a batch request 54 to the metadata manager 24. The metadata manager 24 then looks up in table 56 (at time T1) which fields are batch type (e.g., only intermittently and / or updated at predetermined intervals) and their names (e.g., names as sources for those fields). In this example, the metadata manager 24 identifies cells 56a and 56b that contain the value "batch". The metadata manager 24 looks up the corresponding source names for batch cells 56a' and 56b'. In this example, the corresponding source names are in cells 56a and 56b. Using the values ​​of 56a and 56b, the metadata manager 24 generates a request 52 specifying the names of the batch fields that the metadata manager 24 needs to retrieve from the operating system 16.

[0036] Next, the operating system 16 returns records 60a and 60b containing the field values ​​and corresponding cell values ​​represented in cells 56a and 56b. Records 60a and 60b are returned to the metadata manager 24. The metadata manager 24 relabels the returned field names by looking up cells 58a and 58b in table 58 that correspond to the technical field names in records 60a and 60b (for example, at time T2). The metadata manager 24 looks up the corresponding cells 58a' and 58b' using the business name of the source name represented in cells 58a and 58b.

[0037] The operation described in Figure 3A can be performed as a batch operation during off-peak hours, such as at night, as indicated by symbol 88a. The operating system 16 sends records 60a and 60b from the data store 14 (Figure 1A) to the metadata manager 24. The metadata manager 24 looks up the source name of the batch field and looks up the business name of the returned field.

[0038] In Figure 3A, batch updates are performed intermittently, for example, once a day, at night. The metadata manager 24 stores a table 56 that identifies which fields are batch fields and therefore are updated only, for example, every night. That is, the operating system 16 stores all data, both real-time and batch. However, the data processing system 20 retrieves batch data from the operating system 16 intermittently and / or at predetermined times, for example, once a night. In contrast, real-time data (described later with respect to Figure 3C) is retrieved by the data processing system 20 only when a request for a record of an event or transaction (e.g., a real-time record) is received. For example, the values ​​of real-time fields may be retrieved from the operating system 16 multiple times a day, while the values ​​of batch fields are retrieved only once per data entry. The metadata manager 24 labels fields as real-time or batch depending on how often the data changes. For example, age and the number of children change relatively infrequently compared to how often a user makes a coffee transaction (which could be three or four times a day).

[0039] In this example, a request is made to the operational system 16 for records related to "Cns_age" to "Cns_child" (e.g., batch records). The operational system 16 returns three records with different keys (three of which are illustrated, but there may be more) as shown for "Cns_age", and three records with different keys (three of which are illustrated, but there may be more) as shown for "Cns_child". These records are sent to the metadata manager 24, which performs a lookup in the metadata repository 25 to generate three records (more than one) with different keys that have the business name "age", and three records with different keys that have the business name "child".

[0040] The batch search engine 26 receives three records with different keys that have the business name "age" and three records with different keys that have the business name "children," and sends to the batch repository 30 a record 63 (for example, a concatenated record) of different keys (three of which are shown, but more) from the three records with different keys that have the business name "age" and the three records with different keys that have the business name "children." Key Age ... Child 7342 34 ... 1 4256 45 ... 2 3145 57 ... 0

[0041] Next, referring to Figure 3B, a reduced version of Figure 1A is shown with a highlighted operating system 16 (not referenced in Figure 3B). Refer to Figure 1A as needed to identify where the operating system 16 is located relative to the other components of Figure 1A. At time T1, a new record 66a is sent to the operating system 16, and at time T2, another new record 66b is sent to the operating system 16. These are real-time operations and are records that occur during peak operating hours of the operating system 16, for example, during daytime operations as indicated by symbol 88b. The new record 66a updates record 60a to record 60a', and key 7342 is updated from Cns_age 34 to Cns_age 35, for example, as indicated by part 60a''. Additionally, the new record 66b updates record 60d to record 60d', and key 7342 is updated from Cns_sbuxtx 1.75 to Cns_sbuxtx 1.75,2.13, for example, as indicated by part 60d''. Other records for key 7342, such as Cns_child and Cns_credit as indicated by records 60b and 60c, remain unchanged.

[0042] Next, referring to Figure 3C, a reduced version of Figure 1A is shown (however, no components are referenced in Figure 3C). Refer to Figure 1A as needed to identify where a particular component is located relative to other components in Figure 1A. The client device 17 sends a request 71 to the computing system 18. In this example, request 71 is "Should I send offer for key: 7342?". Request 71 is a key setting request that includes or otherwise specifies the user key (e.g., a unique identifier) ​​associated with the client device 17. The computing system 18 receives this request 71 and sends it to the rule execution engine 34. The computing system 18 may be associated with a bank or a vendor and therefore receives a request regarding whether an offer should be sent. However, the computing system 18 itself does not store the logic for determining whether an offer should be sent. The rule execution engine 34 retrieves or stores a business rule for determining whether to send an offer to the client device 17 associated with key 7342. An example of a business rule 95 is as follows: "(age * 20+ Number of children * #Number of transactions - credits * If (0.087) > 20, send the offer.

[0043] To execute this business rule 95 (or any other business rule), the rule execution engine 34 requires a record (e.g., a wide record) of all information associated with key 7342, and the rule execution engine 34 requires this information in real time as the client device 17 sends request 71. To receive this record (e.g., a wide record), the rule execution engine 34 sends request 73 to the on-demand record generator 32. Request 73 includes the value of the key. Rather than constantly updating the record for key 7342 with the latest information, the on-demand record generator 32 generates this record (e.g., a wide record) in response to receiving request 73, and again when request 73 is received. In this example, the on-demand record generator 32 only generates a record (e.g., a real-time record) in response to receiving request 73 (real-time is as of when request 71 is sent). The on-demand record generator 32 sends a command 75 to the real-time search engine 28 requesting a record (e.g., a real-time record) for key 7342. In response, the real-time search engine 28 sends a request 77 to the metadata manager 24 for a record with key 7342 (e.g., a real-time record). The metadata manager 24 looks up the source names of real-time fields 79b and 79c for real-time fields 79d and 79e in table 79a of the metadata repository 25 (at time T1). The metadata manager 24 sends a request 81 to the operational system 16 for Cns_credit and Cns_sbuxtx (source names of the looked-up real-time fields) with key 7342 to retrieve the real-time values ​​of these fields represented by the source names of real-time fields 79b and 79c. In some examples, the metadata manager 24 sends the request by sending one or more signals specifying the request. The operational system 16 returns records 91a and 91b, "key 7342 Cns_credit 534", and "key 7342 Cns_sbuxtx 1.75, 2.13".At time T2, the metadata manager 24 looks up the business names 79g and 79h in table 79f, which are the returned source names for real-time fields 79b and 79c. The metadata manager 24 generates new records 91c and 91d, which have the business names 79g and 79h and the values ​​specified in records 91a and 91b for key 7342. The metadata repository 25 returns the new records 91c and 91d, "key 7342 credit 534" and "key 7342 coffee deal 1.75, 2.13" to the real-time search engine 28. That is, the records with the returned fields business names "key 7342 credit 534" and "key 7342 coffee deal 1.75, 2.13" are sent to the real-time search engine 28 by the metadata manager 24. The real-time search engine 28 generates record 91e (e.g., a concatenated record), which is "key 7342 credit 534 coffee deal 1.75, 2.13". The real-time search engine 28 sends record 91e (e.g., concatenated record) to record generator 32 (e.g., on-demand record).

[0044] The on-demand record generator 32 sends a record (e.g., a concatenated record) "Key 7342 Credit 534 Coffee Transaction 1.75, 2.13" to the batch repository 30. The batch repository 30 uses the record (e.g., a concatenated record) to retrieve the age and number of children for key 7342 that was sent to the on-demand record generator 32. In another example, the on-demand record generator 32 sends a request 57 to the batch repository 30 for an entry in the lookup table 51 associated with key 7342. In response, the batch repository 30 sends entry 51a to the on-demand record generator 32. The on-demand record generator 32 generates a record 50 (e.g., a wide record) as follows: 7342,34...1 534...1.75, 2.13 Key Batch Field Real-time Field Here, record 50 (for example, a wide record) contains in the batch field of key 7342 the age "34" and the coffee transaction "1.75, 2.13". The rule execution engine 34 executes business rule 95, "(age * 20+ Number of children * Number of transactions - credits * If .087) > 20, execute "Send offer". Regarding key 7342, it is confirmed that key 7342 is eligible for the offer. The confirmation 53 is sent to the computing system 18, which generates an offer 55 that is sent to the client device 17 and received by the client device 17.

[0045] Figures 4A and 4B show real-world examples without batch updates.

[0046] Referring to Figure 4A, the operating system 16 stores records 60a', 60b, 60c, and 60d' as described above. In this example, the batch repository 30 stores the lookup table 51 as described above and as a result of batch searches performed on the operating system 16. The rule execution engine 34 stores rules 34a and 34b.

[0047] Next, referring to Figure 4B, the client device 17 sends request 61 to the computing system 18 (at 1:10 p.m.). In this example, request 61 asks whether the computing system 18 should send an offer to the customer associated with key 7342. The computing system 18 receives this request 61 and sends it to the rule execution engine 34.

[0048] Before the rule execution engine 34 can execute rules, it needs to receive a record of data associated with key 7342 (e.g., a wide record). As described herein, some of the information in the record (e.g., a wide record) will be batch data, and some will be real-time information. To initiate the process of receiving the record (e.g., a wide record), the rule execution engine 34 sends a request for a record 89 (e.g., a dynamic record) to the on-demand record generator 32. The on-demand record generator 32 sends an instruction 65 to the real-time search engine 28 requesting a record for key 7342 (e.g., a real-time record). The real-time search engine 28 sends a request 67 for a record for key 7342 (e.g., a real-time record) to the metadata manager 24. The metadata manager 24 looks up the source names of the real-time fields in the metadata repository 25, as described above. The metadata manager 24 sends a request 69 for Cns_credit and Cns_sbuxtx, key 7342 to the operating system 16 to look up the source names of the real-time field credits and coffee transactions. The operating system 16 returns records 64 and 68, with the keys "7342 Cns_credit 534" and "7342 Cns_sbuxtx 1.75, 2.13". The metadata manager 24 looks up the business names in the returned fields, and the metadata repository 25 returns records 87 and 79, with the keys "7342 credit 534" and "7342 coffee deal 1.75, 2.13". Records 87 and 79, which have the business names "7342 credit 534" and "7342 coffee deal 1.75, 2.13" in the returned fields, are sent by the metadata manager 24 to the real-time search engine 28. The real-time search engine 28 generates record 97 (e.g., a concatenated record), with the keys "7342 credit 534 coffee deal 1.75, 2.13".

[0049] The on-demand record generator 32 sends batch request 30a (or record 97 (e.g., concatenated record) "key 7342 credit 534 coffee deal 1.75, 2.13") to the batch repository 30. The batch repository 30 uses record 97 (e.g., concatenated record) to retrieve record 83 for "key 7342" (e.g., age "34" and child "1" for key 7342), which is sent to the on-demand record generator 32. Using record 83 and record 97 (e.g., concatenated record), the on-demand record generator 32 generates record 50 (e.g., wide record) as follows: 7342,34...1 534...1.75, 2.13 Key Batch Field Real-time Field

[0050] In this example, real-time fields are retrieved in response to request 61. That is, real-time fields are retrieved in real time with respect to when request 61 is received. In contrast, batch fields are retrieved previously from the operational system 16 at a time specified for, for example, batch lookup. Batch fields are retrieved from the batch repository 30, for example, as shown in the first entry of table 51. The rule execution engine 34 executes business rules, If the total value of coffee transactions in the last 24 hours is greater than 5, send 50 loyalty reward points. Regarding key 7342, it is not confirmed that key 7342 is eligible for the 50 loyalty reward points offer. The non-confirmation is sent to computing system 18, and computing system 18 does not generate an offer. Next, Figure 4B shows Rule 2, Test the following: "If age > 25, grant a credit card offer."

[0051] In this example, the customer is 34 years old, and the rule execution engine 34 executes business rules to determine that key 7342 is eligible for the credit card offer. The confirmation 93 is sent to the computing system 18, which generates an offer 85 to be sent to the client device 17.

[0052] Figures 4C to 4D show a second real-world example that does not involve batch updates by the batch search engine 26.

[0053] Next, referring to Figure 4C, the operating system 16 receives new records, namely record 101 (e.g., a transaction record) at 1:54 p.m. and record 103 (e.g., a child record) at 12:45 p.m. In response, the operating system 16 updates its database and records (e.g., data records) as shown in portions 105 and 107 of record 122 (e.g., a data record).

[0054] At 1:55 p.m., client device 17 sends request 109 to computing system 18. In this example, the request is whether the customer should receive the offer. Therefore, request 109 specifies "Key: 7342 Should I send the offer?". In this example, request 109 is keyed to specify the users to whom the offer may be sent. Computing system 18 receives this request 109 and sends the request "Key: 7342 Should I send the offer?" to rule execution engine 34.

[0055] Next, referring to Figure 4D, the rule execution engine 34 sends request 111 to the on-demand record generator 32. The on-demand record generator 32 sends instruction 113 to the real-time search engine 28 requesting a record for key 7342 (e.g., a real-time record). The real-time search engine 28 sends request 115 to the metadata manager 24 for a record for key 7342 (e.g., a real-time record). The metadata manager 24 looks up the source name of the real-time field in the metadata repository 25. The metadata manager 24 sends request 124 to the operating system 16 for Cns_credit and Cns_sbuxtx, key 7342, in order to look up the source names of the real-time field credit and coffee transaction. The operating system 16 returns record 117, "key 7342 Cns_credit 534" and "key 7342 Cns_sbuxtx 1.75, 2.13, 6.33". The metadata manager 24 looks up the business name of the returned field, and the metadata repository 25 returns record 119 with the key 7342 credits 534 and the key 7342 coffee deal 1.75, 2.13, 6.33. Record 119, which has the business name of the returned field 7342 credits 534 and the key 7342 coffee deal 1.75, 2.13, 6.33, is sent by the metadata manager 24 to the real-time search engine 28. The real-time search engine 28 generates record 121 (e.g., a concatenated record) with the key 7342 credits 534 coffee deal 1.75, 2.13, 6.33.

[0056] The on-demand record generator 32 sends batch request 123 to the batch repository 30. Batch request 123 is keyed for key 7342. The batch repository 30 uses the key in batch request 123 to retrieve record 125 for key 7342, which is the age and child of key 7342, and sends it to the on-demand record generator 32. The batch data contained in record 125 does not include the latest data for the child field. The latest data for the child field was received at 12:45 p.m. before request 109 was sent. However, since the child field is a batch field, it is only updated at a predetermined time, for example, every night. At the time batch request 123 is sent, the nightly batch processing has not yet occurred, and as a result, the child field is outdated. However, in the context of this example, the fact that the child field is outdated is acceptable because the fields where it was important that those fields be current were labeled as real-time fields. Other fields were labeled as batch fields. When labeling these fields as batch fields, it has already been decided that it is not important for these fields to be up-to-date. The on-demand record generator 32 generates record 50' (e.g., a wide record) as follows: 7342,34...1 534...1.75, 2.13, 6.33 Key Batch Field Real-time Field

[0057] Record 50' (e.g., a wide record) contains the expired batch fields as described above. Record 50' (e.g., a wide record) also contains the current and most recent real-time fields as of when request 109 is sent. This example demonstrates how specifying in advance which fields are real-time fields and which are batch fields saves system resources (e.g., processing resources and memory resources) by eliminating the need for the data processing system 20 to continuously generate the latest records (e.g., data records) and to retrieve the real-time values ​​of all fields. Rather, at request, the system only needs to retrieve the values ​​of a subset of the fields used by the rule execution engine 34. These subsets of fields are the real-time fields.

[0058] This technique is highly effective when records (e.g., real-time data records) are not frequently needed, but when records (e.g., real-time data records) are required, it is crucial that certain fields are up-to-date in real-time with respect to when the request is received. Therefore, these techniques result in improved computational efficiency for calculating records (e.g., real-time records) in response to requests, because records do not need to be updated continuously, but rather only when needed.

[0059] Furthermore, the system described herein generates records in response to requests when a certain amount of latency (approximately 1 ms) is acceptable when providing records. In other examples, latency is unacceptable when generating records. In these unacceptable latency examples, records (e.g., wide records) are constantly updated even when records are not being used and the trade-off (for constantly updating records) is increased consumption of processing power and memory resources. However, when this certain amount of latency is acceptable, the system described herein takes advantage of the fact that it can generate records (e.g., wide records, also called dynamic records) on demand using real-time data retrieved from the operational system, thereby improving memory consumption and computing resources (compared to the memory consumption and computing resources when records need to be constantly updated) due to the fact that only the information actually requested is retrieved from the operational system and used when updating records (e.g., data records). Furthermore, classifying data fields as either real-time fields or batch fields reduces the amount of latency when generating records in response to requests (compared to the amount of latency when retrieving all data from the operational system in real time). When configuring the system, this classification is performed such that data of relatively high importance (for example, for rule execution) or frequently changing data is classified as "real-time," and data that does not change as frequently or is not very important for real-time decisions (through rule execution) is classified as batch data. Preferably, wide records contain only the values ​​associated with a given key, thereby reducing the consumption of computational resources by generating wide records with values ​​associated with a given key in response to a request compared to the amount of computational resources consumed by continuously generating wide records with values ​​associated with all keys. Accordingly, the techniques described herein reduce latency and memory resources when generating these records (e.g., wide records), while improving computational efficiency in doing so. In this case, the rule execution engine 34 executes the business rules. If the total value of coffee transactions in the last 24 hours is greater than 5, send 50 loyalty reward points. Regarding key 7342, confirm that key 7342 is eligible for the 50 loyalty reward points offer. Confirmation 127 is sent to computing system 18, which generates offer 129.

[0060] Next, referring to Figure 5A, the legend for Figure 5 is shown.

[0061] Next, referring to Figure 5, a timeline 70 of the operations described above is shown. The operating system 16 receives data (72) and updates records and / or memory and / or databases with the received data (74). The updated received data is stored in the operating system 16. The batch search engine 26 requests 78 batch data (e.g., from the metadata manager 24), and the metadata manager 24 fetches 80 field names (e.g., source names) of type==batch from the metadata repository 25 (not shown). The metadata manager 24 then sends a request to the operating system 16 for the data associated with the fetched source names. The requested data (associated with type==batch) is sent by the operating system 16 to the metadata manager 24 (76), and the metadata manager 24 relabels the field names of the requested data (82). For example, the field name Cns_age is relabeled as age. The batch search engine 26 receives data with relabeled field names, combines the relabeled batches (84) (for example, by combining data with relabeled field names into a single profile or data record), sends the combined relabeled batches to the batch repository 30, which stores the combined relabeled batches (86).

[0062] Alternatively, the client device 17 sends an offer request to the computing system 18 (90). The computing system 18 responds to the offer request by sending an eligibility request to the rule execution engine 34 (92). The rule execution engine 34 then sends a request for records (e.g., RT (real-time) records) to the on-demand record generator 32 (94). The on-demand record generator 32 executes an instruction to request RT records and sends the request to the real-time search engine 28 (96). The real-time search engine 28 sends a request for RT data to the metadata manager 24 (98). The metadata manager 24 looks up which fields are real-time fields and fetches the values ​​of those real-time fields from the operational system 16 (99). The operational system 16 responds to the fetch RT request by sending 100 RT data to the metadata manager 24, which then relabels 102 RT data (e.g., by updating technical field names with business field names) and sends the relabeled RT data to the real-time search engine 28. The real-time search engine 28 combines the RT data into, for example, a single record (104). The on-demand record generator 32 receives the combined RT data (106) and sends the combined RT data back to the on-demand record generator 32 as the first input.

[0063] Alternatively, the on-demand record generator 32 also sends a request to the batch repository 30 to retrieve batch data (108). The batch repository 30 sends back batch data (e.g., relabeled, combined batch data) to the on-demand record generator 32 (110), and the on-demand record generator 32 receives the batch data as a second input used when generating records, e.g., on-demand records (112). The on-demand record generator 32 generates records (e.g., RT records) based on the received RT data and received batch data sent to the on-demand record generator 32 as the first and second inputs, respectively (114). Based on the generated records 114 (e.g., RT records), the rule execution engine 34 executes one or more rules (116). The computing system 18 receives the results of the rule execution, and for rules that have successfully qualified for the offer, the computing system 18 sends an offer 118 (118), and the client devices 17 120 receive the offer.

[0064] Referring now to Figure 6, a process 150 is shown which is performed by the system 10, in particular by the data processing system 20 for performing real-time decisions. The process 150 generates a record (e.g., a dynamic record) based on a request for a real-time decision (152). The record (e.g., a dynamic record) includes batch data and real-time data retrieved from the operational system 16 in response to the receipt of the request, and the record (e.g., a real-time record) relates to the time when the request was received by the data processing system 20. The process 150 is triggered by receiving a request 154 for a record (e.g., a data record) associated with a given key. The process 150 retrieves batch fields stored locally in the batch repository 30 of the data processing system 20 (156) which have one or more first values ​​for one or more first fields identified as one or more batch fields, and the one or more retrieved first values ​​are associated with a given key. Process 150 fetches real-time fields by requesting the operational system 16 158 one or more second values ​​of one or more second fields identified as one or more real-time fields, along with one or more retrieved second values ​​associated with a given key. Process 150 receives those real-time fields from the operational system 16 along with the requested one or more second values ​​associated with a given key.

[0065] Process 150 generates a record 50 (e.g., a wide record) having a requested record (e.g., a data record) for a given key, the record 50 (e.g., a wide record) comprising one or more first values ​​of one or more first fields identified as one or more batch fields, and one or more second values ​​of one or more second fields identified as one or more real-time fields.

[0066] Process 150 applies one or more rules to an generated record (e.g., a wide data record), which includes determining whether one or more conditions of one or more rules are met at least partially on one or more second values ​​of one or more second fields identified as one or more real-time fields (162). Process 150 determines whether to send an offer based on the application of one or more rules (164) and whether to instruct an external system to perform one or more actions specified by one or more rules for a given key.

[0067] Another aspect of process 150 includes determining from the metadata repository that one or more first fields are identified as one or more batch fields (not shown).

[0068] Another aspect of process 150 includes, in response to a request, determining from a metadata repository that one or more second fields are identified as one or more real-time fields (not shown).

[0069] Another aspect of process 150 includes retrieving multiple values ​​of one or more first fields identified as one or more batch fields from one or more operating systems, wherein the multiple values ​​are associated with multiple keys, the multiple values ​​contain one or more first values, and the multiple keys contain a given key (not shown).

[0070] Another aspect of process 150 includes storing multiple values ​​associated with multiple keys in a repository of the data processing system (not shown). [Examples]

[0071] Exemplary computing environment Referring to Figure 7, an exemplary operating environment for implementing an embodiment of the present invention is shown, collectively designated as a computing device 180. Essential elements of the computing device 180, or a computer, data processing system, client, or server, are one or more programmable processors 182 for performing actions according to instructions, and one or more memory devices 184 for storing instructions and data. Generally, the computer also includes, or is operably coupled to (via a bus 192, fabric, network, network, etc.) I / O components 186, such as a display device, a network / communication subsystem (not shown), and one or more mass storage devices 188 for storing data and instructions, etc., and a network / communication subsystem 190 powered by a power supply (not shown). The memory device 184 contains an operating system 184a and applications 184b for application programming, such as implementing some or all of a process 150.

[0072] Devices suitable for computer program products, including tangible, non-temporary computer storage media on which computer program instructions and data are stored, include, for example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or incorporated into dedicated logic circuits.

[0073] To provide user interaction, embodiments of the subject matter described herein are implemented on a computer having a display device (monitor) for displaying information to the user, a keyboard that allows the user to provide input to the computer, and a pointing device (e.g., a mouse or trackball). Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user (for example, by sending a web page to a web browser on the user's user device in response to a request received from a web browser).

[0074] Embodiments of the subject matter described herein can be implemented in a computing system that includes a backend component (e.g., as a data server), a middleware component (e.g., an application server), a frontend component (e.g., a user computer having a graphical user interface or a web browser on which a user can interact with the implementation of the subject matter described herein), or any combination of one or more such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0075] A computing system may include user devices, such as client devices and servers. Client devices and servers are typically geographically separated and usually interact through a communication network. The relationship between the client device and the server is established by computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (for example, to display data to a user interacting with the client device and to receive user input from a user interacting with the user device). Data generated on the client device (e.g., the results of user interaction) can be received by the server from the client device.

[0076] This specification includes details of many specific implementations, but these should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features specific to a particular embodiment of a particular invention.

[0077] Similarly, while the operations are depicted in a specific order in the drawings, it should not be understood that such operations must be performed in the specific order shown or in a sequential order, or that all exemplified operations must be performed, in order to achieve the desired result. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0078] Several embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the techniques described herein. For example, some of the above steps may be sequentially independent and can therefore be performed in an order different from that described. Furthermore, any of the aforementioned techniques described in relation to data flow graphs can also be implemented and performed in relation to programs. Therefore, other embodiments are also within the scope of the following claims.

Claims

1. A method implemented by a data processing system for performing a real-time decision by generating a data record based on a request for a real-time decision, wherein the data record includes batch data and real-time data retrieved from one or more operating systems in response to the receipt of the request, and the real-time refers to the time when the request is received by the data processing system. Receiving a request for the data record associated with a given key, The request involves retrieving from the repository of the data processing system one or more first values ​​of one or more first fields that have been identified as batch fields retrieved in batches from one or more operating systems prior to the request and stored in the repository of the data processing system, wherein the one or more retrieved first values ​​are associated with the given key. In response to receiving the aforementioned request, to request from one or more operating systems one or more second values ​​of one or more second fields identified as one or more real-time fields stored by the one or more operating systems and retrieved in real time from the one or more operating systems, wherein the requested one or more second values ​​are associated with the given key, Receiving the requested one or more second values ​​associated with the given key from one or more of the aforementioned operating systems, To generate the requested data record for the given key, wherein the data record is: The one or more first values ​​of the one or more first fields identified as the one or more batch fields, The one or more second values ​​of the one or more second fields identified as the one or more real-time fields, Applying one or more rules to the generated data record, including determining whether one or more conditions of the one or more rules are met, based at least partially on the one or more first values ​​of the one or more first fields identified as batch fields and the one or more second values ​​of the one or more second fields identified as real-time fields; A method comprising: if one or more of the conditions of the one or more rules are met, generating an instruction for the given key to perform one or more actions specified by the one or more rules.

2. The method according to claim 1, comprising reading a table from memory having cells that specify fields as batch fields and fields as real-time fields, and sending one or more signals specifying a request to one or more operating systems in response to the read.

3. The method according to claim 1, further comprising determining from a metadata repository that the one or more first fields are identified as the one or more batch fields.

4. The method according to claim 1, wherein the batch field is a field that is likely to change at a first frequency, and the real-time field is a field that is likely to change at a second frequency that is higher than the first frequency.

5. The method according to claim 1, wherein the batch field is a field that is updated intermittently and / or at predetermined intervals.

6. The method according to claim 1, wherein the real-time field is a field that is updated in real time.

7. The method according to claim 1, wherein the data record is generated only in response to the receipt of the request for the data record.

8. The method according to claim 1, wherein the requested one or more second values ​​are received from the operating system only in response to the receipt of the request for the data record.

9. The method according to claim 1, wherein the sole data requested and / or received from the operating system to generate the requested data record for the given key is one or more requested second values ​​associated with the given key.

10. The method according to claim 1, wherein the data record includes only the value associated with the given key.

11. The method according to claim 1, further comprising determining from a metadata repository that, in response to the request, the one or more second fields are identified as the one or more real-time fields.

12. The method according to claim 1, further comprising retrieving from one or more operating systems a plurality of values ​​of the one or more first fields identified as the one or more batch fields, wherein the plurality of values ​​are associated with a plurality of keys, the plurality of values ​​include the one or more first values, and the plurality of keys include the given keys.

13. The method according to claim 1, further comprising storing the plurality of values ​​in association with the plurality of keys in the repository of the data processing system.

14. A data processing system for performing real-time decisions by generating data records based on requests for real-time decisions, wherein the data records include batch data and real-time data retrieved from one or more operating systems in response to the receipt of the request, and the real-time refers to the time when the request is received by the data processing system. One or more processing devices, The system includes one or more machine-readable hardware storage devices that store instructions executable by one or more processing devices in order to perform an operation, wherein the operation is Receiving a request for the data record associated with a given key, The request involves retrieving from the repository of the data processing system one or more first values ​​of one or more first fields identified as batch fields retrieved in batch from one or more operating systems prior to the request and stored in the repository of the data processing system, wherein the one or more retrieved first values ​​are associated with the given key. In response to receiving the aforementioned request, to request from one or more operating systems one or more second values ​​of one or more second fields identified as one or more real-time fields stored by the one or more operating systems and retrieved in real time from the one or more operating systems, wherein the requested one or more second values ​​are associated with the given key, Receiving the requested one or more second values ​​associated with the given key from one or more of the aforementioned operating systems, To generate the requested data record for the given key, wherein the data record is: The one or more first values ​​of the one or more first fields identified as the one or more batch fields, The one or more second values ​​of the one or more second fields identified as the one or more real-time fields, Applying one or more rules to the generated data record, including determining whether one or more conditions of the one or more rules are met, based at least partially on the one or more first values ​​of the one or more first fields identified as batch fields and the one or more second values ​​of the one or more second fields identified as real-time fields; The data processing system according to claim 1, comprising: generating an instruction for a given key to perform one or more actions specified by the one or more rules if one or more of the conditions of the one or more rules are met.

15. The aforementioned operation is, The data processing system according to claim 14, further comprising: reading a table from memory having cells that specify fields as batch fields and fields as real-time fields; and sending one or more signals specifying a request to one or more operating systems in response to the read.

16. The aforementioned operation is, The data processing system according to claim 14, further comprising determining from a metadata repository that the one or more first fields are identified as the one or more batch fields.

17. The data processing system according to claim 14, wherein the batch field is a field that is likely to change at a first frequency, and the real-time field is a field that is likely to change at a second frequency that is higher than the first frequency.

18. The data processing system according to claim 14, wherein the batch field is a field that is updated intermittently and / or at predetermined intervals.

19. The data processing system according to claim 14, wherein the real-time field is a field that is updated in real time.

20. The data processing system according to claim 14, wherein the real-time data record is generated only in response to the receipt of the request for the data record.

21. One or more machine-readable hardware storage devices for performing real-time decisions by generating data records based on requests for real-time decisions, wherein the data records include batch data and real-time data retrieved from one or more operating systems in response to the receipt of a request, where real-time refers to the time when the request is received by a data processing system, and the one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform an operation, the operation is Receiving a request for the data record associated with a given key, The request involves retrieving from the repository of the data processing system one or more first values ​​of one or more first fields that have been identified as batch fields retrieved in batches from one or more operating systems prior to the request and stored in the repository of the data processing system, wherein the one or more retrieved first values ​​are associated with the given key. In response to receiving the aforementioned request, to request from one or more operating systems one or more second values ​​of one or more second fields identified as one or more real-time fields stored by the one or more operating systems and retrieved in real time from the one or more operating systems, wherein the requested one or more second values ​​are associated with the given key, Receiving the requested one or more second values ​​associated with the given key from one or more of the aforementioned operating systems, To generate the requested data record for the given key, wherein the data record is: The one or more first values ​​of the one or more first fields identified as the one or more batch fields, The one or more second values ​​of the one or more second fields identified as the one or more real-time fields, Applying one or more rules to the generated data record, including determining whether one or more conditions of the one or more rules are met, based at least partially on the one or more first values ​​of the one or more first fields identified as batch fields and the one or more second values ​​of the one or more second fields identified as real-time fields; One or more machine-readable hardware storage devices, which include generating an instruction for a given key to perform one or more actions specified by the one or more rules if one or more of the conditions of the one or more rules are met.

Images