Feature calculation batch request processing method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure relate to a feature calculation batch request processing method and device, electronic equipment and a storage medium. The method comprises: receiving a feature calculation batch request, the batch request comprising a plurality of data rows, the plurality of data rows sharing a calculation target and input field information, the input field information being used to specify information of a plurality of columns included in the data rows; generating a first execution plan based on the input field information and the calculation target, the first execution plan comprising a plurality of calculation steps and input and output of each calculation step; determining information of one or more common columns in the input field information, and optimizing the calculation steps and / or the input of the calculation steps in the first execution plan based on the information of the common columns to obtain a second execution plan; and obtaining a batch calculation result of the batch request based on the second execution plan. It can be seen that the execution plan of the batch request is generated but not directly executed, but combined with artificial heuristic information, i.e. common column information, the execution plan is optimized before execution, thereby improving the processing efficiency of the batch request.

Description

[0001] This application is a divisional application of the granted patent (patent application number 2020115537344). Technical Field

[0002] This disclosure relates to the field of artificial intelligence technology, specifically to a batch request processing method, apparatus, electronic device, and storage medium for feature calculation. Background Technology

[0003] With the development of artificial intelligence, feature computation is a crucial step in processing online service requests (such as online recommendation requests) using machine learning. In specific service scenarios (such as recommendation scenarios), online feature computation requests arrive at the computing system in batches.

[0004] Currently, the naive approach to batch processing of feature computation requests is to iterate through each request in the batch, call the internal computation logic for each request individually, and finally merge all the computation results together and return them. This approach has the following drawbacks:

[0005] (1) Input and output data are too large

[0006] A batch request input contains multiple request inputs, and the return result contains the calculation results of multiple requests. When the batch size is large (i.e., the batch contains a large number of requests), it will generate high data transmission pressure.

[0007] (2) Large computational load and low efficiency

[0008] A batch of input requests contains multiple input requests. When the batch size is large, iterating through each request in the batch and calling the internal calculation logic for each request separately results in a large amount of computation and low efficiency.

[0009] The above description of the problem discovery process is only for the purpose of assisting in understanding the technical solution of this disclosure, and does not represent an admission that the above content is prior art. Summary of the Invention

[0010] To address at least one problem existing in the prior art, at least one embodiment of this disclosure provides a batch request processing method, apparatus, electronic device, and storage medium for feature computation.

[0011] In a first aspect, embodiments of this disclosure propose a batch request processing method for feature calculation, including:

[0012] Receive a batch request for feature calculation; wherein the batch request includes multiple data rows, the multiple data rows share calculation target and input field information, and the input field information is used to specify information of multiple columns included in the data row;

[0013] A first execution plan is generated based on the input field information and the calculation target; wherein, the first execution plan includes multiple calculation steps and the input and output of each calculation step;

[0014] Determine the information of one or more common columns in the input field information, and optimize the calculation steps and / or the input of the calculation steps in the first execution plan based on the information of the common columns to obtain a second execution plan;

[0015] Based on the second execution plan, the batch calculation results of the batch request are obtained.

[0016] In some embodiments, determining the information for one or more common columns in the input field information includes:

[0017] Information about one or more common columns in the input field information is obtained through a pre-established common column interface;

[0018] or,

[0019] Based on the multiple data rows included in the batch request, information from one or more common columns in the input field information is determined.

[0020] In some embodiments, optimizing the computation steps and / or the inputs of the computation steps in the first execution plan based on the information from the common columns to obtain the second execution plan includes:

[0021] For each computation step in the first execution plan:

[0022] Determine whether the calculation step satisfies the first optimization condition, wherein the first optimization condition is:

[0023] This calculation step is based on data from one or more of the common columns, and not on data from any of the non-common columns;

[0024] If the first optimization condition is met, then the input for optimizing this calculation step is the data in one or more of the common columns.

[0025] In some embodiments, if it is determined that the calculation step does not satisfy the first optimization condition, then:

[0026] Determine whether the calculation step satisfies the second optimization condition, wherein the second optimization condition is: the calculation step is performed based on data in one or more of the common columns and on data in one or more non-common columns;

[0027] If the second optimization condition is met, the calculation step is optimized to at least one first sub-step and at least one second sub-step; wherein the first sub-step is calculated based only on data in one or more of the common columns; and the second sub-step is calculated based at least on data in one or more non-common columns.

[0028] In some embodiments, the calculation step is time window retrieval, the input field information includes a primary key column and a time column, and the time window retrieval is based on the data in the primary key column and the data in the time column to retrieve window data that conforms to the calculation target;

[0029] If the primary key column and the time column are both common columns, then the input for optimizing the time window pull is the data in the primary key column and the data in the time column.

[0030] In some embodiments, the calculation step is table concatenation, wherein the table concatenation is performed on the left table and the right table based on concatenation conditions, wherein the left table is obtained based on the batch request, and the right table is obtained based on the table stored in the database;

[0031] If the concatenation condition is a common column, then the table concatenation is optimized into one first sub-step and two second sub-steps;

[0032] The first sub-step is used to: perform a first concatenation based on the concatenation conditions on the data of all common columns in the left table and the data of all common columns in the right table;

[0033] The two second sub-steps include: a right splicing sub-step and a left splicing sub-step;

[0034] The right concatenation sub-step is used to: perform a second concatenation based on the unique index of the right table on the output of the first sub-step and the data of all non-common columns in the right table;

[0035] The left concatenation sub-step is used to: perform a third concatenation on the output of the right concatenation sub-step and the data of all non-common columns in the left table;

[0036] The output of the left splicing sub-step serves as the calculation result of the table splicing.

[0037] In some embodiments, the calculation step is an aggregate calculation, which generates multiple calculation results, each of which is a common calculation result, a first non-common calculation result, or a second non-common calculation result.

[0038] The common calculation result is calculated based solely on data from one or more of the common columns; the first non-common calculation result is calculated based on data from one or more of the common columns and data from one or more non-common columns; and the second non-common calculation result is calculated based on data from one or more non-common columns.

[0039] In some embodiments, the aggregation computation is optimized into multiple first sub-steps and multiple second sub-steps; wherein the multiple first sub-steps include at least one first common computation step and at least one second common computation step; and the multiple second sub-steps include at least one first non-common computation step and at least one second non-common computation step.

[0040] The input to the first common calculation step is data from one or more of the common columns, and the output is common intermediate data;

[0041] The input to the second common calculation step is data from one or more of the common columns, and the output is the common calculation result;

[0042] The input to the first non-public calculation step is one or more public intermediate data output by the first public calculation step and data from one or more non-public columns, and the output is the first non-public calculation result.

[0043] The input to the second non-public calculation step is data from one or more non-public columns, and the output is the result of the second non-public calculation.

[0044] In some embodiments, after determining the information of one or more common columns in the input field information, the method further includes:

[0045] Establish a shared storage area;

[0046] Public data is stored in the shared storage area; wherein, the public data includes: data in the public columns, public intermediate data, and public calculation results;

[0047] The common intermediate data refers to the intermediate data generated by the first common calculation step in the second execution plan, and the first common calculation step is a step of generating intermediate data based on data in one or more of the common columns.

[0048] The common calculation result is the calculation result generated by the second common calculation step in the second execution plan, and the second common calculation step is a step that generates a calculation result based on data in one or more of the common columns.

[0049] In some embodiments, obtaining the batch computation result of the batch request based on the second execution plan includes:

[0050] For each data row in the batch request, the second execution plan is executed to obtain the calculation result of each data row, and then the results are merged into the batch calculation result of the batch request.

[0051] In some embodiments, when executing the second execution plan:

[0052] For the first common computation step in the second execution plan:

[0053] Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area;

[0054] If the identifier of the first common computation step is found, the first storage location corresponding to the identifier of the first common computation step is determined from the index relationship;

[0055] Based on the first storage location, data is obtained from the shared storage area as public intermediate data generated by the first public computing step.

[0056] In some embodiments, the method further includes:

[0057] If the identifier of the first common calculation step cannot be found, then common intermediate data is generated based on the data in the common column through the first common calculation step;

[0058] The public intermediate data generated by the first public computing step is stored in the shared storage area, and the relationship between the identifier of the first public computing step and its storage location is recorded in the index relationship.

[0059] In some embodiments, when executing the second execution plan:

[0060] Regarding the second common computation step in the second execution plan:

[0061] Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area;

[0062] If the identifier of the second common computation step is found, the second storage location corresponding to the identifier of the second common computation step is determined from the index relationship;

[0063] Based on the second storage location, data is obtained from the shared storage area as the public computation result generated by the second public computation step.

[0064] In some embodiments, the method further includes:

[0065] If the identifier of the second common calculation step cannot be found, a common calculation result is generated based on the data in the common column through the second common calculation step.

[0066] The public computation result generated by the second public computation step is stored in the shared storage area, and the relationship between the identifier of the second public computation step and its storage location is recorded in the index relationship.

[0067] In some embodiments, the query index relationship includes:

[0068] The index relationship is queried through a pre-established public data cache interface.

[0069] In some embodiments, storing public data in the shared storage area includes:

[0070] Public data is stored in the shared storage area through a pre-established data encoding and storage interface.

[0071] Secondly, embodiments of this disclosure also provide a batch request processing apparatus for feature calculation, comprising:

[0072] A receiving unit is configured to receive a batch request for feature calculation; wherein the batch request includes multiple data rows, the multiple data rows share calculation target and input field information, and the input field information is used to specify information of multiple columns included in the data rows;

[0073] A generation unit is configured to generate a first execution plan based on the input field information and the calculation target; wherein the first execution plan includes multiple calculation steps and the input and output of each calculation step;

[0074] An optimization unit is configured to determine information from one or more common columns in the input field information, and optimize the calculation steps and / or the input of the calculation steps in the first execution plan based on the information from the common columns, to obtain a second execution plan;

[0075] A calculation unit is used to obtain the batch calculation result of the batch request based on the second execution plan.

[0076] In some embodiments, the optimization unit determines information from one or more common columns in the input field information, including:

[0077] Information about one or more common columns in the input field information is obtained through a pre-established common column interface;

[0078] or,

[0079] Based on the multiple data rows included in the batch request, information from one or more common columns in the input field information is determined.

[0080] In some embodiments, the optimization unit optimizes the computation steps and / or the inputs of the computation steps in the first execution plan based on the information of the common column to obtain a second execution plan, including:

[0081] For each computation step in the first execution plan:

[0082] Determine whether the calculation step satisfies the first optimization condition, wherein the first optimization condition is:

[0083] This calculation step is based on data from one or more of the common columns, and not on data from any of the non-common columns;

[0084] If the first optimization condition is met, then the input for optimizing this calculation step is the data in one or more of the common columns.

[0085] In some embodiments, if the optimization unit determines that the calculation step does not satisfy the first optimization condition, then:

[0086] Determine whether the calculation step satisfies the second optimization condition, wherein the second optimization condition is: the calculation step is performed based on data in one or more of the common columns and on data in one or more non-common columns;

[0087] If the second optimization condition is met, the calculation step is optimized to at least one first sub-step and at least one second sub-step; wherein the first sub-step is calculated based only on data in one or more of the common columns; and the second sub-step is calculated based at least on data in one or more non-common columns.

[0088] In some embodiments, the calculation step is time window retrieval, the input field information includes a primary key column and a time column, and the time window retrieval is based on the data in the primary key column and the data in the time column to retrieve window data that conforms to the calculation target;

[0089] If the primary key column and the time column are both common columns, then the optimization unit optimizes the input of the time window pull as the data in the primary key column and the data in the time column.

[0090] In some embodiments, the calculation step is table concatenation, which concatenates the left table and the right table based on concatenation conditions, wherein the left table is obtained based on the batch request, and the right table is obtained based on the table stored in the database;

[0091] If the concatenation condition is a common column, then the optimization unit optimizes the table concatenation into one first sub-step and two second sub-steps;

[0092] The first sub-step is used to: perform a first concatenation based on the concatenation conditions on the data of all common columns in the left table and the data of all common columns in the right table;

[0093] The two second sub-steps include: a right splicing sub-step and a left splicing sub-step;

[0094] The right concatenation sub-step is used to: perform a second concatenation based on the unique index of the right table on the output of the first sub-step and the data of all non-common columns in the right table;

[0095] The left concatenation sub-step is used to: perform a third concatenation on the output of the right concatenation sub-step and the data of all non-common columns in the left table;

[0096] The output of the left splicing sub-step serves as the calculation result of the table splicing.

[0097] In some embodiments, the calculation step is an aggregate calculation, which generates multiple calculation results, each of which is a common calculation result, a first non-common calculation result, or a second non-common calculation result.

[0098] The common calculation result is calculated based solely on data from one or more of the common columns; the first non-common calculation result is calculated based on data from one or more of the common columns and data from one or more non-common columns; and the second non-common calculation result is calculated based on data from one or more non-common columns.

[0099] In some embodiments, the optimization unit optimizes the aggregation calculation into a plurality of first sub-steps and a plurality of second sub-steps; wherein the plurality of first sub-steps include at least one first common calculation step and at least one second common calculation step; and the plurality of second sub-steps include at least one first non-common calculation step and at least one second non-common calculation step.

[0100] The input to the first common calculation step is data from one or more of the common columns, and the output is common intermediate data;

[0101] The input to the second common calculation step is data from one or more of the common columns, and the output is the common calculation result;

[0102] The input to the first non-public calculation step is one or more public intermediate data output by the first public calculation step and data from one or more non-public columns, and the output is the first non-public calculation result.

[0103] The input to the second non-public calculation step is data from one or more non-public columns, and the output is the result of the second non-public calculation.

[0104] In some embodiments, after determining the information of one or more common columns in the input field information, the optimization unit is further configured to:

[0105] Establish a shared storage area;

[0106] Public data is stored in the shared storage area; wherein, the public data includes: data in the public columns, public intermediate data, and public calculation results;

[0107] The common intermediate data refers to the intermediate data generated by the first common calculation step in the second execution plan, and the first common calculation step is a step of generating intermediate data based on data in one or more of the common columns.

[0108] The common calculation result is the calculation result generated by the second common calculation step in the second execution plan, and the second common calculation step is a step that generates a calculation result based on data in one or more of the common columns.

[0109] In some embodiments, the computing unit is used for:

[0110] For each data row in the batch request, the second execution plan is executed to obtain the calculation result of each data row, and then the results are merged into the batch calculation result of the batch request.

[0111] In some embodiments, when the computing unit executes the second execution plan:

[0112] For the first common computation step in the second execution plan:

[0113] Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area;

[0114] If the identifier of the first common computation step is found, the first storage location corresponding to the identifier of the first common computation step is determined from the index relationship;

[0115] Based on the first storage location, data is obtained from the shared storage area as public intermediate data generated by the first public computing step.

[0116] In some embodiments, the computing unit is further configured to:

[0117] If the identifier of the first common calculation step cannot be found, then common intermediate data is generated based on the data in the common column through the first common calculation step;

[0118] The public intermediate data generated by the first public computation step is stored in the shared storage area, and the relationship between the identifier of the first public computation step and its storage location is recorded in the index relationship.

[0119] In some embodiments, when the computing unit executes the second execution plan:

[0120] Regarding the second common computation step in the second execution plan:

[0121] Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area;

[0122] If the identifier of the second common computation step is found, the second storage location corresponding to the identifier of the second common computation step is determined from the index relationship;

[0123] Based on the second storage location, data is obtained from the shared storage area as the public computation result generated by the second public computation step.

[0124] In some embodiments, the computing unit is further configured to:

[0125] If the identifier of the second common calculation step cannot be found, a common calculation result is generated based on the data in the common column through the second common calculation step.

[0126] The public computation result generated by the second public computation step is stored in the shared storage area, and the relationship between the identifier of the second public computation step and its storage location is recorded in the index relationship.

[0127] In some embodiments, the calculation unit queries the index relationship including:

[0128] The index relationship is queried through a pre-established public data cache interface.

[0129] In some embodiments, the computing unit stores public data in the shared storage area by: storing public data in the shared storage area through a pre-established data encoding storage interface.

[0130] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor and a memory; the processor executes the steps of the method as described in any embodiment of the first aspect by invoking a program or instructions stored in the memory.

[0131] Fourthly, embodiments of this disclosure also provide a non-transitory computer-readable storage medium for storing a program or instructions that cause a computer to perform the steps of the method as described in any embodiment of the first aspect.

[0132] As can be seen, in at least one embodiment of this disclosure, an execution plan for batch requests is generated but not executed directly. Instead, the relationship between multiple requests in the batch is mined by combining heuristic information, i.e., common column information, to optimize the execution plan. The optimized execution plan is then executed. Compared with the current naive processing method for batch requests, this effectively reduces the communication and computational overhead of batch request computation, thereby improving the processing efficiency of batch requests. Attached Figure Description

[0133] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.

[0134] Figure 1 This is a schematic diagram of a directed acyclic graph of the execution plan within a feature calculation service provided in this embodiment of the disclosure;

[0135] Figure 2 This is a flowchart of a batch request processing method for feature calculation provided in an embodiment of this disclosure;

[0136] Figure 3 This is a schematic diagram of the input to the calculation step in an optimized execution plan for a feature calculation service provided in this embodiment of the present disclosure;

[0137] Figure 4 This is a schematic diagram of the calculation steps themselves in an optimized execution plan for a feature calculation service provided in this embodiment of the present disclosure;

[0138] Figure 5 This is a schematic diagram of the calculation steps themselves in another feature calculation service optimization execution plan provided by this embodiment of the disclosure;

[0139] Figure 6 This is a block diagram of a batch request processing apparatus for feature calculation provided in an embodiment of this disclosure;

[0140] Figure 7 This is an exemplary block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0141] To better understand the above-described objectives, features, and advantages of this disclosure, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It is to be understood that the described embodiments are only some, not all, of the embodiments of this disclosure. The specific embodiments described herein are merely for explaining this disclosure and are not intended to limit it. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure are within the scope of protection of this disclosure.

[0142] It should be noted that in this article, relational terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0143] For ease of understanding, the terms used in the embodiments of this disclosure are explained below:

[0144] Online feature calculation

[0145] In the field of machine learning, online feature computation is carried out by a feature computation service, which is a long-running program with specific computational logic to compute specific features.

[0146] Feature computation request

[0147] A request for feature computation logically consists of three parts:

[0148] (1) Calculate the target, specifying what features to calculate as output;

[0149] (2) Input schema, which specifies which columns the input data includes and the data type of each column;

[0150] (3) Input data rows. The data format of each data row needs to match the input schema.

[0151] For example, for the feature calculation: "total number of purchases by the same user ID within 3 days", the online feature calculation request might be:

[0152] (1) Calculation objective: "Total cost of fruit purchased by the same user ID within 3 days";

[0153] (2) Input schema: User ID: int64, Time: timestamp, Purchased item: string;

[0154] (3) Input data row: (User ID = 10081, Time = 2020.8.1, Purchased goods = Fruit).

[0155] In practical applications, feature computation may be requested in batch form. This can be understood as a batch request for feature computation, which means aggregating a group of requests with the same computation goal and input schema together before requesting the feature computation service.

[0156] Implementation Plan

[0157] For a given computation target and input schema, the feature computation service internally generates specific computation steps. When a feature computation request arrives, the feature computation service performs computations on the input data rows according to the generated computation steps to obtain the return result. The execution plan is organized in the form of a directed acyclic graph (DAG), where each computation step is a node in the graph. The input edges of the node represent the input of the computation step, and the output edges of the node represent the output of the computation step.

[0158] For example, for the aforementioned feature computation request, the directed acyclic graph of the execution plan within the feature computation service is as follows: Figure 1 As shown, calculation steps 101 to 104 are included:

[0159] Calculation Step 101: Input Data Row. The data row is the data row included in the feature calculation request, for example: User ID = 10081, Time = 2020.8.1, Purchased Item = Fruit.

[0160] Calculation Step 102: Retrieve the Time Window. For example, the feature calculation service uses the user ID of the input data row as the query primary key and the time as the query time to query window data from the time series database, that is, to query the behavioral data of the same user ID within 3 days. The window data includes multiple columns, which must include at least a column for purchased items.

[0161] Calculation step 103: Table concatenation. For example, the feature calculation service is based on window data. For each row of the window data, the product price is retrieved from the product price table using the purchased product column and then concatenated.

[0162] Calculation step 104: Aggregation calculation. For example, the feature calculation service sums the values ​​in all price columns for the product = fruit based on the concatenated data.

[0163] The inventors of this disclosure have discovered that multiple requests in a batch may share the same input schema and execution plan for the computation target, and the input values ​​on some columns may also be exactly the same. Repeatedly calculating features on the same input each time results in a huge waste of computing power.

[0164] The inventors of this disclosure have also discovered that in batch request processing in recommendation scenarios, users typically possess prior domain knowledge of the input schema information of the requests; specifically, before batch request processing, users can pre-determine which columns have the same values ​​in a batch of input request rows. The inventors of this disclosure define a batch request common column as one whose value is identical across all request rows in a single batch request. The batch request common column constitutes the artificial prior knowledge required by this disclosure, i.e., artificial heuristic information.

[0165] To address this, this disclosure provides a batch request processing scheme based on feature computation. Instead of directly executing the batch request execution plan, it combines heuristic information, i.e., common column information, to mine the relationships within multiple requests in the batch, thereby optimizing the execution plan. The optimized execution plan is then executed. Compared to the current naive batch request processing method, this effectively reduces the communication and computational overhead of batch request computation, thereby improving the processing efficiency of batch requests.

[0166] Figure 2 This disclosure provides a batch request processing method for feature calculation, wherein the execution subject of the method is a feature calculation service. The method may include the following steps 201 to 204:

[0167] 201. Receive batch requests for feature calculation; wherein, the batch request includes multiple data rows, the multiple data rows share the calculation target and input field information, the input field information is used to specify the information of multiple columns included in the data row, and the input field information can be understood as input schema information.

[0168] 202. Based on the input field information and the calculation target, generate a first execution plan; wherein, the first execution plan includes multiple calculation steps and the input and output of each calculation step.

[0169] 203. Determine the information of one or more common columns in the input field information, and optimize the calculation steps and / or the input of the calculation steps in the first execution plan based on the information of the common columns to obtain the second execution plan.

[0170] In this embodiment, the feature calculation service can determine the information of the common column in a variety of ways. For example, the information of the common column can be specified manually or determined automatically.

[0171] In some embodiments, the feature calculation service obtains information about one or more common columns from the input field information through a pre-established common column interface. In this approach, the feature calculation service does not actively determine the common column information; instead, the common column information is specified manually by calling the common column interface. The common column interface is used to specify which columns in the input schema information of a batch request are common columns. In some embodiments, the common column interface can be set for all batch requests under the same calculation objective and the same input schema.

[0172] In some embodiments, the feature calculation service determines information about one or more common columns in the input field information based on the multiple data rows included in the batch request. In this manner, the feature calculation service automatically determines the information of the common columns. In some embodiments, the feature calculation service determines whether the values ​​on a certain column in the multiple data rows included in the batch request are completely identical. If they are completely identical, the column is determined as a common column.

[0173] 204. Based on the second execution plan, obtain the batch calculation results of the batch requests.

[0174] Below, examples are given for optimizing the calculation steps and / or the inputs of the calculation steps in the first execution plan mentioned in step 203.

[0175] Optimize the input of the calculation steps

[0176] In some embodiments, the feature calculation service may optimize the input of calculation steps in the first execution plan based on information from common columns. Specifically, the feature calculation service targets each calculation step in the first execution plan as follows:

[0177] Determine whether the calculation step satisfies the first optimization condition, wherein the first optimization condition is: the calculation step is based on data in one or more common columns, and not on any non-common column.

[0178] Calculate the data in the column;

[0179] If the first optimization condition is met, the input for optimizing this calculation step is data from one or more common columns.

[0180] In some embodiments, for a calculation step called time window fetching, the input fields of the batch request include a primary key column and a time column. Time window fetching retrieves window data that matches the calculation objective based on the data in the primary key column and the data in the time column. If both the primary key column and the time column are common columns, the input for "time window fetching" is optimized to include the data in the primary key column and the data in the time column.

[0181] Figure 3The inputs for the calculation steps in the feature calculation service optimization execution plan are shown. Specifically, the inputs for the "time window retrieval" step in the first execution plan of the feature calculation service optimization are shown. The "time window retrieval" step satisfies the first optimization condition, that is, the "time window retrieval" step performs calculations based on data in one or more common columns and does not perform calculations based on data in any non-common columns.

[0182] Figure 3 The diagram on the left shows the input and output of the "Time Window Fetch" step in the first execution plan. Here, "request" represents any request in the batch requests, "window" represents the "Time Window Fetch" step, "request" is the input to "window," and "window of request" is the output of "window." "Window of request" represents the window data retrieved by "window."

[0183] Figure 3 The diagram on the right shows the execution plan obtained after inputting the "Time Window Fetch" step in the first execution plan of the Feature Calculation Service Optimization. The primary key column and time column in the batch request input fields are common columns. Correspondingly, the input to the Feature Calculation Service Optimization window is the data from the common columns upon which the window performs its calculations. Figure 3 The `common` column contains data from the primary key column and the time column. The output of `window` remains unchanged, still showing `window of request`.

[0184] Figure 3 In the diagram on the right, "non-common" represents data from non-common columns. This data is used in the concat step of the first execution plan. "Common" data also participates in the concat step; therefore, the input for this step is represented by a dashed arrow to distinguish it from window input. In other words, "non-common" inputs to concat are indicated by dashed lines, meaning that non-common inputs do not enter window data.

[0185] so Figure 3 Compared to the image on the right Figure 3 The image on the left shows that the optimized window requires less input data, reducing data transmission pressure.

[0186] Optimize the calculation steps themselves

[0187] In some embodiments, the feature calculation service may optimize the calculation step itself in the first execution plan. Specifically, if the feature calculation service determines that the calculation step does not meet the first optimization condition, then:

[0188] Determine whether the calculation step satisfies the second optimization condition, wherein the second optimization condition is: the calculation step is performed based on data in one or more common columns and on data in one or more non-common columns;

[0189] If the second optimization condition is met, the calculation step is optimized to at least one first sub-step and at least one second sub-step; wherein the first sub-step is calculated based solely on data from one or more common columns; and the second sub-step is calculated based at least on data from one or more non-common columns. In some embodiments, the second sub-step may also be calculated based on common intermediate data, wherein the common intermediate data is data calculated solely based on data from common columns.

[0190] In some embodiments, for the calculation step of table concatenation, the table concatenation performs concatenation on the left and right tables based on concatenation conditions, wherein the left table is obtained based on batch requests, and the right table is obtained based on tables stored in the database. If the concatenation condition is a common column, the optimized table concatenation is divided into one first sub-step and two second sub-steps.

[0191] The first sub-step is used to perform a first concatenation based on the concatenation conditions on the data of all common columns in the left table and the data of all common columns in the right table.

[0192] The two second sub-steps include a right splicing sub-step and a left splicing sub-step.

[0193] The right concatenation sub-step is used to perform a second concatenation based on the unique index of the right table, using the output of the first sub-step and the data of all non-common columns in the right table.

[0194] The left concatenation sub-step is used to perform a third concatenation on the output of the right concatenation sub-step and the data of all non-common columns in the left table.

[0195] The output of the left concatenation sub-step is used as the calculation result of the table concatenation.

[0196] Figure 4 The diagram illustrates the calculation steps themselves in the feature calculation service optimization execution plan. Specifically, the "table concatenation" step in the first execution plan of the feature calculation service optimization is shown. The "table concatenation" step satisfies the second optimization condition, which is that the "table concatenation" performs calculations based on data in one or more common columns and on data in one or more non-common columns.

[0197] Figure 4The diagram on the left illustrates the input and output of the "table concatenation" step in the first execution plan. The "+" sign indicates the "table concatenation" step. `left` and `right` serve as the left and right tables as inputs for "table concatenation," and both `left` and `right` use common columns for concatenation, meaning the concatenation conditions are common. `new left` is the output of "table concatenation," where "table concatenation" appends `right` to `left`, forming `new left`.

[0198] Figure 4 The diagram on the right shows the execution plan obtained after the "table concatenation" step in the first execution plan of the feature calculation service optimization. Specifically, the "table concatenation" step in the feature calculation service optimization is the first sub-step 401, the right concatenation sub-step 402, and the left concatenation sub-step 403.

[0199] The first sub-step 401 performs calculations based solely on the data in the common columns. `left2` and `right2` are used as inputs to the first sub-step 401, where `left2` is the common column in `left` and `right2` is the common column in `right`. The first sub-step 401 performs the first concatenation of `left2` and `right2`. Since the first sub-step 401 calculates based only on the data in the common columns, the concatenation result output by the first sub-step 401 is called the common concatenation result, denoted as `common join`. Furthermore, since the common join participates in subsequent calculations but is not the final result, it can also be understood as common intermediate data.

[0200] The right concatenation sub-step 402 performs calculations based on both non-common and common columns. `right1` and `common join` serve as inputs to right concatenation sub-step 402, with `right1` being a non-common column in `right`. Right concatenation sub-step 402 is used to perform a second concatenation of `right1` with the `common join` through the unique index of the right table.

[0201] The left concatenation sub-step 403 performs calculations based solely on non-common columns. The outputs of left1 and right concatenation sub-step 402 serve as inputs to left concatenation sub-step 403, where left1 represents the non-common columns in left concatenation. Since the output of right concatenation sub-step 402 is calculated based on non-common columns, it contains non-common data. Left concatenation sub-step 403 then performs an unconditional third concatenation (concat) of the outputs of left1 and right concatenation sub-step 402, outputting non-common data as the result of the "table concatenation" calculation.

[0202] so Figure 4 Compared to the image on the right Figure 4 The diagram on the left shows that optimizing the "table concatenation" process reduces the input data for each sub-step, thus lowering the data transmission pressure.

[0203] In some embodiments, the calculation step is an aggregate calculation, which produces multiple calculation results, each of which is a common calculation result, a first non-common calculation result, or a second non-common calculation result.

[0204] Among them, the public calculation result is calculated based solely on the data in one or more public columns; the first non-public calculation result is calculated based on the data in one or more public columns and the data in one or more non-public columns; and the second non-public calculation result is calculated based on the data in one or more non-public columns.

[0205] In some embodiments, for a computation step that is an aggregation computation, the feature computation service optimizes the aggregation computation itself into multiple first sub-steps and multiple second sub-steps; wherein, the multiple first sub-steps include at least one first common computation step and at least one second common computation step; and the multiple second sub-steps include at least one first non-common computation step and at least one second non-common computation step.

[0206] The input to the first common calculation step is data from one or more common columns, and the output is common intermediate data.

[0207] The input to the second common calculation step is data from one or more common columns, and the output is the common calculation result.

[0208] The input to the first non-public computation step is the public intermediate data output by one or more first public computation steps and the data in one or more non-public columns, and the output is the first non-public computation result.

[0209] The input to the second non-public calculation step is data from one or more non-public columns, and the output is the result of the second non-public calculation.

[0210] Figure 5 The calculation step itself is shown in the feature calculation service optimization execution plan. Specifically, the "aggregate calculation" step itself in the first execution plan of the feature calculation service optimization is shown. The "aggregate calculation" satisfies the second optimization condition, that is, the "aggregate calculation" is calculated based on data in one or more common columns and based on data in one or more non-common columns.

[0211] Figure 5 The diagram on the left shows the inputs and outputs of the "aggregate calculation" step in the first execution plan, where proj represents the "aggregate calculation" step. Figure 5The diagram on the right shows the execution plan obtained after optimizing the "aggregate calculation" step in the first execution plan of the feature calculation service. Specifically, the "aggregate calculation" step itself is the first common calculation step proj1, the second common calculation step proj2, the first non-common calculation step concat, and the second non-common calculation step proj3.

[0212] The first common calculation step, proj1, takes as input the data (common) of the common column in input and outputs as common intermediate data (common state).

[0213] The second common calculation step, proj2, takes the data from the common column of input as input and outputs the common calculation result.

[0214] The input to the first non-common computation step concat is the non-common data in the input column and the common intermediate data (common state) output by the first common computation step proj1. The output is the result of the first non-common computation.

[0215] The input to the second non-common computation step proj3 is the first non-common computation result output by concat of the first non-common computation step, and the output is the second non-common computation result (non-common output).

[0216] so Figure 5 Compared to the image on the right Figure 5 The diagram on the left shows that optimizing the "aggregate calculation" itself reduces the input data for each sub-step, thus lowering the data transmission pressure.

[0217] The present disclosure provides the following solution to the problem of wasted computing power caused by repeatedly calculating features on the same input in current batch request processing.

[0218] Shared storage

[0219] In some embodiments, after the feature calculation service determines the information of one or more common columns in the input field information of the batch request, it further establishes a shared storage area. The feature calculation service stores the common data in the shared storage area; wherein, the common data includes: data in the common columns, common intermediate data, and common calculation results.

[0220] Data in common columns, for example Figure 3 In the scenario shown, the primary key column and time column in the input field information of the batch request are both common columns.

[0221] Common intermediate data includes intermediate data generated by the first common computation step in the second execution plan, such as... Figure 5 The common state output by proj1 in the first common computation step is common intermediate data. For example... Figure 3 In this context, since the window of request will still be used in subsequent calculations, the window of request output by the window is public intermediate data.

[0222] Common computation results include the computation results generated by the second common computation step in the second execution plan, such as... Figure 5 The common output of the second common computation step proj2 is the common computation result.

[0223] In some embodiments, the feature calculation service can store common data in a shared storage area through a pre-established data encoding and storage interface. In some embodiments, the feature calculation service is located on the server side, and the batch requests received by the feature calculation service are generated by the client. When generating batch requests, the client can optimize the storage of batch requests through a pre-established data encoding and storage interface, that is, store the common column data of the input field information of the batch requests in the shared storage area established by the client.

[0224] In some embodiments, the feature calculation service may further use a compression algorithm to compress the data in the batch request or the data in the common columns or non-common columns of the batch request.

[0225] In some embodiments, the server can be understood as a server capable of creating feature calculation services. The server can be a single server or a group of servers. The server group can be centralized or distributed. The client can be any type of electronic device, such as a smartphone, tablet, laptop, or other portable mobile device, or a desktop computer, smart TV, or other fixed device.

[0226] In some embodiments, the feature calculation service executes a second execution plan for each data row in the batch request, obtains the calculation result for each data row, and merges the calculation results of each data row into the batch calculation result of the batch request.

[0227] Reusing common intermediate data

[0228] In some embodiments, when the feature calculation service executes the second execution plan:

[0229] For the first common computation step in the second execution plan:

[0230] The query index relationship is the relationship between the identifier of a common computation step and the storage location of the data generated by that common computation step in the shared storage area. In some embodiments, the query index relationship can be queried through a pre-established public data cache interface.

[0231] If the identifier of the first common computation step is found, the first storage location corresponding to the identifier of the first common computation step is determined from the index relationship.

[0232] Based on the first storage location, data is obtained from the shared storage area as public intermediate data generated by the first public computing step.

[0233] In this way, the common intermediate data stored in the shared storage area can be directly reused, eliminating the need to recalculate the first common calculation step and reducing the waste of computing power.

[0234] In some embodiments, if the feature calculation service cannot find the identifier of the first common calculation step, it generates common intermediate data based on the data in the common column through the first common calculation step; then it stores the common intermediate data generated by the first common calculation step in the shared storage area, and records the relationship between the identifier of the first common calculation step and the storage location in the index relationship, so that the common intermediate data stored in the shared storage area can be reused for the first common calculation step in the future, without having to repeatedly calculate features for the same input, thus reducing the waste of computing power.

[0235] Reusing common computation results

[0236] In some embodiments, when the feature calculation service executes the second execution plan:

[0237] Regarding the second common computation step in the second execution plan:

[0238] The query index relationship is the relationship between the identifier of a common computation step and the storage location of the data generated by that common computation step in the shared storage area. In some embodiments, the query index relationship can be queried through a pre-established public data cache interface.

[0239] If the identifier of the second common computation step is found, the second storage location corresponding to the identifier of the second common computation step is determined from the index relationship;

[0240] Based on the second storage location, data is retrieved from the shared storage area as the public computation result generated by this second public computation step.

[0241] In this way, the common computation results stored in the shared storage area can be directly reused, eliminating the need to perform the second common computation step again and reducing the waste of computing power.

[0242] In some embodiments, if the feature calculation service cannot find the identifier of the second common calculation step, it generates a common calculation result based on the data in the common column through the second common calculation step; then, the common calculation result generated by the second common calculation step is stored in the shared storage area, and the relationship between the identifier of the second common calculation step and the storage location is recorded in the index relationship, so that the common calculation result stored in the shared storage area can be reused for the second common calculation step in the future, without having to repeatedly calculate features for the same input, thus reducing the waste of computing power.

[0243] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art will understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in the specification are all optional embodiments.

[0244] Figure 6 This disclosure provides a batch request processing apparatus for feature calculation in an embodiment. For example... Figure 6 As shown, the batch request processing device for feature calculation may include, but is not limited to, a receiving unit 61, a generating unit 62, an optimizing unit 63, and a calculation unit 64. The specific descriptions of each unit are as follows:

[0245] The receiving unit 61 is used to receive a batch request for feature calculation; wherein the batch request includes multiple data rows, the multiple data rows share the calculation target and input field information, and the input field information is used to specify the information of multiple columns included in the data row;

[0246] The generation unit 62 is used to generate a first execution plan based on the input field information and the calculation target; wherein, the first execution plan includes multiple calculation steps and the input and output of each calculation step;

[0247] Optimization unit 63 is used to determine the information of one or more common columns in the input field information, and optimize the calculation steps and / or the input of the calculation steps in the first execution plan based on the information of the common columns to obtain the second execution plan;

[0248] The calculation unit 64 is used to obtain the batch calculation results of the batch request based on the second execution plan.

[0249] In some embodiments, the optimization unit 63 determines information from one or more common columns in the input field information, including:

[0250] Information about one or more common columns in the input field information can be obtained through a pre-established common column interface; or, information about one or more common columns in the input field information can be determined based on multiple data rows included in a batch request.

[0251] In some embodiments, the optimization unit 63 optimizes the calculation steps and / or the inputs of the calculation steps in the first execution plan based on the information of the common columns to obtain the second execution plan, including:

[0252] For each computational step in the first execution plan:

[0253] Determine whether the calculation step meets the first optimization condition, wherein the first optimization condition is: the calculation step is based on data in one or more common columns and is not based on data in any non-common column;

[0254] If the first optimization condition is met, the input for optimizing this calculation step is data from one or more common columns.

[0255] In some embodiments, if the optimization unit 63 determines that the calculation step does not meet the first optimization condition, then:

[0256] Determine whether the calculation step satisfies the second optimization condition, wherein the second optimization condition is: the calculation step is performed based on data in one or more common columns and on data in one or more non-common columns;

[0257] If the second optimization condition is met, the calculation step is optimized to at least one first sub-step and at least one second sub-step; wherein the first sub-step is calculated based only on data in one or more common columns; and the second sub-step is calculated based at least on data in one or more non-common columns.

[0258] In some embodiments, the calculation step is time window retrieval, the input field information includes a primary key column and a time column, and the time window retrieval is based on the data in the primary key column and the data in the time column to retrieve window data that meets the calculation target;

[0259] If both the primary key column and the time column are common columns, then optimization unit 63 optimizes the input of the time window pull, which consists of the data in the primary key column and the data in the time column.

[0260] In some embodiments, the calculation step is table concatenation, which concatenates the left and right tables based on concatenation conditions. The left table is obtained based on a batch request, and the right table is obtained based on tables stored in the database.

[0261] If the concatenation condition is a common column, then optimization unit 63 will concatenate the optimized table into one first sub-step and two second sub-steps;

[0262] The first sub-step is used to: perform a first concatenation based on the concatenation conditions on the data of all common columns in the left table and the data of all common columns in the right table;

[0263] The two second sub-steps include: a right splicing sub-step and a left splicing sub-step;

[0264] The right concatenation sub-step is used to: perform a second concatenation based on the unique index of the right table on the output of the first sub-step and the data of all non-common columns in the right table;

[0265] The left concatenation sub-step is used to perform a third concatenation on the output of the right concatenation sub-step and the data of all non-common columns in the left table;

[0266] The output of the left concatenation sub-step is used as the calculation result of the table concatenation.

[0267] In some embodiments, the calculation step is an aggregate calculation, which generates multiple calculation results, each of which is a common calculation result, a first non-common calculation result, or a second non-common calculation result.

[0268] Among them, the public calculation result is calculated based solely on the data in one or more public columns; the first non-public calculation result is calculated based on the data in one or more public columns and the data in one or more non-public columns; and the second non-public calculation result is calculated based on the data in one or more non-public columns.

[0269] In some embodiments, the optimization unit 63 optimizes the aggregation calculation into a plurality of first sub-steps and a plurality of second sub-steps; wherein the plurality of first sub-steps include at least one first common calculation step and at least one second common calculation step; and the plurality of second sub-steps include at least one first non-common calculation step and at least one second non-common calculation step.

[0270] The input to the first common calculation step is data from one or more common columns, and the output is common intermediate data;

[0271] The input to the second common calculation step is data from one or more common columns, and the output is the common calculation result;

[0272] The input to the first non-public computation step is one or more public intermediate data output by the first public computation step and data from one or more non-public columns, and the output is the first non-public computation result.

[0273] The input to the second non-public calculation step is data from one or more non-public columns, and the output is the result of the second non-public calculation.

[0274] In some embodiments, after determining the information of one or more common columns in the input field information, the optimization unit 63 is further configured to:

[0275] Establish a shared storage area;

[0276] Public data is stored in a shared storage area; public data includes: data in public columns, public intermediate data, and public calculation results.

[0277] The common intermediate data refers to the intermediate data generated by the first common calculation step in the second execution plan. The first common calculation step is a step that generates intermediate data based on data in one or more common columns.

[0278] The common calculation result is the calculation result generated by the second common calculation step in the second execution plan. The second common calculation step is a step that generates a calculation result based on data in one or more common columns.

[0279] In some embodiments, the calculation unit 64 is configured to: execute the second execution plan for each data row in the batch request, obtain the calculation result of each data row, and merge them into the batch calculation result of the batch request.

[0280] In some embodiments, when the computing unit 64 executes the second execution plan:

[0281] For the first common computation step in the second execution plan:

[0282] Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area;

[0283] If the identifier of the first common computation step is found, the first storage location corresponding to the identifier of the first common computation step is determined from the index relationship;

[0284] Based on the first storage location, data is obtained from the shared storage area as public intermediate data generated by the first public computing step.

[0285] In some embodiments, the computing unit 64 is further configured to: if the identifier of the first common computing step cannot be found, generate common intermediate data based on the data in the common column through the first common computing step; store the common intermediate data generated by the first common computing step in the shared storage area, and record the relationship between the identifier of the first common computing step and the storage location in the index relationship.

[0286] In some embodiments, when the computing unit 64 executes the second execution plan:

[0287] Regarding the second common computation step in the second execution plan:

[0288] Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area;

[0289] If the identifier of the second common computation step is found, the second storage location corresponding to the identifier of the second common computation step is determined from the index relationship;

[0290] Based on the second storage location, data is retrieved from the shared storage area as the public computation result generated by this second public computation step.

[0291] In some embodiments, the calculation unit 64 is further configured to: if the identifier of the second common calculation step cannot be found, generate a common calculation result based on the data in the common column through the second common calculation step; store the common calculation result generated by the second common calculation step in the shared storage area, and record the relationship between the identifier of the second common calculation step and the storage location in the index relationship.

[0292] In some embodiments, the calculation unit 64 queries the index relationship including:

[0293] The index relationship is queried through a pre-established public data cache interface.

[0294] In some embodiments, the computing unit 64 stores public data in a shared storage area by storing public data in the shared storage area through a pre-established data encoding storage interface.

[0295] For implementation details of the feature computation batch request processing apparatus, please refer to the various embodiments of the feature computation batch request processing method. To avoid repetition, they will not be repeated here.

[0296] In some embodiments, the division of units in the feature computation batch request processing apparatus is only a logical functional division. In actual implementation, there may be other division methods; for example, multiple units may be implemented as one unit, or one unit may be divided into multiple sub-units. It is understood that each unit or sub-unit can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application.

[0297] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 7 As shown, the electronic device includes at least one processor 71, at least one memory 72, and at least one communication interface 73. The various components of the electronic device are coupled together via a bus system 74. The communication interface 73 is used for information transmission with external devices. Understandably, the bus system 74 is used to implement communication between these components. In addition to a data bus, the bus system 74 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 7 The general labeled all buses as Bus System 74.

[0298] It is understood that the memory 72 in this embodiment may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0299] In some implementations, memory 72 stores elements such as executable units or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0300] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic tasks and handle hardware-based tasks. The application programs include various applications, such as media players and browsers, used to implement various application tasks. The program implementing the batch request processing method for feature calculation provided in the embodiments of this disclosure can be included in the application programs.

[0301] In this embodiment of the disclosure, the processor 71 executes the steps of the batch request processing method for feature calculation provided in this embodiment by calling the program or instructions stored in the memory 72, specifically, the program or instructions stored in the application program.

[0302] The batch request processing method for feature calculation provided in this disclosure can be applied to or implemented by processor 71. Processor 71 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the hardware of processor 71 or by instructions in software form. Processor 71 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor.

[0303] The steps of the batch request processing method for feature calculation provided in this disclosure can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software units in the decoding processor. The software units can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 72, and processor 71 reads information from memory 72 and combines it with hardware to complete the steps of the method.

[0304] This disclosure also proposes a non-transitory computer-readable storage medium that stores a program or instructions that cause a computer to perform steps of various embodiments of a batch request processing method, such as feature computation. To avoid repetition, these steps will not be repeated here.

[0305] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0306] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this disclosure and form different embodiments.

[0307] Those skilled in the art will understand that the descriptions of the various embodiments have different focuses, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0308] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A batch request processing method for feature calculation, comprising: Receive a batch request for feature calculation; wherein the batch request includes multiple data rows, the multiple data rows share calculation target and input field information, and the input field information is used to specify information of multiple columns included in the data row; A first execution plan is generated based on the input field information and the calculation target; wherein, the first execution plan includes multiple calculation steps and the input and output of each calculation step; Determine the information of one or more common columns in the input field information, and optimize the calculation steps and / or the input of the calculation steps in the first execution plan based on the information of the common columns to obtain a second execution plan; the common column is defined as follows: the value of this column is exactly the same in all request rows of a batch request; Based on the second execution plan, the batch calculation results of the batch requests are obtained; The step of determining information from one or more common columns in the input field information, and optimizing the calculation steps and / or the inputs of the calculation steps in the first execution plan based on the information from the common columns to obtain a second execution plan includes: For each computation step in the first execution plan: Determine whether the calculation step meets the first optimization condition, wherein the first optimization condition is: the calculation step is based on data in one or more of the common columns, and is not based on data in any non-common column; If the first optimization condition is met, then the input for optimizing this calculation step is the data in one or more of the common columns; or, If the first optimization condition is not met, then: Determine whether the calculation step meets the second optimization condition, wherein the second optimization condition is: the calculation step is performed based on data in one or more of the common columns and on data in one or more of the non-common columns; if the second optimization condition is met, optimize the calculation step to at least one first sub-step and at least one second sub-step; wherein the first sub-step is performed based only on data in one or more of the common columns; and the second sub-step is performed based at least on data in one or more of the non-common columns.

2. The method according to claim 1, wherein, The information used to determine one or more common columns in the input field information includes: Information about one or more common columns in the input field information is obtained through a pre-established common column interface; or, Based on the multiple data rows included in the batch request, information from one or more common columns in the input field information is determined.

3. The method according to claim 1, wherein, For this calculation step, which involves time window retrieval, the input field information includes a primary key column and a time column, and the time window retrieval is based on the data in the primary key column and the data in the time column to retrieve window data that matches the calculation objective. If the primary key column and the time column are both common columns, then the input for optimizing the time window pull is the data in the primary key column and the data in the time column.

4. The method according to claim 1, wherein, The calculation step involves table concatenation, which concatenates the left and right tables based on concatenation conditions. The left table is obtained based on the batch request, and the right table is obtained based on tables stored in the database. If the concatenation condition is a common column, then the table concatenation is optimized into one first sub-step and two second sub-steps; The first sub-step is used to: perform a first concatenation based on the concatenation conditions on the data of all common columns in the left table and the data of all common columns in the right table; The two second sub-steps include: a right splicing sub-step and a left splicing sub-step; The right concatenation sub-step is used to: perform a second concatenation based on the unique index of the right table on the output of the first sub-step and the data of all non-common columns in the right table; The left concatenation sub-step is used to: perform a third concatenation on the output of the right concatenation sub-step and the data of all non-common columns in the left table; The output of the left splicing sub-step serves as the calculation result of the table splicing.

5. The method according to claim 1, wherein, This calculation step is an aggregate calculation, which generates multiple calculation results, each of which is a common calculation result, a first non-common calculation result, or a second non-common calculation result. The common calculation result is calculated based solely on data from one or more of the common columns; the first non-common calculation result is calculated based on data from one or more of the common columns and data from one or more non-common columns; and the second non-common calculation result is calculated based on data from one or more non-common columns.

6. The method according to claim 5, wherein, The aggregation calculation is optimized into multiple first sub-steps and multiple second sub-steps; wherein the multiple first sub-steps include at least one first common calculation step and at least one second common calculation step; the multiple second sub-steps include at least one first non-common calculation step and at least one second non-common calculation step; The input to the first common calculation step is data from one or more of the common columns, and the output is common intermediate data; The input to the second common calculation step is data from one or more of the common columns, and the output is the common calculation result; The input to the first non-public calculation step is one or more public intermediate data output by the first public calculation step and data from one or more non-public columns, and the output is the first non-public calculation result. The input to the second non-public calculation step is data from one or more non-public columns, and the output is the result of the second non-public calculation.

7. The method according to claim 1, wherein, After determining the information of one or more common columns in the input field information, the method further includes: Establish a shared storage area; Public data is stored in the shared storage area; wherein, the public data includes: data in the public columns, public intermediate data, and public calculation results; The common intermediate data refers to the intermediate data generated by the first common calculation step in the second execution plan, and the first common calculation step is a step of generating intermediate data based on data in one or more of the common columns. The common calculation result is the calculation result generated by the second common calculation step in the second execution plan, and the second common calculation step is a step that generates a calculation result based on data in one or more of the common columns.

8. The method according to claim 7, wherein, The process of obtaining the batch computation result of the batch request based on the second execution plan includes: For each data row in the batch request, the second execution plan is executed to obtain the calculation result of each data row, and then the results are merged into the batch calculation result of the batch request.

9. The method according to claim 8, wherein, When executing the second execution plan: For the first common computation step in the second execution plan: Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area; If the identifier of the first common computation step is found, the first storage location corresponding to the identifier of the first common computation step is determined from the index relationship; Based on the first storage location, data is obtained from the shared storage area as public intermediate data generated by the first public computing step.

10. The method according to claim 9, wherein, The method further includes: If the identifier of the first common calculation step cannot be found, then common intermediate data is generated based on the data in the common column through the first common calculation step; The public intermediate data generated by the first public computation step is stored in the shared storage area, and the relationship between the identifier of the first public computation step and its storage location is recorded in the index relationship.

11. The method according to claim 8, wherein, When executing the second execution plan: Regarding the second common computation step in the second execution plan: Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area; If the identifier of the second common computation step is found, the second storage location corresponding to the identifier of the second common computation step is determined from the index relationship; Based on the second storage location, data is obtained from the shared storage area as the public computation result generated by the second public computation step.

12. The method according to claim 11, wherein, The method further includes: If the identifier of the second common calculation step cannot be found, a common calculation result is generated based on the data in the common column through the second common calculation step. The public computation result generated by the second public computation step is stored in the shared storage area, and the relationship between the identifier of the second public computation step and its storage location is recorded in the index relationship.

13. The method according to any one of claims 9 to 12, wherein, The query index relationship includes: The index relationship is queried through a pre-established public data cache interface.

14. The method according to any one of claims 7 to 12, wherein, The step of storing public data in the shared storage area includes: Public data is stored in the shared storage area through a pre-established data encoding and storage interface.

15. A batch request processing apparatus for feature calculation, comprising: A receiving unit is configured to receive a batch request for feature calculation; wherein the batch request includes multiple data rows, the multiple data rows share calculation target and input field information, and the input field information is used to specify information of multiple columns included in the data rows; A generation unit is configured to generate a first execution plan based on the input field information and the calculation target; wherein the first execution plan includes multiple calculation steps and the input and output of each calculation step; An optimization unit is configured to determine information from one or more common columns in the input field information, and optimize the calculation steps and / or the inputs of the calculation steps in the first execution plan based on the information from the common columns to obtain a second execution plan; the common column is defined as a column whose value is exactly the same across all request rows in a batch request; A calculation unit is configured to obtain the batch calculation result of the batch request based on the second execution plan; The optimization unit optimizes the calculation steps and / or the inputs of the calculation steps in the first execution plan based on the information of the common column, to obtain a second execution plan including: For each computation step in the first execution plan: Determine whether the calculation step meets the first optimization condition, wherein the first optimization condition is: the calculation step is based on data in one or more of the common columns, and is not based on data in any non-common column; If the first optimization condition is met, then the input for optimizing this calculation step is the data in one or more of the common columns; or, If the first optimization condition is not met, then: Determine whether the calculation step meets the second optimization condition, wherein the second optimization condition is: the calculation step is performed based on data in one or more of the common columns and on data in one or more of the non-common columns; if the second optimization condition is met, optimize the calculation step to at least one first sub-step and at least one second sub-step; wherein the first sub-step is performed based only on data in one or more of the common columns; and the second sub-step is performed based at least on data in one or more of the non-common columns.

16. The apparatus according to claim 15, wherein, The optimization unit determines information from one or more common columns in the input field information, including: Information about one or more common columns in the input field information is obtained through a pre-established common column interface; or, Based on the multiple data rows included in the batch request, information from one or more common columns in the input field information is determined.

17. The apparatus according to claim 15, wherein, For this calculation step, which involves time window retrieval, the input field information includes a primary key column and a time column, and the time window retrieval is based on the data in the primary key column and the data in the time column to retrieve window data that matches the calculation objective. If the primary key column and the time column are both common columns, then the optimization unit optimizes the input of the time window pull as the data in the primary key column and the data in the time column.

18. The apparatus according to claim 15, wherein, The calculation step involves table concatenation, which concatenates the left and right tables based on concatenation conditions. The left table is obtained based on the batch request, and the right table is obtained based on tables stored in the database. If the concatenation condition is a common column, then the optimization unit optimizes the table concatenation into one first sub-step and two second sub-steps; The first sub-step is used to: perform a first concatenation based on the concatenation conditions on the data of all common columns in the left table and the data of all common columns in the right table; The two second sub-steps include: a right splicing sub-step and a left splicing sub-step; The right concatenation sub-step is used to: perform a second concatenation based on the unique index of the right table on the output of the first sub-step and the data of all non-common columns in the right table; The left concatenation sub-step is used to: perform a third concatenation on the output of the right concatenation sub-step and the data of all non-common columns in the left table; The output of the left splicing sub-step serves as the calculation result of the table splicing.

19. The apparatus according to claim 15, wherein, This calculation step is an aggregate calculation, which generates multiple calculation results, each of which is a common calculation result, a first non-common calculation result, or a second non-common calculation result. The common calculation result is calculated based solely on data from one or more of the common columns; the first non-common calculation result is calculated based on data from one or more of the common columns and data from one or more non-common columns; and the second non-common calculation result is calculated based on data from one or more non-common columns.

20. The apparatus according to claim 19, wherein, The optimization unit optimizes the aggregation calculation into multiple first sub-steps and multiple second sub-steps; wherein, the multiple first sub-steps include at least one first common calculation step and at least one second common calculation step; the multiple second sub-steps include at least one first non-common calculation step and at least one second non-common calculation step; The input to the first common calculation step is data from one or more of the common columns, and the output is common intermediate data; The input to the second common calculation step is data from one or more of the common columns, and the output is the common calculation result; The input to the first non-public calculation step is one or more public intermediate data output by the first public calculation step and data from one or more non-public columns, and the output is the first non-public calculation result. The input to the second non-public calculation step is data from one or more non-public columns, and the output is the result of the second non-public calculation.

21. The apparatus according to claim 15, wherein, After determining the information of one or more common columns in the input field information, the optimization unit is further used to: Establish a shared storage area; Public data is stored in the shared storage area; wherein, the public data includes: data in the public columns, public intermediate data, and public calculation results; The common intermediate data refers to the intermediate data generated by the first common calculation step in the second execution plan, and the first common calculation step is a step of generating intermediate data based on data in one or more of the common columns. The common calculation result is the calculation result generated by the second common calculation step in the second execution plan, and the second common calculation step is a step that generates a calculation result based on data in one or more of the common columns.

22. The apparatus according to claim 21, wherein, The computing unit is used for: For each data row in the batch request, the second execution plan is executed to obtain the calculation result of each data row, and then the results are merged into the batch calculation result of the batch request.

23. The apparatus according to claim 22, wherein, When the computing unit executes the second execution plan: For the first common computation step in the second execution plan: Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area; If the identifier of the first common computation step is found, the first storage location corresponding to the identifier of the first common computation step is determined from the index relationship; Based on the first storage location, data is obtained from the shared storage area as public intermediate data generated by the first public computing step.

24. The apparatus according to claim 23, wherein, The computing unit is also used for: If the identifier of the first common calculation step cannot be found, then common intermediate data is generated based on the data in the common column through the first common calculation step; The public intermediate data generated by the first public computation step is stored in the shared storage area, and the relationship between the identifier of the first public computation step and its storage location is recorded in the index relationship.

25. The apparatus according to claim 22, wherein, When the computing unit executes the second execution plan: Regarding the second common computation step in the second execution plan: Query the index relationship, which is the relationship between the identifier of the common computing step and the storage location of the data generated by the common computing step in the shared storage area; If the identifier of the second common computation step is found, the second storage location corresponding to the identifier of the second common computation step is determined from the index relationship; Based on the second storage location, data is obtained from the shared storage area as the public computation result generated by the second public computation step.

26. The apparatus according to claim 25, wherein, The computing unit is also used for: If the identifier of the second common calculation step cannot be found, a common calculation result is generated based on the data in the common column through the second common calculation step. The public computation result generated by the second public computation step is stored in the shared storage area, and the relationship between the identifier of the second public computation step and its storage location is recorded in the index relationship.

27. The apparatus according to any one of claims 23 to 26, wherein, The calculation unit queries the index relationship, including: The index relationship is queried through a pre-established public data cache interface.

28. The apparatus according to any one of claims 21 to 26, wherein, The computing unit stores public data in the shared storage area, including: Public data is stored in the shared storage area through a pre-established data encoding and storage interface.

29. An electronic device, characterized in that, include: Processor and memory; The processor executes the steps of the method as described in any one of claims 1 to 14 by invoking programs or instructions stored in the memory.

30. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a program or instructions that cause a computer to perform the steps of the method as described in any one of claims 1 to 14.

Images