Data index generation method, system and device, electronic equipment and medium

By merging and processing multiple source data in memory, and directly calculating data indicators using the BEAM data processing process, the problem of time-consuming generation of data indicators in the existing technology is solved, and a more efficient data indicator generation rate is achieved.

CN119025580APending Publication Date: 2024-11-26CHONGQING CHANGAN AUTOMOBILE CO LTD

Patent Information

Application Number
CN202310603391.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing data metric generation method requires a lot of time and cost, which is mainly due to the data storage and maintenance process, resulting in a slow generation speed.

Method used

Merge multiple source data in memory and use the pre-generated BEAM data processing process for calculations, directly performing data processing during the calculation process, avoiding the storage and maintenance of intermediate process data.

Benefits of technology

The rate of multi-source data indicator generation is improved, the storage and maintenance of intermediate process data is reduced, and the efficiency of data indicator generation is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025580A_ABST
    Figure CN119025580A_ABST
Patent Text Reader

Abstract

The invention relates to a data index generation method, system and device, electronic equipment and a medium, and belongs to the technical field of data processing. The method comprises the following steps: in response to an operation of generating a data index, respectively obtaining source data from a plurality of databases; merging the multiple pieces of source data in the memory to obtain target data; and calculating the target data in the memory by adopting a pre-generated BEAM data processing flow to obtain a target data index for the multiple pieces of source data. Through the embodiment of the invention, the calculation can be directly performed based on the memory in the calculation process, and the calculation can be released after the calculation is completed, so that the storage and maintenance of the data in the intermediate process do not exist. Therefore, the generation rate of the data index based on the multi-source data is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method, system, device, electronic device and medium for generating data indicators. Background Art

[0002] Data indicators are the processing of recorded historical information in the process of enterprise operation and converted into numbers. According to specific business purposes: relevant numbers are processed according to certain business logic and certain technical means to become tools for describing, measuring, analyzing and predicting business results.

[0003] In order to generate data indicators, we can first obtain data sources from multiple databases and store them in a storage space for data cleaning and processing; then we can calculate and process the cleaned source data to obtain data indicators. Although this process can generate data indicators, it also involves data storage and maintenance, which will result in a lot of time-consuming data indicators. Summary of the invention

[0004] In view of the above problems, a method, system, device, electronic device and medium for generating a data indicator is proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A method for generating a data indicator, the method comprising:

[0006] In response to the operation of generating data indicators, source data are obtained from multiple databases respectively;

[0007] Merge multiple source data in memory to obtain target data;

[0008] The target data is calculated in the memory using a pre-generated BEAM data processing flow to obtain target data indicators for the multiple source data.

[0009] Optionally, the method further comprises:

[0010] In response to the configuration information submitted by the user, an operator link graph is generated, where the operator link graph is formed by connecting multiple operator models;

[0011] Parsing each operator model in the operator chain diagram to obtain multiple BEAM model instructions;

[0012] A data processing flow of the BEAM is generated according to the multiple BEAM model instructions.

[0013] Optionally, the method further comprises:

[0014] Displaying an interactive interface, wherein the interactive interface includes multiple operator components;

[0015] In response to a user's operation on the plurality of operator components, determining a target operator component;

[0016] The configuration information is generated according to the target operator component.

[0017] Optionally, generating the data processing flow of the BEAM according to the multiple BEAM model instructions includes:

[0018] Optimizing the plurality of BEAM model instructions to obtain a plurality of target execution instructions;

[0019] The data processing flow of the BEAM is generated according to the target execution instruction.

[0020] Optionally, the generating a data processing flow of the BEAM according to the target execution instruction includes:

[0021] Parsing the first operator model in the operator link graph to obtain indicator input parameters;

[0022] Parsing the last operator model in the operator link graph to obtain an indicator output field;

[0023] A data processing flow of the BEAM is generated according to the target execution instruction, the indicator input parameter, and the indicator output field.

[0024] Optionally, parsing each operator model in the operator chain graph to obtain multiple BEAM model instructions includes:

[0025] Analyze the operator model with 0 information content, obtain a BEAM model instruction, and reduce the information content of the next operator model by 1;

[0026] When the information amount of the next operator model is 0, the next operator model is parsed to obtain another BEAM model instruction.

[0027] The present invention also provides a data indicator generation system, the system comprising: a front-end indicator model arrangement canvas module, a back-end model parser module, a parameter parser module, a BEAM model compiler module, a BEAM model optimizer module, a BEAM data processing flow generator module, an execution engine module, a data source module, and an indicator library module;

[0028] The front-end indicator model arrangement canvas module is used to display an interactive interface, which includes multiple operator components; in response to the user's operation on the multiple operator components, determine the target operator component; and generate configuration information according to the target operator component;

[0029] The backend model parser module is used to generate an operator chain graph according to the configuration information; the operator chain graph is formed by connecting multiple operator models;

[0030] The parameter parser module is used to parse each operator model in the operator link graph;

[0031] The BEAM model compiler module is used to compile multiple BEAM model instructions according to the parsing results;

[0032] The BEAM model optimizer module is used to optimize the multiple BEAM model instructions to obtain multiple target execution instructions;

[0033] The BEAM data processing flow generator module is used to generate the BEAM data processing flow according to the target execution instruction;

[0034] The execution engine module is used to obtain source data from multiple databases of the data source module, and merge the multiple source data in the memory to obtain target data; use the pre-generated BEAM data processing flow to calculate the target data in the memory to obtain target data indicators for the multiple source data;

[0035] The data source module is used to store source data of each database;

[0036] The indicator library module is used to store the target data indicators.

[0037] The present invention also provides a device for generating a data indicator, the device comprising:

[0038] An acquisition module, used for acquiring source data from multiple databases respectively in response to an operation of generating data indicators;

[0039] A merging module is used to merge multiple source data in memory to obtain target data;

[0040] The indicator generation module is used to use the pre-generated BEAM data processing flow to calculate the target data in the memory to obtain target data indicators for the multiple source data.

[0041] Optionally, the device further comprises:

[0042] The process generation module is used to generate an operator chain diagram in response to the configuration information submitted by the user, where the operator chain diagram is composed of multiple operator models connected; parse each operator model in the operator chain diagram to obtain multiple BEAM model instructions; and generate the data processing process of the BEAM according to the multiple BEAM model instructions.

[0043] Optionally, the device further comprises:

[0044] The configuration information generation module is used to display an interactive interface, which includes multiple operator components; determine a target operator component in response to a user's operation on the multiple operator components; and generate the configuration information according to the target operator component.

[0045] Optionally, the process generation module is used to optimize the multiple BEAM model instructions to obtain multiple target execution instructions; and generate the data processing process of the BEAM according to the target execution instructions.

[0046] Optionally, the process generation module is used to parse the first operator model in the operator link graph to obtain indicator input parameters; parse the last operator model in the operator link graph to obtain indicator output fields; and generate the data processing process of the BEAM according to the target execution instructions, the indicator input parameters, and the indicator output fields.

[0047] Optionally, the process generation module is used to parse the operator model with an information amount of 0, obtain a BEAM model instruction, and reduce the information amount of the next operator model by 1; when the information amount of the next operator model is 0, parse the next operator model to obtain another BEAM model instruction.

[0048] The present invention also provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above method for generating data indicators when executed by the processor.

[0049] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for generating data indicators as described above is implemented.

[0050] Beneficial effects of the present invention:

[0051] In the present invention, in response to the operation of generating data indicators, source data are obtained from multiple databases respectively; multiple source data are merged in memory to obtain target data; and the target data is calculated in memory using the pre-generated BEAM data processing flow to obtain target data indicators for multiple source data. Through the embodiment of the present invention, calculations can be performed directly based on memory during the calculation process, and the memory will be released after the calculation is completed, so there is no storage and maintenance of intermediate process data. Thus, the generation rate of data indicators based on multi-source data is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flowchart of a method for generating a data indicator according to an embodiment of the present invention;

[0053] Figure 2 is a flowchart of steps of a method for generating a data processing flow of a BEAM according to an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of an operator link graph according to an embodiment of the present invention;

[0055] Figure 4 is a flowchart of another method for generating a data processing flow of a BEAM according to an embodiment of the present invention;

[0056] Figure 5 It is a structural schematic diagram of a data indicator generation system according to an embodiment of the present invention;

[0057] Figure 6 It is a structural schematic diagram of a data indicator generation device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In order to improve the generation rate of data indicators based on multi-source data, an embodiment of the present invention provides a method for generating data indicators. After obtaining multiple source data, multiple source data can be merged in memory to obtain target data; and after obtaining the target data, a pre-generated BEAM data processing flow is used to calculate the target data in memory to obtain target data indicators for multiple source data. Through the embodiment of the present invention, calculations can be performed directly based on memory during the calculation process, and the memory will be released after the calculation is completed, so there is no storage and maintenance of intermediate process data. Thereby, the generation rate of data indicators based on multi-source data is improved.

[0060] Reference Figure 1 , shows a flowchart of a method for generating a data indicator according to an embodiment of the present invention, which may include the following steps:

[0061] Step 101: In response to the operation of generating data indicators, source data are obtained from multiple databases respectively.

[0062] When it is necessary to generate data indicators of source data in each current database, the user can first perform an operation for generating data indicators; in response to the operation of generating data indicators, the source data can be obtained from each database that pre-stores the source data. The multiple databases can be different types of databases.

[0063] Step 102: Merge multiple source data in memory to obtain target data.

[0064] After obtaining multiple source data, in order to improve the efficiency of data indicator generation, the multiple source data can be directly merged in the memory to obtain the target data.

[0065] Step 103: Use the pre-generated BEAM data processing flow to calculate the target data in the memory to obtain target data indicators for multiple source data.

[0066] After obtaining the target data, a data processing flow of a BEAM generated in advance can be used to calculate the target data in the memory to obtain target data indicators for multiple source data. The data processing flow of the BEAM will be described later.

[0067] As an example, for non-relational databases, the corresponding non-relational database Transfrom can be encapsulated based on the non-relational database link IOReader provided by Apache Beam. For example: mongoDB, Redis database, Apache Beam provides MongoDbIO.Read, RedisIO.Read, which can read and connect to mongodb and Redis.

[0068] For API (Application Program Interface) data sources: You can encapsulate HTTPTransfrom based on org.apache.http.impl.client.CloseableHttpClient to connect to the API data source.

[0069] In an embodiment of the present invention, in response to an operation of generating data indicators, source data are obtained from multiple databases respectively; multiple source data are merged in memory to obtain target data; and the target data is calculated in memory using a pre-generated BEAM data processing flow to obtain target data indicators for multiple source data. Through an embodiment of the present invention, calculations can be performed directly based on memory during the calculation process, and the memory will be released after the calculation is completed, so there is no storage and maintenance of intermediate process data. Thus, the generation rate of data indicators based on multi-source data is improved.

[0070] The following is an explanation of how to generate the data processing flow of the above BEAM. Figure 2 , showing a flowchart of a method for generating a data processing flow of a BEAM according to an embodiment of the present invention, which may include the following steps:

[0071] Step 201: In response to the configuration information submitted by the user, an operator link graph is generated, where the operator link graph is formed by connecting multiple operator models.

[0072] First, when a user needs to generate a data processing flow of a BEAM for generating data indicators, he or she may first submit a configuration information; the configuration information may be the specific process of processing the source data.

[0073] After obtaining the configuration information mentioned by the user, an operator link graph may be generated in response to the configuration information; the operator link graph may be formed by connecting multiple operator models, such as Figure 3 As shown; the operator model may refer to a model that can perform a process on the source data, such as aggregation, input, summation, averaging, etc., which is not limited in this embodiment of the present invention.

[0074] Step 202: parse each operator model in the operator link graph to obtain multiple BEAM model instructions.

[0075] After obtaining the operator chain graph, each operator model in the operator chain graph can be parsed one by one to obtain the BEAM model instructions corresponding to each operator model.

[0076] Step 203: Generate a BEAM data processing flow according to multiple BEAM model instructions.

[0077] After obtaining the BEAM model instructions corresponding to all operator models in the operator link diagram, the BEAM data processing flow can be generated according to all the obtained BEAM model instructions; the BEAM data processing flow can be used to perform calculations and other processing on the source data to obtain corresponding data indicators.

[0078] In an embodiment of the present invention, in response to configuration information submitted by a user, an operator chain graph is generated, and the operator chain graph is formed by connecting multiple operator models; each operator model in the operator chain graph is parsed to obtain multiple BEAM model instructions; according to the multiple BEAM model instructions, a BEAM data processing flow is generated. For an embodiment of the present invention, the data processing flow developed based on Beam can be executed on any distributed computing engine, thereby improving the reusability of the flow. And Apache Beam, as a model interface paradigm, can link different data sources in the execution environment, including but not limited to relational databases, non-relational databases, file data, API data, realize the merged calculation of different data sources, reduce the pressure of intermediate data storage, and the cost of intermediate data maintenance.

[0079] Reference Figure 4 , shows a flowchart of another method for generating a data processing flow of a BEAM according to an embodiment of the present invention, which may include the following steps:

[0080] Step 401: display an interactive interface, which includes multiple operator components.

[0081] In actual applications, the SQL (Structured Query Language) method of writing indicators makes it difficult for general business personnel to get started, requiring a good technical foundation. It is difficult to unify the upper-level business with the underlying business database, and SQL writing is difficult.

[0082] In order to enable business personnel who do not have writing capabilities to complete writing, the embodiment of the present invention can first display an interactive interface, which may include multiple operator components, such as: union, intersection, calculation, aggregation, function, filtering, counting, etc. The embodiment of the present invention does not limit this.

[0083] Step 402: In response to a user's operation on multiple operator components, determine a target operator component.

[0084] When an operator component is needed, the business personnel can perform an operation on the operator component displayed in the interactive interface, such as a dragging operation, so as to use it as a target operator component.

[0085] In response to a user's operation on one or more operator components, the one or more operator components may be used as target operator components.

[0086] Step 403: Generate configuration information according to the target operator component.

[0087] After determining one or more target operator components, a configuration information may be generated according to the calculation method corresponding to the one or more target operator components, so as to generate a data processing flow of the BEAM for generating data indicators based on the configuration information.

[0088] After generating the configuration information, the business personnel can click the submit operation so that the background responds to the configuration information and generates the BEAM data processing flow.

[0089] Step 404: Generate an operator link graph in response to the configuration information submitted by the user.

[0090] Specifically, after receiving the configuration information submitted by the user, the backend may first determine one or more corresponding operator models according to the configuration information; and then generate an operator link graph according to the one or more operator models.

[0091] Step 405: parse each operator model in the operator link graph to obtain multiple BEAM model instructions.

[0092] After obtaining the operator chain graph, each operator model in the operator chain graph can be parsed one by one to obtain the BEAM model instructions corresponding to each operator model.

[0093] In one embodiment of the present invention, each operator model may be parsed through the following sub-steps to obtain the BEAM model instructions:

[0094] Sub-step 11: parse the operator model with an information amount of 0, obtain a BEAM model instruction, and reduce the information amount of the next operator model by 1.

[0095] The amount of information may refer to the number of preceding operator models of the operator model.

[0096] After obtaining the operator chain graph, the operator models whose information amount is 0 may be determined first, and these operator models whose information amount is 0 may be analyzed to obtain the BEAM model instructions corresponding to these operator models whose information amount is 0.

[0097] After parsing an operator model with an information amount of 0, the information amount of the next operator model can be reduced by 1; for example: the information amount of the next operator model is 3, after parsing the operator model with an information amount of 0, the information amount of the next operator model is reduced by 1 and becomes 2. The embodiment of the present invention does not limit this.

[0098] Sub-step 12: When the amount of information of the next operator model is 0, the next operator model is parsed to obtain another BEAM model instruction.

[0099] When the amount of information of the next operator model also becomes 0, the parsing operation is triggered, the next operator model can be parsed, and another BEAM model instruction corresponding to the next operator model can be obtained.

[0100] In practical applications, each operator model in the operator link graph may be parsed through sub-steps 11 and 12 as above, until all operator models are parsed.

[0101] Step 406: Generate a BEAM data processing flow according to the multiple BEAM model instructions.

[0102] After obtaining the BEAM model instructions corresponding to each operator model, the BEAM data processing flow can be generated based on one or more BEAM model instructions. Specifically, the BEAM data processing flow can be generated through the following sub-steps:

[0103] Sub-step 21: Optimize multiple BEAM model instructions to obtain multiple target execution instructions.

[0104] In practical applications, in order to remove redundant instructions, after obtaining multiple BEAM model instructions, these multiple BEAM model instructions can be optimized first, so as to obtain multiple target execution instructions.

[0105] As a strength, multiple data sources can be optimized to remove redundant front-end instructions by determining whether the data sources are from the same source.

[0106] For example: if there are limiting conditions in the input operator model and there are also limiting conditions in the filter operator model, it is determined whether the two operator models are derived from the same data source; if so, the limiting conditions of the filter operator model can be added to the limiting conditions of the input operator model, thereby removing the filter operator model and achieving the purpose of optimization.

[0107] Sub-step 22: Generate the data processing flow of BEAM according to the target execution instruction.

[0108] After the optimization, the data processing flow of BEAM can be generated according to the multiple target execution instructions obtained by the optimization; specifically, the data processing flow of BEAM can be generated according to the target execution instructions in the following manner:

[0109] Parse the first operator model in the operator link diagram to obtain the indicator input parameters; parse the last operator model in the operator link diagram to obtain the indicator output field; generate the BEAM data processing flow according to the target execution instructions, indicator input parameters, and indicator output fields.

[0110] In practical applications, the first operator model in the operator link diagram can be parsed first to obtain the indicator input parameters; the indicator input parameters can be used to determine the location of the source data and determine which source data needs to be input.

[0111] In addition, the last operator model in the operator link diagram may be parsed to obtain an indicator output field; the indicator output field may represent a field and type corresponding to the output data indicator.

[0112] After determining the target execution instructions, indicator input parameters, and indicator output fields, the data processing flow of BEAM can be generated based on the target execution instructions, indicator input parameters, and indicator output fields, so that the source data can be obtained based on the indicator input parameters, and the source data can be processed based on the target execution instructions, and the target data indicators can be output based on the indicator output fields.

[0113] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0114] Reference Figure 5 , shows a schematic diagram of the structure of a data indicator generation system 500 according to an embodiment of the present invention, which may include: a front-end indicator model orchestration canvas module 510, a back-end model parser module 520, a parameter parser module 530, a BEAM model compiler module 540, a BEAM model optimizer module 550, a BEAM data processing flow generator module 560, an execution engine module 570, a data source module 580, and an indicator library module 590;

[0115] The front-end indicator model arrangement canvas module 510 is used to display an interactive interface, which includes multiple operator components; in response to the user's operation on the multiple operator components, determine the target operator component; and generate configuration information according to the target operator component;

[0116] The backend model parser module 520 is used to generate an operator chain diagram according to the configuration information; the operator chain diagram is formed by connecting multiple operator models;

[0117] A parameter parser module 530 is used to parse each operator model in the operator link graph;

[0118] A BEAM model compiler module 540 is used to compile a plurality of BEAM model instructions according to the parsing result;

[0119] A BEAM model optimizer module 550, configured to optimize a plurality of BEAM model instructions to obtain a plurality of target execution instructions;

[0120] BEAM data processing flow generator module 560, used to generate BEAM data processing flow according to target execution instructions;

[0121] The execution engine module 570 is used to obtain source data from multiple databases of the data source module 580, and merge the multiple source data in the memory to obtain target data; use the pre-generated BEAM data processing flow to calculate the target data in the memory to obtain target data indicators for multiple source data;

[0122] The data source module 580 is used to store source data of each database;

[0123] The indicator library module 590 is used to store target data indicators.

[0124] In actual applications, the system's front-end indicator model orchestration canvas module 510 can be used to display an interactive interface to the user, which can display one or more operator components; when the user mainly generates data indicators, the operation can be performed on the operator components required.

[0125] The front-end indicator model arrangement canvas module 510 can determine one or more target operator components in response to the user's operation on multiple operator components, and then generate configuration information according to the one or more target operator components.

[0126] After generating the configuration information, the front-end indicator model arrangement canvas module 510 can send it to the back-end model parser module 520; the back-end model parser module 520 can generate an operator link graph connected by multiple operator models according to the configuration information. After generating the operator link graph, the back-end model parser module 520 can send it to the parameter parsing module for parsing.

[0127] The parameter parsing module can parse each operator model in the operator link graph and generate parsing results. The BEAM model compiler module 540 can compile multiple BEAM model instructions according to the parsing results generated by the parameter parsing module and send them to the BEAM model optimizer module 550 for optimization.

[0128] After obtaining a plurality of BEAM model instructions, the BEAM model optimizer module 550 may optimize the plurality of BEAM model instructions to determine a plurality of target execution instructions.

[0129] After obtaining multiple target execution instructions, the BEAM model optimizer module 550 can send them to the BEAM data processing flow generator module 560 to generate the BEAM data processing flow.

[0130] After receiving multiple target execution instructions, the BEAM data processing flow generator module 560 can generate the BEAM data processing flow according to the multiple target execution instructions and send it to the execution engine module 570 for running.

[0131] After receiving the data processing flow of BEAM, the execution engine module 570 can save it first; then, when it is necessary to generate data indicators, it can first obtain the source data from the data sources of each database stored in the data source module 580; then, merge these multiple source data in the memory to obtain the target data.

[0132] After obtaining the target data, the execution engine module 570 can use the BEAM data processing flow to calculate the target data in the memory, obtain the target data indicators for multiple source data, and output the target data indicators to the indicator library module 590 for storage and use.

[0133] In actual applications, the user uses the front-end indicator model orchestration canvas module 510 to drag and drop different operators to build a business indicator calculation model. After the construction is completed, the back-end model parser module 520 is called to parse the front-end model and parse it into a corresponding indicator model link graph. The back-end model parser module 520 generates an operator model with a semaphore according to the number of operator predecessor nodes, and builds a model link graph according to the connection relationship, where the operator link graph is a directed acyclic graph. The parameter parser module 530 further parses the input parameters of the operator with an operator model semaphore of 0 and the output parameters of the operator with an empty post-node, and parses the dynamically generated parameters in the model, including but not limited to dynamic date dimensions and constant parameters.

[0134] The BEAM model compiler module 540 starts compiling into a BEAM instruction model by searching for operators whose semaphore in the operator link graph is 0. After the operator compilation is completed, the post-operator will be notified to reduce the semaphore of the post-operator by 1. When the semaphore is 0, the compilation will start until all operators are compiled.

[0135] The BEAM model optimizer module 550 optimizes the BEAM instruction model generated by the BEAM model compiler module 540, and can remove redundant instructions based on whether the input operator recursively infers the successor operator to be of the same source, so that the instruction model is the optimal instruction model. The BEAM data processing flow generator module 560 generates a data processing flow through the instruction model, that is, each operator instruction encapsulates the Transform of BEAM, and constructs the relationship between each Transfrom to construct the execution pipeline PiplineOptions object, set the Pipline Runner, and include data source connection information, and output the indicator library address information. After the processing is completed, different execution engines are submitted for execution for different PiplineRunners. The execution engine module 570 will query the source data information of the data source module 580 according to the data source connection information, and generate the indicator results after the data processing flow and output them to the indicator library module 590.

[0136] The indicator library module 5909 is an indicator result data set, using mongoDB as the indicator library storage database, and each indicator generates a Conllection separately.

[0137] In an embodiment of the present invention, the system can generate an operator chain diagram in response to configuration information submitted by a user, where the operator chain diagram is connected by multiple operator models; parse each operator model in the operator chain diagram to obtain multiple BEAM model instructions; and generate a BEAM data processing flow according to the multiple BEAM model instructions. For an embodiment of the present invention, the data processing flow developed based on Beam can be executed on any distributed computing engine, thereby improving the reusability of the flow. And Apache Beam, as a model interface paradigm, makes it possible to link different data sources in the execution environment, including but not limited to relational databases, non-relational databases, file data, API data, to achieve merged calculations of different data sources, reduce the pressure of intermediate data storage, and the cost of intermediate data maintenance.

[0138] Reference Figure 6 , shows a schematic diagram of the structure of a device for generating data indicators according to an embodiment of the present invention, which may include the following modules:

[0139] The acquisition module 601 is used to respectively acquire source data from multiple databases in response to the operation of generating data indicators;

[0140] A merging module 602 is used to merge multiple source data in the memory to obtain target data;

[0141] The indicator generation module 603 is used to use the pre-generated BEAM data processing flow to calculate the target data in the memory to obtain the target data indicators for multiple source data.

[0142] In one embodiment of the present invention, the device further includes:

[0143] The process generation module is used to generate an operator chain diagram in response to the configuration information submitted by the user. The operator chain diagram is composed of multiple operator models connected; each operator model in the operator chain diagram is analyzed to obtain multiple BEAM model instructions; and a BEAM data processing flow is generated according to the multiple BEAM model instructions.

[0144] In one embodiment of the present invention, the device further includes:

[0145] The configuration information generation module is used to display an interactive interface, which includes multiple operator components; determine the target operator component in response to the user's operation on the multiple operator components; and generate configuration information according to the target operator component.

[0146] In one embodiment of the present invention, the process generation module is used to optimize multiple BEAM model instructions to obtain multiple target execution instructions; and generate the BEAM data processing process according to the target execution instructions.

[0147] In one embodiment of the present invention, a process generation module is used to parse the first operator model in the operator link diagram to obtain the indicator input parameters; parse the last operator model in the operator link diagram to obtain the indicator output field; and generate the BEAM data processing process according to the target execution instructions, the indicator input parameters, and the indicator output field.

[0148] In one embodiment of the present invention, the process generation module is used to parse the operator model with an information amount of 0, obtain a BEAM model instruction, and reduce the information amount of the next operator model by 1; when the information amount of the next operator model is 0, parse the next operator model to obtain another BEAM model instruction.

[0149] In an embodiment of the present invention, in response to an operation of generating data indicators, source data are obtained from multiple databases respectively; multiple source data are merged in memory to obtain target data; and the target data is calculated in memory using a pre-generated BEAM data processing flow to obtain target data indicators for multiple source data. Through an embodiment of the present invention, calculations can be performed directly based on memory during the calculation process, and the memory will be released after the calculation is completed, so there is no storage and maintenance of intermediate process data. Thus, the generation rate of data indicators based on multi-source data is improved.

[0150] An embodiment of the present invention also provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the method for generating data indicators as described above is implemented.

[0151] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for generating data indicators as described above is implemented.

[0152] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0153] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0154] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0155] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0158] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0159] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0160] The above is a detailed introduction to the method, system, device, electronic device and medium for generating a data indicator. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for generating a data indicator, characterized in that: The method comprises: In response to the operation of generating data indicators, source data are obtained from multiple databases respectively; Merge multiple source data in memory to obtain target data; The target data is calculated in the memory using a pre-generated BEAM data processing flow to obtain target data indicators for the multiple source data.

2. The method according to claim 1, characterized in that The method further comprises: In response to the configuration information submitted by the user, an operator link graph is generated, where the operator link graph is formed by connecting multiple operator models; Parsing each operator model in the operator chain diagram to obtain multiple BEAM model instructions; A data processing flow of the BEAM is generated according to the multiple BEAM model instructions.

3. The method according to claim 2, characterized in that The method further comprises: Displaying an interactive interface, wherein the interactive interface includes multiple operator components; In response to a user's operation on the plurality of operator components, determining a target operator component; The configuration information is generated according to the target operator component.

4. The method according to claim 2, characterized in that: The step of generating the data processing flow of the BEAM according to the plurality of BEAM model instructions comprises: Optimizing the plurality of BEAM model instructions to obtain a plurality of target execution instructions; The data processing flow of the BEAM is generated according to the target execution instruction.

5. The method according to claim 4, characterized in that The step of executing the instruction according to the target to generate the data processing flow of the BEAM includes: Parsing the first operator model in the operator link graph to obtain indicator input parameters; Parsing the last operator model in the operator link graph to obtain an indicator output field; A data processing flow of the BEAM is generated according to the target execution instruction, the indicator input parameter, and the indicator output field.

6. The method according to claim 2, characterized in that The parsing of each operator model in the operator chain diagram to obtain multiple BEAM model instructions includes: Analyze the operator model with 0 information content, obtain a BEAM model instruction, and reduce the information content of the next operator model by 1; When the information amount of the next operator model is 0, the next operator model is parsed to obtain another BEAM model instruction.

7. A data indicator generation system, characterized in that: The system includes: a front-end indicator model arrangement canvas module, a back-end model parser module, a parameter parser module, a BEAM model compiler module, a BEAM model optimizer module, a BEAM data processing flow generator module, an execution engine module, a data source module, and an indicator library module; The front-end indicator model arrangement canvas module is used to display an interactive interface, which includes multiple operator components; in response to the user's operation on the multiple operator components, determine the target operator component; and generate configuration information according to the target operator component; The backend model parser module is used to generate an operator chain graph according to the configuration information; the operator chain graph is formed by connecting multiple operator models; The parameter parser module is used to parse each operator model in the operator link graph; The BEAM model compiler module is used to compile multiple BEAM model instructions according to the parsing results; The BEAM model optimizer module is used to optimize the multiple BEAM model instructions to obtain multiple target execution instructions; The BEAM data processing flow generator module is used to generate the BEAM data processing flow according to the target execution instruction; The execution engine module is used to obtain source data from multiple databases of the data source module, and merge the multiple source data in the memory to obtain target data; use the pre-generated BEAM data processing flow to calculate the target data in the memory to obtain target data indicators for the multiple source data; The data source module is used to store source data of each database; The indicator library module is used to store the target data indicators.

8. A device for generating data indicators, characterized in that: The device comprises: An acquisition module, used for acquiring source data from multiple databases respectively in response to an operation of generating data indicators; A merging module is used to merge multiple source data in memory to obtain target data; The indicator generation module is used to use the pre-generated BEAM data processing flow to calculate the target data in the memory to obtain target data indicators for the multiple source data.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method for generating a data indicator as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a data indicator as described in any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Method, device and electronic device for generating index data under abnormal environment

    CN109145996A

  • Multi-dimensional index calculation method and device

    CN110955654A

  • Data processing method, device and equipment

    CN111291095A

  • Real-time data processing system and method

    CN111813871A

  • Cross-platform spatio-temporal big data distributed processing method and software

    CN112732852A

Cited By

  • Test instruction generation method and device, electronic equipment, medium and product

    CN120973695A

  • Test instruction generation method and device, electronic equipment, medium and product

    CN120973695B