Data statistical processing method and system, electronic device and storage medium

Abstract

The application provides a data statistical processing method and system, electronic equipment and storage medium, the data source of the method at least contains the data table which has entered the lake and the data table which has been recorded into the data source; the connection information between the upstream data table and the downstream data table in the data source is used to determine the join statement, and the join statement is used to determine the query statement of the data source; then the query statement is used to determine the calculation data of the data source, and the data interaction instruction corresponding to the data source is generated according to the calculation data; finally, the data interaction instruction is used to interact with the data source in sequence, and the interaction result of the data source is saved to the downstream data table after statistical processing; the method fully utilizes the connection information between the lake data and the lake data in the data source to generate relevant data interaction instructions, and the data interaction instructions can be used to automatically interact with the data source, thereby reducing the professional requirement of the operation and processing data of the business personnel, and improving the business processing efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of data governance, and in particular to a data statistical processing method, system, electronic device, and storage medium. Background Technology

[0002] In data processing, it is often necessary to filter data from large amounts of data storage under various constraints to obtain the desired data. Therefore, the process of setting constraints requires a high level of expertise in data manipulation, necessitating a thorough understanding of the structure of the stored data and relevant data processing skills. Since most data processing is achieved through manually writing and executing statements, this process requires business personnel to consider not only the multidimensionality of the data but also the executability of the statements. Consequently, the process of multidimensional data association, fusion, and governance suffers from high technical barriers, poor usability, and low efficiency in use and maintenance due to the manual writing of execution statements. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a data statistical processing method, system, electronic device and storage medium. This method makes full use of the connection information between data inside and outside the lake in the data source to generate relevant data interaction instructions, realizes flexible configuration of the process of multi-dimensional data association and fusion governance, and can use data interaction instructions to automatically interact with the data source, thereby reducing the professional requirements of business personnel in operating and processing data, and improving business processing efficiency and accuracy.

[0004] In a first aspect, embodiments of the present invention provide a data statistical processing method, the method comprising:

[0005] Obtain the data source; the data source must include at least the data tables that have been entered into the lake and the data tables that have been entered into the data source; the data tables contain database data;

[0006] Based on the user's business processing logic, associate the data tables that have been entered into the lake with the data tables that have been entered into the data source; determine the join statement based on the association result, and use the join statement to determine the query statement for the data source;

[0007] Use query statements to determine the calculated data of the data source, and generate data interaction instructions corresponding to the data source based on the calculated data;

[0008] The system sequentially interacts with the data source using data interaction commands, and then saves the interaction results from the data source to the downstream data table after statistical processing.

[0009] In one implementation, determining the join statement based on the join result includes:

[0010] Retrieve the join order and join conditions between upstream and downstream data tables in the data source;

[0011] The query information of the data source is determined based on the order of the join tables and the join conditions, and the join table statement is generated based on the query information.

[0012] In one implementation, the process of determining the query statement for the data source using a join statement includes:

[0013] Use join statements to determine the upstream directory of the upstream data table in the data source, generate a query statement to determine the first directory based on the upstream directory, and add the first directory query statement to the preset SQL group;

[0014] Use join statements to determine the downstream directory where the downstream data tables in the data source are located, generate a query statement to determine the second directory based on the downstream directory, and add the second directory query statement to the preset SQL group;

[0015] The query statements that determine the first directory query statement and the second directory query statement as the data source.

[0016] In one implementation, the steps of determining the computational data of the data source using a query statement and generating a data interaction instruction corresponding to the data source based on the computational data include:

[0017] Use a query statement to query all fields in the upstream data table, identify the operation instructions corresponding to all fields as calculated fields, and identify the calculated fields as the first calculated data;

[0018] The calculation conditions corresponding to all fields are determined based on the keywords contained in the query statement, and these calculation conditions are used as the second set of calculation data. Among them, the keywords must at least correspond to the following operations: filtering, grouping, and sorting the data corresponding to the fields.

[0019] Generate data interaction instructions for all fields using the first and second calculated data.

[0020] In one implementation, the process of querying all fields in the upstream data table using a query statement and determining the operation instructions corresponding to all fields as calculated fields includes:

[0021] When an external data source exists, the field type corresponding to the upstream data table is retrieved based on the mapping relationship between the external data source and the upstream data table; where the external data source is structured data.

[0022] Update all fields in a data table using field types.

[0023] In one implementation, the process of determining the calculation conditions corresponding to all fields based on keywords contained in the query statement includes:

[0024] Determine whether the query statement contains keywords;

[0025] If the keyword "where" is included, the calculation condition for the field will be filtering.

[0026] If the keyword "group by" is included, the calculation condition for the corresponding field will be grouping;

[0027] If the keyword "order by" is included, the calculation condition for the field will be sorting.

[0028] In one implementation, the process of sequentially interacting with the data source using data interaction commands includes:

[0029] Retrieve the corresponding interaction task in the data interaction instruction, and retrieve the corresponding task execution parameters in the interaction task; wherein, the interaction task includes one or more of the following: running mode, running memory, startup time, data mechanism, data expiration time, storage data table, and synchronization time;

[0030] The execution statement corresponding to the interactive task is determined based on the interactive task and its running parameters.

[0031] The data source is interacted with sequentially using execution statements.

[0032] Secondly, embodiments of the present invention provide a data statistical processing system, the system comprising:

[0033] The data source acquisition module is used to acquire data sources; the data sources include at least the data tables that have been entered into the lake and the data tables that have been entered into the data sources; the data tables contain database data;

[0034] The join table determination module is used for user business processing logic to join data tables that have been entered into the lake and data tables that have been entered into the data source; it determines the join table statement based on the join table result, and uses the join table statement to determine the query statement of the data source;

[0035] The computational data processing module is used to determine the computational data of the data source using query statements, and to generate data interaction instructions corresponding to the data source based on the computational data.

[0036] The statistical processing module is used to interact with the data source sequentially using data interaction commands, and then save the interaction results of the data source to the downstream data table after statistical processing.

[0037] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the steps of the data statistical processing method provided in the first aspect.

[0038] Fourthly, embodiments of the present invention also provide a storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the steps of the data statistical processing method provided in the first aspect.

[0039] This invention provides a data statistical processing method, system, electronic device, and storage medium. The method first acquires a data source; the data source includes at least a data table already included in the lake and a data table already entered into the data source; the data tables contain database data. Then, based on the connection information between the upstream and downstream data tables in the data source, a join statement is determined, and the join statement is used to determine the query statement for the data source. Next, the query statement is used to determine the calculated data for the data source, and data interaction instructions corresponding to the data source are generated based on the calculated data. Finally, the data interaction instructions are used sequentially to interact with the data source, and the interaction results are statistically processed and saved to the downstream data table. This method fully utilizes the connection information between data within and outside the lake in the data source to generate relevant data interaction instructions, enabling flexible configuration of the multi-dimensional data association and fusion governance process. It also allows for automatic interaction with the data source using data interaction instructions, thereby reducing the professional requirements for data processing by business personnel and improving business processing efficiency and accuracy.

[0040] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0042] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 A flowchart of a data statistical processing method provided in an embodiment of the present invention;

[0044] Figure 2 A flowchart illustrating the process of determining join statements based on association results in a data statistical processing method provided in this embodiment of the invention;

[0045] Figure 3 A flowchart of a query statement for determining the data source using a join statement, provided in an embodiment of the present invention, is included.

[0046] Figure 4 In a data statistical processing method provided in an embodiment of the present invention, a flowchart is generated based on a query statement to determine the calculated data of the data source and to generate a data interaction instruction corresponding to the data source based on the calculated data.

[0047] Figure 5 In a data statistical processing method provided in an embodiment of the present invention, a flowchart is used to query all fields in the upstream data table using a query statement and determine the operation instructions corresponding to all fields as the calculation fields.

[0048] Figure 6 A flowchart illustrating the process of determining the calculation conditions corresponding to all fields based on keywords contained in a query statement in a data statistical processing method provided in this embodiment of the invention.

[0049] Figure 7 A flowchart illustrating how data interaction commands are sequentially used to interact with a data source in a data statistical processing method provided in this embodiment of the invention.

[0050] Figure 8 A flowchart of another data statistical processing method provided in an embodiment of the present invention;

[0051] Figure 9 This is a schematic diagram of the structure of a data statistical processing system provided in an embodiment of the present invention;

[0052] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0053] icon:

[0054] 910 - Data Source Acquisition Module; 920 - Table Joining Module; 930 - Calculated Data Processing Module; 940 - Statistical Processing Module;

[0055] 101 - Processor; 102 - Memory; 103 - Bus; 104 - Communication interface. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] In data processing, it is often necessary to filter data from large amounts of data storage under various constraints to obtain the desired data. Therefore, the process of setting constraints requires a high level of expertise in data manipulation, necessitating a thorough understanding of the structure of the stored data and relevant data processing skills. Since most data processing is achieved through manually writing and executing statements, this process requires business personnel to consider not only the multidimensionality of the data but also the executability of the statements. Consequently, the process of multidimensional data association, fusion, and governance suffers from high technical barriers, poor usability, and low efficiency in use and maintenance due to the manual writing of execution statements.

[0058] Based on this, the present invention provides a data statistical processing method, system, electronic device and storage medium. The method makes full use of the connection information between data inside and outside the lake in the data source to generate relevant data interaction instructions, realizes the flexible configuration of the process of multi-dimensional data association and fusion governance, and can use data interaction instructions to automatically interact with the data source, thereby reducing the professional requirements of business personnel to operate and process data, and improving the efficiency and accuracy of business processing.

[0059] To facilitate understanding of this embodiment, a data statistical processing method disclosed in this embodiment of the invention will first be described in detail, such as... Figure 1 As shown, the method includes:

[0060] Step S101: Obtain the data source; wherein, the data source includes at least the data tables that have been entered into the lake and the data tables that have been entered into the data source; the data tables contain database data.

[0061] The data sources in this step include multiple data sources, encompassing both data already included in the lake and data entered into the data sources. Each data source contains a data table, and each data table contains a database.

[0062] Step S102: Based on the user's business processing logic, associate the data tables that have been entered into the lake with the data tables that have been entered into the data source; determine the join statement based on the association result, and use the join statement to determine the query statement for the data source.

[0063] This step can be understood as the process of selecting join tables. It stipulates that at least one upstream data source is required for the data to be used. The join statement is generated based on the join order and join conditions, using the join information between the upstream and downstream tables contained in the data source. Once the join statement is determined, it can be set as a query statement to query the data source. Generally, the same join information cannot appear in the same table, and join information from the same table cannot contain the same fields.

[0064] Step S103: Use a query statement to determine the calculated data of the data source, and generate a data interaction instruction corresponding to the data source based on the calculated data.

[0065] This step primarily corresponds to the data processing logic. The calculated data includes operations on fields and data filtering. Specifically, the calculated data can correspond to calculated fields and calculation conditions. Calculated fields involve operations on fields, including adding fields and performing function operations on fields; calculation conditions involve data filtering, including function operations, data grouping and statistics, and data filtering. In other words, this step selects query field information based on the user's business processing logic, generates query data, and ultimately obtains the corresponding data interaction instructions.

[0066] The calculated data contains various instructions for manipulating fields, so the data interaction instructions corresponding to the data source can be obtained from the calculated data.

[0067] Step S104: Use data interaction commands to interact with the data source in sequence, and save the interaction results of the data source to the downstream data table after statistical processing.

[0068] This step can be understood as the process of saving the results. By sequentially using data interaction commands to interact with the data source, the obtained interaction results are statistically processed to run the data processing task, and the processing results are saved to the downstream data table.

[0069] In practical scenarios, the data statistical processing can be divided into five parts: table joins, calculation conditions, calculated fields, and result saving. These five parts are further divided into four levels based on function: data source (level 1), table joins (level 2), calculation conditions and calculated fields (level 3), and calculation results (level 4). By setting rules and sorting the levels from smallest to largest, the corresponding ranking results are obtained. According to the data processing logic, higher-level parts can point to parts of the same or lower level, and the same component cannot be connected to each other.

[0070] Therefore, by flexibly using the above four steps to complete the data processing workflow, the professional requirements for business personnel to operate and process data can be reduced, and the efficiency and accuracy of business processing can be improved.

[0071] In one implementation, the process of determining the join statement based on the join result is as follows: Figure 2 As shown, it includes:

[0072] Step S201: Obtain the table connection order and connection conditions between the upstream and downstream data tables in the data source;

[0073] Step S202: Determine the query information of the data source based on the join order and join conditions, and generate a join statement based on the query information.

[0074] When a table join is required, a join statement is generated based on the join order and join conditions. The join order can be obtained by concatenating the join information in the order it is connected. The join conditions are that the same join information cannot appear in the same table, and the join information of the same table cannot have the same fields.

[0075] In real-world scenarios, data statistical processing is primarily implemented using SQL statements. Therefore, it's essential to first prepare SQL statement templates, including: SQL templates for creating directories, SQL templates for creating mapping tables for external data tables, SQL templates for creating downstream data tables, and SQL templates for querying and saving data. By organizing all necessary data source types, categorizing field types according to different data sources, and generating a field type mapping table based on the correspondence between external data source types and data lake types, a mapping relationship table for field types can be created.

[0076] In one implementation, the process of determining the query statement for the data source using a join statement is as follows: Figure 3 As shown, it includes:

[0077] Step S301: Use a join statement to determine the upstream directory where the upstream data table in the data source is located, generate a query statement to determine the first directory based on the upstream directory, and add the first directory query statement to the preset SQL group.

[0078] Obtain the directory where the upstream data is located based on the selected data source, generate the SQL statement for creating the directory based on the above-mentioned directory creation template SQL, and designate this statement as the first directory query statement, while adding it to the preset SQL group.

[0079] Step S302: Use the join statement to determine the downstream directory where the downstream data table in the data source is located, generate a query statement to determine the second directory based on the downstream directory, and add the second directory query statement to the preset SQL group.

[0080] Based on the directory information in the selected downstream table, generate the downstream directory name from the pinyin abbreviation of the topic where the downstream data is located. Generate the SQL statement to create the directory based on the directory creation template SQL, and add this statement to the SQL group. Generate downstream field information based on the calculated field information and the user-selected primary key information. Generate the downstream table creation statement based on the template for creating downstream data tables, and add it to the SQL group.

[0081] Step S303: Determine the first directory query statement and the second directory query statement as the query statements for the data source.

[0082] In one implementation, step S103 involves using a query statement to determine the computational data of the data source and generating a data interaction instruction corresponding to the data source based on the computational data. Figure 4 As shown, it includes:

[0083] Step S401: Use a query statement to query all fields in the upstream data table, determine the operation instructions corresponding to all fields as calculated fields, and determine the calculated fields as the first calculated data.

[0084] After obtaining the calculated fields, a list of fields to be queried is generated based on the filter fields and function information in the calculated fields, which includes the field names and field types.

[0085] Step S402: Determine the calculation conditions corresponding to all fields based on the keywords contained in the query statement, and determine the calculation conditions as the second calculation data; wherein, the keywords correspond at least to the operations of filtering, grouping and sorting the data corresponding to the fields.

[0086] The calculation conditions mainly consist of three parts: filtering, grouping statistics, and sorting. These primarily address data filtering processes, including sorting, data grouping statistics, and data filtering.

[0087] Step S403: Generate data interaction instructions corresponding to all fields using the first and second calculated data.

[0088] When an external data source exists, an external data source mapping table is generated based on the connection information of the external data source and added to the SQL group. In one implementation, a query statement is used to query all fields in the upstream data table, and the operation instructions corresponding to all fields are determined as calculated fields, such as... Figure 5 As shown, it includes:

[0089] Step S501: When an external data source exists, the field type corresponding to the upstream data table is obtained by querying the mapping relationship between the external data source and the upstream data table; wherein, the external data source is structured data.

[0090] Step S502: Update all fields in the data table using the field type.

[0091] In one implementation, the process of determining the calculation conditions corresponding to all fields based on the keywords contained in the query statement, such as... Figure 6 As shown, it includes:

[0092] Step S601: Determine whether the query statement contains keywords;

[0093] In step S602, if the keyword "where" is included, the calculation condition corresponding to the field is filtering; if the keyword "group by" is included, the calculation condition corresponding to the field is grouping; if the keyword "order by" is included, the calculation condition corresponding to the field is sorting.

[0094] Specifically, the filtering and selection of field attributes involves WHERE clauses for field data filtering, GROUP BY clauses for statistics, and ORDER BY clauses for sorting based on fields. For each component present, a condition is appended to the query criteria, and the final query statement is generated based on the existing condition information.

[0095] In one implementation, the process involves sequentially interacting with the data source using data interaction commands, such as... Figure 7 As shown, it includes:

[0096] Step S701: Obtain the corresponding interaction task in the data interaction instruction, and obtain the corresponding task running parameters in the interaction task; wherein, the interaction task includes one or more of the following: running mode, running memory, startup time, data mechanism, data expiration time, storage data table, and synchronization time;

[0097] Step S702: Determine the execution statement corresponding to the interactive task based on the interactive task and its task execution parameters;

[0098] Step S703: Use execution statements to interact with the data source sequentially.

[0099] The above steps allow for dynamic modification of task execution parameters such as running mode, memory usage, startup time, data mechanism, data expiration time, storage tables, and synchronization time. By integrating upstream query data tables, query field information, query conditions, and downstream data tables, and based on the query and save data template, data saving statements are generated and added to the SQL group. Finally, according to the task configuration information, the SQL is submitted to the relevant cluster, and the SQL statements are executed sequentially according to the order added to the SQL group, thereby processing data statistics.

[0100] The above process is as follows: Figure 8 As shown in the flowchart of another data statistical processing method, SQL statements are generated based on business processing logic. The specific logic is as follows:

[0101] Step 1: Prepare SQL statement templates, including template SQL for creating a directory, template SQL for creating a mapping table for external data tables, template SQL for creating downstream data tables, and template SQL for querying and saving data. Organize all required data source types, categorize field types according to different data sources, and generate a field type mapping table based on the correspondence between external data source types and data lake types.

[0102] Step 2: Based on the selected data source, obtain the directory where the upstream data is located, and generate the SQL statement for creating the directory according to the directory creation template, then add it to the SQL group. When an external data source exists, generate an external data source mapping table based on the connection information of the external data source, and add it to the SQL group.

[0103] Step 3: When table joins are required, generate a join statement based on the join order and join conditions, setting it to query data from the data source tables. The join order is determined by concatenating the join information. It is stipulated that the same join entry cannot appear in the same table, and join entries from the same table cannot contain the same fields.

[0104] Step 4: Query all fields in the upstream data table. Generate a list of fields to be queried based on the filter fields and function information in the calculated fields, including field names and field types. When an external data source exists, query the downstream field types corresponding to the upstream field types based on the external data source type and the mapping table, generating new upstream data table fields. It is stipulated that external data sources can only be structured data.

[0105] Step 5: Based on the conditions in the calculation, the process is divided into three parts: filtering, statistics, and sorting. Filtering involves handling field attributes and using the WHERE clause for field data filtering; statistics involves grouping queries using GROUP BY; and sorting involves sorting by field using ORDER BY. For each part, a condition is appended to the query conditions, and the query statement is generated based on the existing condition information.

[0106] Step 6: Based on the directory information in the selected downstream table, generate the downstream directory name from the pinyin abbreviation of the topic where the downstream data is located. Generate the SQL statement for creating the directory based on the directory creation template and add it to the SQL group. Generate downstream field information based on the calculated field information and the user-selected primary key information. Generate the downstream table creation statement based on the template for creating the downstream data table and add it to the SQL group.

[0107] Step 7: Integrate the upstream query data table, query field information, query conditions, and downstream data table. Based on the query and save data template, generate the data save statement and add it to the SQL group.

[0108] Step 8: Based on the task configuration information, submit the SQL to the cluster and execute the SQL in the order they were added to the SQL group to run the data processing task.

[0109] As can be seen from the data statistical processing method mentioned in the above embodiments, this method makes full use of the connection information between the data inside the lake and the data outside the lake in the data source to generate relevant data interaction instructions, realizes the flexible configuration of the process of multi-dimensional data association and fusion governance, and can use data interaction instructions to automatically interact with the data source, thereby reducing the professional requirements of business personnel in operating and processing data, and improving the efficiency and accuracy of business processing.

[0110] Regarding the data statistical processing method provided in the foregoing embodiments, this invention provides a data statistical processing system, such as... Figure 9As shown, the system includes:

[0111] The data source acquisition module 910 is used to acquire data sources; wherein, the data source includes at least the data tables that have been entered into the lake and the data tables that have been entered into the data source; the data tables contain database data;

[0112] The join table determination module 920 is used to join the data tables that have been entered into the lake and the data tables that have been entered into the data source according to the user's business processing logic; determine the join table statement based on the join table result, and use the join table statement to determine the query statement of the data source;

[0113] The calculation data processing module 930 is used to determine the calculation data of the data source using a query statement, and to generate data interaction instructions corresponding to the data source based on the calculation data;

[0114] The statistical processing module 940 is used to interact with the data source sequentially using data interaction commands, and then save the interaction results of the data source to the downstream data table after statistical processing.

[0115] In one implementation, the join determination module 920, in the process of determining the join statement based on the join result, is also used to: obtain the join order and connection conditions between the upstream data table and the downstream data table in the data source; determine the query information of the data source based on the join order and connection conditions; and generate the join statement based on the query information.

[0116] In one implementation, during the process of determining the query statement of the data source using the join statement, the join statement determination module 920 is further configured to: determine the upstream directory of the upstream data table in the data source using the join statement, generate a first directory determination query statement based on the upstream directory, and add the first directory query statement to a preset SQL group; determine the downstream directory of the downstream data table in the data source using the join statement, generate a second directory determination query statement based on the downstream directory, and add the second directory query statement to a preset SQL group; and determine the first directory query statement and the second directory query statement as the query statement of the data source.

[0117] In one embodiment, the computational data processing module 930 is further configured to: query all fields in the upstream data table using a query statement, determine the operation instructions corresponding to all fields as computational fields, and determine the computational fields as first computational data; determine the computational conditions corresponding to all fields based on the keywords contained in the query statement, and determine the computational conditions as second computational data; wherein, the keywords at least correspond to: operations of filtering, grouping, and sorting the data corresponding to the fields; and generate data interaction instructions corresponding to all fields using the first computational data and the second computational data.

[0118] In one implementation, the data processing module 930, while querying all fields in the upstream data table using a query statement and determining the operation instructions corresponding to all fields as calculated fields, is also used to: when an external data source exists, query the field type corresponding to the upstream data table based on the mapping relationship between the external data source and the upstream data table; wherein the external data source is structured data; and update all fields in the data table using the field type.

[0119] In one implementation, the data processing module 930, in the process of determining the calculation conditions corresponding to all fields based on the keywords contained in the query statement, is further configured to: determine whether the query statement contains keywords; if it contains the keyword "where", then the calculation condition corresponding to the field is filtering; if it contains the keyword "group by", then the calculation condition corresponding to the field is grouping; if it contains the keyword "order by", then the calculation condition corresponding to the field is sorting.

[0120] In one implementation, during the process of sequentially interacting with the data source using data interaction instructions, the statistical processing module 940 is further configured to: obtain the interaction task corresponding to the data interaction instruction, and obtain the task execution parameters corresponding to the interaction task; wherein, the interaction task includes one or more of the following: running mode, running memory, startup time, data mechanism, data expiration time, stored data table, and synchronization time; determine the execution statement corresponding to the interaction task based on the interaction task and its task execution parameters; and sequentially interact with the data source using the execution statement.

[0121] The data statistical processing system provided in this invention fully utilizes the connection information between data within and outside the lake in the data source to generate relevant data interaction instructions. This enables flexible configuration of the process of multi-dimensional data association, fusion, and governance, and allows for automatic interaction with the data source using data interaction instructions. This reduces the professional requirements for business personnel to operate and process data, and improves the efficiency and accuracy of business processing.

[0122] The data statistical processing system provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned data statistical processing method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0123] This embodiment also provides an electronic device, the structural schematic diagram of which is shown below. Figure 10 As shown, the device includes a processor 101 and a memory 102; wherein, the memory 102 is used to store one or more computer instructions, which are executed by the processor to implement the above-mentioned data statistical processing method.

[0124] Figure 10The electronic device shown also includes a bus 103 and a communication interface 104, with the processor 101, communication interface 104 and memory 102 connected via the bus 103.

[0125] The memory 102 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. The bus 103 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0126] The communication interface 104 is used to connect to at least one user terminal and other network units through a network interface, and to send encapsulated IPv4 packets or IPv4 packets to the user terminal through the network interface.

[0127] Processor 101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 101 or by instructions in software form. The processor 101 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 102. The processor 101 reads the information in memory 102 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0128] This invention also provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of the methods described in the foregoing embodiments.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0132] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data statistical processing method, characterized in that, The method includes: Obtain the data source; wherein, the data source includes at least a data table that has been entered into the lake and a data table that has been entered into the data source; the data table contains database data; the data table includes an upstream data table and a downstream data table; Based on the user's business processing logic, associate the data tables that have been entered into the lake with the data tables that have been entered into the data source; determine the join statement based on the association result, and use the join statement to determine the query statement for the data source; The query statement is used to determine the calculated data of the data source, and a data interaction instruction corresponding to the data source is generated based on the calculated data. The data interaction commands are used sequentially to interact with the data source, and the interaction results of the data source are statistically processed and saved to the downstream data table; The step of determining the calculated data of the data source using the query statement and generating a data interaction instruction corresponding to the data source based on the calculated data includes: The query statement is used to query all fields in the upstream data table, the operation instructions corresponding to all fields are identified as calculated fields, and the calculated fields are identified as the first calculated data. The calculation conditions corresponding to all fields are determined based on the keywords contained in the query statement, and the calculation conditions are determined as the second calculation data; wherein, the keywords at least correspond to: the operation of filtering, grouping and sorting the data corresponding to the fields; Data interaction instructions corresponding to all fields are generated using the first calculated data and the second calculated data.

2. The data statistical processing method according to claim 1, characterized in that, The process of determining the join statement based on the join result includes: Obtain the join order and join conditions between the upstream data table and the downstream data table in the data source; The query information of the data source is determined based on the join order and the join conditions, and the join statement is generated based on the query information.

3. The data statistical processing method according to claim 1, characterized in that, The process of determining the query statement for the data source using the join statement includes: The upstream directory of the upstream data table in the data source is determined using the join statement, and a first directory query statement is generated based on the upstream directory, and the first directory query statement is added to a preset SQL group. The downstream directory of the downstream data table in the data source is determined by the join statement, and a second directory query statement is generated based on the downstream directory. The second directory query statement is then added to a preset SQL group. The first directory query statement and the second directory query statement are determined as the query statements of the data source.

4. The data statistical processing method according to claim 1, characterized in that, The process of querying all fields in the upstream data table using the query statement and determining the operation instructions corresponding to all fields as calculation fields includes: When the data source contains an external data table, the field type corresponding to the upstream data table is obtained by querying according to the mapping relationship between the external data table and the upstream data table; wherein, the external data table is a structured data table; Update all fields in the data table using the field type.

5. The data statistical processing method according to claim 1, characterized in that, The process of determining the calculation conditions corresponding to all fields based on the keywords contained in the query statement includes: Determine whether the query statement contains the keyword; If the keyword "where" is included, then the calculation condition corresponding to the field is filtering; If the keyword "group by" is included, then the calculation condition corresponding to the field is grouping; If the keyword "order by" is included, then the calculation condition corresponding to the field is sorting.

6. The data statistical processing method according to claim 1, characterized in that, The process of sequentially interacting with the data source using the data interaction instructions includes: Obtain the corresponding interaction task in the data interaction instruction, and obtain the corresponding task running parameters in the interaction task; wherein, the interaction task includes one or more of the following: running mode, running memory, startup time, data mechanism, data expiration time, storage data table, and synchronization time; The execution statement corresponding to the interactive task is determined based on the interactive task and its running parameters. The data source is interacted with sequentially using the execution statements.

7. A data statistical processing system, characterized in that, The system includes: A data source acquisition module is used to acquire data sources; wherein, the data source includes at least a data table that has been entered into the lake and a data table that has been entered into the data source; the data table contains database data; the data table includes an upstream data table and a downstream data table; The table join determination module is used to join the data tables that have been entered into the lake and the data tables that have been entered into the data source according to the user's business processing logic; determine the table join statement according to the join result, and use the table join statement to determine the query statement of the data source; The computational data processing module is used to determine the computational data of the data source using the query statement, and to generate a data interaction instruction corresponding to the data source based on the computational data; The statistical processing module is used to sequentially interact with the data source using the data interaction instructions, and to save the interaction results of the data source to the downstream data table after statistical processing. The computational data processing module is further configured to: query all fields in the upstream data table using the query statement, determine the operation instructions corresponding to all fields as computational fields, and determine the computational fields as first computational data; determine the computational conditions corresponding to all fields based on the keywords contained in the query statement, and determine the computational conditions as second computational data; wherein, the keywords at least correspond to: operations of filtering, grouping, and sorting the data corresponding to the fields; and generate data interaction instructions corresponding to all fields using the first computational data and the second computational data.

8. An electronic device, characterized in that, include: Processors and storage devices; The storage device stores computer-executable instructions that can be executed by a processor, and the processor executes the computer-executable instructions to implement the steps of the data statistical processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the steps of the data statistical processing method according to any one of claims 1 to 6.

Images