Batch-stream fusion information processing method, device, and storage medium

Abstract

The present application discloses an information processing method and device for batch-stream fusion, and a storage medium. The method includes obtaining an index based on an input query statement; extracting a pre-calculated index data segment as a query result based on the index; and extracting a newly calculated index data segment to update the query result. The present application solves the technical problem that it is difficult to perform fusion analysis on real-time data and offline data.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to an information processing method and device for batch-stream fusion, and a storage medium. Background Art

[0002] With the advent of Industry 4.0 and the 5G era, people have an increasing demand for the timeliness of data processing. Studies have shown that the value of data decreases with the passage of time. Conversely, the more real-time the data is, the higher its value. Against this background, a number of excellent open source stream computing frameworks such as Storm, Spark, Kafka, and Flink have emerged in the industry. However, although many stream computing frameworks claim to have batch and stream processing capabilities, in fact, due to the differences in usage scenarios and focus between batch processing and stream computing - stream computing pays more attention to the timeliness of data, while batch processing pays more attention to the integrity, accuracy and computing cost of data, batch processing systems such as Hive cannot be completely replaced. This makes it very difficult for people to merge and analyze real-time data with offline data. Problems such as inconsistent data caliber, inconsistent semantics, inability to query across systems, and query performance that cannot meet requirements often become insurmountable gaps that plague big data architects and engineers.

[0003] In summary, the prior art has the following technical problems:

[0004] It becomes very difficult to integrate real-time data with offline data for analysis. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an information processing method for batch-stream fusion, comprising the steps of:

[0006] Get the index based on the input query statement;

[0007] Extract pre-calculated index data segments based on the index as query results;

[0008] Extract the newly calculated index data segment to update the query results.

[0009] Preferably, the extracting pre-calculated index data segments based on the index as the query result specifically includes:

[0010] Get index statistics;

[0011] Retrieve the pre-created index with the statistics in memory;

[0012] Locating pre-computed index data segments in a storage medium based on the pre-created index;

[0013] The located pre-computed index data segment is used as the query result.

[0014] Preferably, the pre-calculated index data segment specifically includes:

[0015] Pre-create indexes based on pre-defined unified model definitions;

[0016] Pre-calculate the index data segment based on the pre-created index;

[0017] Stores statistics of pre-created indexes in memory and stores pre-computed index data segments in storage media.

[0018] Preferably, the pre-creating index based on the pre-defined unified model definition specifically includes:

[0019] Obtain a pre-created stream data source mapping table and a batch data source mapping table, wherein the stream data source mapping table is a fact table and the batch data source mapping table is a shadow table bound thereto;

[0020] Get the dimensions and measures of the pre-created fact table and shadow table connection;

[0021] Based on the acquired dimensions and measures, determine the connection relationship between the fact table and shadow table and the dimension table;

[0022] Pre-create indexes based on join relationship definitions.

[0023] Preferably, the step of calculating the index data segment based on the pre-created index specifically includes:

[0024] Get pre-created indexes, where pre-created indexes are divided into batch indexes, stream indexes, and fusion indexes;

[0025] Based on the pre-created index, pre-calculation is performed in the data source through a computing engine, wherein the computing engine includes a batch processing computing engine and a stream processing computing engine, and the data source includes a batch data source and a stream data source;

[0026] Precomputed index data segments are obtained based on the precomputed data, wherein the precomputed index data segments are divided into batch index data segments and stream index data segments.

[0027] Preferably, storing the statistical information of the pre-created index in the memory specifically includes:

[0028] Extract statistics for pre-created indexes, where the statistics include number of rows, size, and cardinality.

[0029] Preferably, the extracting of the newly calculated index data segment to update the query result specifically includes:

[0030] Perform new calculations in data sources based on indexes, where indexes are divided into batch indexes, stream indexes, and fusion indexes, and data sources include batch data sources and stream data sources;

[0031] The newly calculated index data segment is obtained by a new calculation through a calculation engine, wherein the calculation engine includes a batch processing calculation engine and a stream processing calculation engine, and the newly calculated index data segment is divided into a batch index data segment and a stream index data segment;

[0032] storing the newly calculated index data segment in a storage medium;

[0033] Extract the newly calculated index data segment in the storage medium to update the query result.

[0034] An information processing device for batch-stream fusion, characterized by comprising:

[0035] The client is used to obtain the index based on the input query statement;

[0036] A query engine, used to extract pre-calculated index data segments as query results based on the index;

[0037] The update module is used to extract the newly calculated index data segment to update the query results.

[0038] Preferably, the query engine:

[0039] Get index statistics;

[0040] Retrieve the pre-created index with the statistics in memory;

[0041] Locating pre-computed index data segments in a storage medium based on the pre-created index;

[0042] The located pre-computed index data segment is used as the query result.

[0043] Preferably, the query engine comprises:

[0044] A pre-created index definition module, used to define a pre-created index based on a pre-defined unified model;

[0045] A pre-calculation module, used to pre-calculate index data segments based on pre-created indexes;

[0046] The information storage module stores the statistical information of the pre-created index in the memory and stores the pre-calculated index data segments in the storage medium.

[0047] Preferably, the pre-created index definition module:

[0048] Obtain a pre-created stream data source mapping table and a batch data source mapping table, wherein the stream data source mapping table is a fact table and the batch data source mapping table is a shadow table bound thereto;

[0049] Get the dimensions and measures of the pre-created fact table and shadow table connection;

[0050] Based on the acquired dimensions and measures, determine the connection relationship between the fact table and shadow table and the dimension table;

[0051] Pre-create indexes based on join relationship definitions.

[0052] Preferably, the pre-calculation module:

[0053] Get pre-created indexes, where pre-created indexes are divided into batch indexes, stream indexes, and fusion indexes;

[0054] Based on the pre-created index, pre-calculation is performed in the data source through a computing engine, wherein the computing engine includes a batch processing computing engine and a stream processing computing engine, and the data source includes a batch data source and a stream data source;

[0055] Precomputed index data segments are obtained based on the precomputed data, wherein the precomputed index data segments are divided into batch index data segments and stream index data segments.

[0056] Preferably, the information storage module:

[0057] Extract statistics for pre-created indexes, where the statistics include number of rows, size, and cardinality.

[0058] Preferably, the update module:

[0059] Perform new calculations in data sources based on indexes, where indexes are divided into batch indexes, stream indexes, and fusion indexes, and data sources include batch data sources and stream data sources;

[0060] The newly calculated index data segment is obtained by a new calculation through a calculation engine, wherein the calculation engine includes a batch processing calculation engine and a stream processing calculation engine, and the newly calculated index data segment is divided into a batch index data segment and a stream index data segment;

[0061] storing the newly calculated index data segment in a storage medium;

[0062] Extract the newly calculated index data segment in the storage medium to update the query result.

[0063] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and wherein the computer program can be executed in the processor to implement any of the above methods.

[0064] A storage medium stores a computer program, wherein the computer program can implement any of the above methods when executed in a processor.

[0065] The present invention achieves the technical effect of simultaneously analyzing historical data and real-time data through one SQL query statement by defining a unified model; achieves the technical effect of effectively reducing query response time by using pre-calculation and real-time merging technology; achieves the technical effect of ensuring system stability by storing calculation results in a storage medium for repeated use; and achieves the technical effect of updating query results by continuously storing new calculation results in a storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flowchart of the information processing method for batch-stream fusion of the present application;

[0067] Figure 2 A predefined unified model diagram for this application;

[0068] Figure 3 A flowchart for pre-creating an index for the definitions of this application;

[0069] Figure 4 This is a flowchart of the present application for calculating and obtaining index data segments based on pre-created indexes. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that in the description of the present invention, unless otherwise clearly specified and limited, the term "storage medium" can be various media that can store computer programs, such as ROM, RAM, magnetic disk or optical disk. The term "processor" can be a chip or circuit with data processing function, such as CPLD (Complex Programmable Logic Device), FPGA (Field-Programmable Gate Array), MCU (Microcontroller Unit), PLC (Programmable Logic Controller) and CPU (Central Processing Unit). The term "electronic device" can be any device with data processing function and storage function, which can generally include fixed terminals and mobile terminals. Fixed terminals such as desktop computers. Mobile terminals such as mobile phones, PADs and mobile robots. In addition, the technical features involved in the different embodiments of the present invention described later can be combined with each other as long as they do not conflict with each other.

[0071] Below, the present invention proposes some preferred embodiments to teach those skilled in the art to implement.

[0072] Embodiment 1

[0073] This embodiment provides a batch-stream fusion information processing method, such as Figure 1 As shown, the steps include:

[0074] S100, obtaining an index based on an input query statement;

[0075] S200, extracting pre-calculated index data segments based on the index as query results;

[0076] S300: extract newly calculated index data and obtain index segments based on the input query statement to update the query result.

[0077] In a further embodiment, the acquiring of the index based on the input query statement specifically includes:

[0078] S110, obtaining an input query statement;

[0079] S120, searching for keywords in the query statement;

[0080] S130, according to the keyword association dimension obtained by retrieval, measure;

[0081] S140. Create an index based on the associated dimensions and metrics.

[0082] In a further embodiment, the keyword association index specifically includes:

[0083] S131. Set keywords based on dimension and measurement types:

[0084] S132, establishing a mapping relationship between keywords and the dimensions and measurement types they represent;

[0085] S133. When a keyword is retrieved, the dimension and measurement type represented by the keyword are obtained.

[0086] In a further embodiment, extracting the pre-calculated index data segment based on the index as the query result specifically includes:

[0087] S210, obtaining statistical information of the index;

[0088] S220, searching the memory for a pre-created index having the statistical information;

[0089] S230, locating the pre-calculated index data segment in the storage medium based on the pre-created index;

[0090] S240: Use the located pre-calculated index data segment as the query result.

[0091] In a further embodiment, the obtaining of statistical information of the index specifically includes:

[0092] S211, receiving the created index;

[0093] S212, performing statistics on the index;

[0094] S213. Obtain relevant statistical information based on the statistical results, including but not limited to the number of rows, size, and cardinality.

[0095] In a further embodiment, the pre-calculated index data segment specifically includes:

[0096] S250, defining a pre-created index based on a pre-defined unified model;

[0097] S260, performing pre-calculation based on the pre-created index to obtain an index data segment;

[0098] S270: Store the statistical information of the pre-created index in the memory, and store the pre-calculated index data segments in the storage medium.

[0099] In a further embodiment, the pre-created index is defined based on a pre-defined unified model, such as Figure 2 , Figure 3 As shown, specifically including:

[0100] S251, obtaining a pre-created stream data source mapping table and a batch data source mapping table, wherein the stream data source mapping table is a fact table, and the batch data source mapping table is a shadow table bound thereto;

[0101] S252, obtaining dimensions and measures for connecting the pre-created fact table and the shadow table;

[0102] S253. Based on the acquired dimensions and metrics, determine the connection relationship between the fact table and the shadow table and the dimension table;

[0103] S254: Pre-create an index based on the connection relationship definition.

[0104] In a further embodiment, the index data segment is obtained based on the pre-created index calculation, such as Figure 4 As shown, specifically including:

[0105] S261, obtaining a pre-created index, wherein the pre-created index is divided into a batch index, a stream index, and a fusion index;

[0106] S262: Based on the pre-created index, pre-calculate in the data source through a computing engine, wherein the computing engine includes a batch processing computing engine and a stream processing computing engine, and the data source includes a batch data source and a stream data source;

[0107] S263. Obtain pre-calculated index data segments based on the pre-calculation, wherein the pre-calculated index data segments are divided into batch index data segments and stream index data segments.

[0108] In a further embodiment, storing the statistical information of the pre-created index in the memory specifically includes:

[0109] Extract statistics for pre-created indexes, where statistics include but are not limited to number of rows, size, and cardinality.

[0110] In a further embodiment, extracting the newly calculated index data segment to update the query result specifically includes:

[0111] S310, performing new calculation in a data source based on an index, wherein the index is divided into a batch index, a stream index, and a fusion index, and the data source includes a batch data source and a stream data source;

[0112] S320, obtaining a newly calculated index data segment through a new calculation by a calculation engine, wherein the calculation engine includes a batch processing calculation engine and a stream processing calculation engine, and the newly calculated index data segment is divided into a batch index data segment and a stream index data segment;

[0113] S330, storing the newly calculated index data segment in a storage medium;

[0114] S340: extract the newly calculated index data segment in the storage medium to update the query result.

[0115] From the above description, it can be seen that the present invention achieves the following technical effects:

[0116] 1. By defining a unified model, the technical effect of analyzing historical data and real-time data simultaneously through a single SQL query statement is achieved;

[0117] 2. Through pre-calculation and real-time merging technology, the technical effect of effectively reducing query response time is achieved;

[0118] 3. By storing the calculation results in the storage medium for repeated use, the technical effect of ensuring the stability of the system is achieved;

[0119] 4. By continuously storing the newly calculated results in the storage medium, the technical effect of updating the query results is achieved.

[0120] Embodiment 2

[0121] This embodiment provides a device for batch-stream fusion information processing, which is characterized by comprising:

[0122] The client is used to obtain the index based on the input query statement;

[0123] A query engine, used to extract pre-calculated index data segments as query results based on the index;

[0124] The update module is used to extract the newly calculated index data segment to update the query results.

[0125] In a further embodiment, the query engine:

[0126] Get index statistics;

[0127] Retrieve the pre-created index with the statistics in memory;

[0128] Locating pre-computed index data segments in a storage medium based on the pre-created index;

[0129] The located pre-computed index data segment is used as the query result.

[0130] In a further embodiment, the query engine comprises:

[0131] A pre-created index definition module, used to define a pre-created index based on a pre-defined unified model;

[0132] A pre-calculation module, used to pre-calculate index data segments based on pre-created indexes;

[0133] The information storage module stores the statistical information of the pre-created index in the memory and stores the pre-calculated index data segments in the storage medium.

[0134] In a further embodiment, the pre-created index definition module:

[0135] Obtain a pre-created stream data source mapping table and a batch data source mapping table, wherein the stream data source mapping table is a fact table and the batch data source mapping table is a shadow table bound thereto;

[0136] Get the dimensions and measures of the pre-created fact table and shadow table connection;

[0137] Based on the acquired dimensions and measures, determine the connection relationship between the fact table and shadow table and the dimension table;

[0138] Pre-create indexes based on join relationship definitions.

[0139] In a further embodiment, the pre-calculation module:

[0140] Get pre-created indexes, where pre-created indexes are divided into batch indexes, stream indexes, and fusion indexes;

[0141] Based on the pre-created index, pre-calculation is performed in the data source through a computing engine, wherein the computing engine includes a batch processing computing engine and a stream processing computing engine, and the data source includes a batch data source and a stream data source;

[0142] Precomputed index data segments are obtained based on the precomputed data, wherein the precomputed index data segments are divided into batch index data segments and stream index data segments.

[0143] In a further embodiment, the information storage module:

[0144] Extract statistics for pre-created indexes, where the statistics include number of rows, size, and cardinality.

[0145] In a further embodiment, the update module:

[0146] Perform new calculations in data sources based on indexes, where indexes are divided into batch indexes, stream indexes, and fusion indexes, and data sources include batch data sources and stream data sources;

[0147] The newly calculated index data segment is obtained by a new calculation through a calculation engine, wherein the calculation engine includes a batch processing calculation engine and a stream processing calculation engine, and the newly calculated index data segment is divided into a batch index data segment and a stream index data segment;

[0148] storing the newly calculated index data segment in a storage medium;

[0149] Extract the newly calculated index data segment in the storage medium to update the query result.

[0150] Embodiment 3

[0151] In this embodiment, the dimensions of a pre-created index are insurance salesperson (seller_id) and date (date), and the metric is the sum of the policy amounts sum(amount). Since there may be a large number of salespersons, the aggregation degree of this index may not be very high. The data content corresponding to this pre-created index may be as shown in Table 1 below, which is the result of summarizing the transaction amount of each salesperson's sales record on each day:

[0152] Seller_Id Date Sum(amount) 10001 2020-05-01 00:00:00 100 10001 2020-05-01 00:10:00 200 10002 2020-05-01 00:20:00 150 10003 2020-05-01 00:30:00 80 10003 2020-05-01 00:40:00 30

[0153] Table 1

[0154] Assuming there are 100,000 salespeople in total, the remaining 100,000 rows of pre-calculated results are omitted here.

[0155] The data content of this table is the batch index data segment. The system will calculate the pre-created index and save the calculated pre-calculation results, that is, save the table data in real time.

[0156] Based on the information processing method of batch-stream fusion provided in this embodiment, the steps are as follows:

[0157] S100, obtaining an index based on an input query statement;

[0158] The user provides a query:

[0159] SQL 1 Analyze the total transaction amount of salesperson number 10003 from May 1, 2020 to the present: select sum(amount) from transactions where date>timestamp('2020-05-01 00:00:00')and seller_id='10003'

[0160] When consuming the continuously generated streaming real-time data, the system will create an index close to the above pre-created index for this query.

[0161] In a further embodiment, the acquiring of the index based on the input query statement specifically includes:

[0162] S110, obtaining an input query statement;

[0163] S120, searching for keywords in the query statement;

[0164] S130, according to the keyword association dimension obtained by retrieval, measure;

[0165] S140. Create an index based on the associated dimensions and metrics.

[0166] In a further embodiment, the keyword association index specifically includes:

[0167] S131. Set keywords based on dimension and measurement types:

[0168] S132, establishing a mapping relationship between keywords and the dimensions and measurement types they represent;

[0169] S133. When a keyword is retrieved, the dimension and measurement type represented by the keyword are obtained.

[0170] The keywords retrieved in the query statement SQL 1 are: salesperson (seller_id), May 1, 2020 (date), and total transaction amount sum (amount). The associated dimensions of salesperson and May 1, 2020 are insurance salesperson (seller_id) and date (date), respectively, and the associated measure of total transaction amount sum (amount) is the total policy amount sum (amount).

[0171] Based on the information processing method of batch-stream fusion provided in this embodiment, the steps are as follows:

[0172] S200, extracting pre-calculated index data segments based on the index as query results;

[0173] When querying the statements in SQL1, the results previously stored in the storage medium are directly used for the answer, that is, the data in Table 1, which is also batch data, ensuring the performance, efficiency and stability of the system.

[0174] In a further embodiment, extracting the pre-calculated index data segment based on the index as the query result specifically includes:

[0175] S210, obtaining statistical information of the index;

[0176] S220, searching the memory for a pre-created index having the statistical information;

[0177] S230, locating the pre-calculated index data segment in the storage medium based on the pre-created index;

[0178] S240: Use the located pre-calculated index data segment as the query result.

[0179] There are three columns in the index, and the cardinality of each column is Seller_Id, Date, and Sum (amount). The pre-created index is detected in the statistical information of the pre-created index stored in the memory, and the pre-calculated index data segment, that is, Table 1, is located in the storage medium, and Table 1 is used as the query result answer.

[0180] In a further embodiment, the obtaining of statistical information of the index specifically includes:

[0181] S211, receiving the created index;

[0182] S212, performing statistics on the index;

[0183] S213. Obtain relevant statistical information based on the statistical results, including but not limited to the number of rows, size, and cardinality.

[0184] The statistical information of the index obtained by counting is three columns, and the cardinality of each column is Seller_Id, Date, and Sum(amount).

[0185] In a further embodiment, the pre-calculated index data segment specifically includes:

[0186] S250, defining a pre-created index based on a pre-defined unified model;

[0187] S260, performing pre-calculation based on the pre-created index to obtain an index data segment;

[0188] S270: Store the statistical information of the pre-created index in the memory, and store the pre-calculated index data segments in the storage medium.

[0189] In a further embodiment, the pre-created index is defined based on a pre-defined unified model, such as Figure 2 , Figure 3 As shown, specifically including:

[0190] S251, obtaining a pre-created stream data source mapping table and a batch data source mapping table, wherein the stream data source mapping table is a fact table, and the batch data source mapping table is a shadow table bound thereto;

[0191] S252, obtaining dimensions and measures for connecting the pre-created fact table and the shadow table;

[0192] S253. Based on the acquired dimensions and metrics, determine the connection relationship between the fact table and the shadow table and the dimension table;

[0193] S254: Pre-create an index based on the connection relationship definition.

[0194] The dimensions are insurance salesperson (seller_id) and date (date), and the measure is the total amount of insurance policies, sum(amount).

[0195] In a further embodiment, the index data segment is obtained based on the pre-created index calculation, such as Figure 4 As shown, specifically including:

[0196] S261, obtaining a pre-created index, wherein the pre-created index is divided into a batch index, a stream index, and a fusion index;

[0197] S262: Based on the pre-created index, pre-calculate in the data source through a computing engine, wherein the computing engine includes a batch processing computing engine and a stream processing computing engine, and the data source includes a batch data source and a stream data source;

[0198] S263. Obtain pre-calculated index data segments based on the pre-calculation, wherein the pre-calculated index data segments are divided into batch index data segments and stream index data segments.

[0199] The pre-calculated batch index data segment is shown in Table 1.

[0200] In a further embodiment, storing the statistical information of the pre-created index in the memory specifically includes:

[0201] Extract statistics for pre-created indexes, where statistics include but are not limited to number of rows, size, and cardinality.

[0202] Based on the information processing method of batch-stream fusion provided in this embodiment, the steps are as follows:

[0203] S300: extract newly calculated index data and obtain index segments based on the input query statement to update the query result.

[0204] At the same time, as the new calculation results are continuously stored in the storage medium, the results in the query will also be updated accordingly.

[0205] In a further embodiment, extracting the newly calculated index data segment to update the query result specifically includes:

[0206] S310, performing new calculation in a data source based on an index, wherein the index is divided into a batch index, a stream index, and a fusion index, and the data source includes a batch data source and a stream data source;

[0207] S320, obtaining a newly calculated index data segment through a new calculation by a calculation engine, wherein the calculation engine includes a batch processing calculation engine and a stream processing calculation engine, and the newly calculated index data segment is divided into a batch index data segment and a stream index data segment;

[0208] S330, storing the newly calculated index data segment in a storage medium;

[0209] S340: extract the newly calculated index data segment in the storage medium to update the query result.

[0210] Embodiment 4

[0211] An embodiment of the present invention further includes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed in the processor, the computer program is used to implement the above-mentioned information processing method for batch-stream fusion, the method including:

[0212] Get the index based on the input query statement;

[0213] Extract pre-calculated index data segments based on the index as query results;

[0214] Extract the newly calculated index data segment to update the query results.

[0215] Embodiment 5

[0216] In this embodiment, the present invention further provides a readable storage medium, wherein a computer program is stored in the readable storage medium. When the computer program is executed by a processor, the information processing method for batch-stream fusion is implemented. The method includes:

[0217] Get the index based on the input query statement;

[0218] Extract pre-calculated index data segments based on the index as query results;

[0219] Extract the newly calculated index data segment to update the query results.

[0220] Among them, the readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general or special-purpose computer. For example, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the readable storage medium can also exist in a communication device as discrete components. The readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0221] The present invention also provides a program product, which includes an execution instruction, which is stored in a readable storage medium. At least one processor of a device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the device implements the methods provided in the above various embodiments.

[0222] In the above-mentioned terminal or server embodiment, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the present invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0223] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0224] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0225] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A batch-stream fusion information processing method, characterized in that: include: Get the index based on the input query statement; Extract pre-calculated index data segments based on the index as query results; Specifically include: obtaining statistical information of the index; retrieving the pre-created index with the statistical information in the memory; locating the pre-calculated index data segment in the storage medium based on the pre-created index; using the located pre-calculated index data segment as the query result; the pre-calculated index data segment specifically includes: defining the pre-created index based on a pre-defined unified model; specifically include: obtaining a pre-created stream data source mapping table and a batch data source mapping table, wherein the stream data source mapping table is a fact table and the batch data source mapping table is a shadow table bound thereto; obtaining the dimensions and metrics of the pre-created fact table and the shadow table; determining the connection relationship between the fact table and the shadow table and the dimension table based on the obtained dimensions and metrics; defining the pre-created index based on the connection relationship; Pre-calculating based on the pre-created index to obtain an index data segment; obtaining the pre-created index, wherein the pre-created index is divided into a batch index, a stream index, and a fusion index; based on the pre-created index, pre-calculating in the data source through a computing engine, wherein the computing engine includes a batch processing computing engine and a stream processing computing engine, and the data source includes a batch data source and a stream data source; obtaining a pre-calculated index data segment based on the pre-calculation, wherein the pre-calculated index data segment is divided into a batch index data segment and a stream index data segment; Storing statistical information of pre-created indexes in memory and storing pre-computed index data segments in storage media; Extract the newly calculated index data segment to update the query results.

2. The method according to claim 1, characterized in that The storing of the statistical information of the pre-created index in the memory specifically includes: Extract statistics for pre-created indexes, where the statistics include number of rows, size, and cardinality.

3. The method according to claim 1, characterized in that The extracting of the newly calculated index data segment to update the query result specifically includes: Perform new calculations in data sources based on indexes, where indexes are divided into batch indexes, stream indexes, and fusion indexes, and data sources include batch data sources and stream data sources; The newly calculated index data segment is obtained by a new calculation through a calculation engine, wherein the calculation engine includes a batch processing calculation engine and a stream processing calculation engine, and the newly calculated index data segment is divided into a batch index data segment and a stream index data segment; storing the newly calculated index data segment in a storage medium; Extract the newly calculated index data segment in the storage medium to update the query result.

4. A batch-stream fusion information processing device, characterized in that: include: The client is used to obtain the index based on the input query statement; A query engine, used to extract pre-calculated index data segments as query results based on the index; Specifically include: obtaining the statistical information of the index; retrieving the pre-created index with the statistical information in the memory; locating the pre-calculated index data segment in the storage medium based on the pre-created index; using the located pre-calculated index data segment as the query result; the pre-calculated index data segment specifically includes: defining the pre-created index based on a pre-defined unified model; specifically include: obtaining the pre-created stream data source mapping table and batch data source mapping table, wherein the stream data source mapping table is the fact table and the batch data source mapping table is the shadow table bound thereto; obtaining the dimensions and metrics connecting the pre-created fact table and the shadow table; determining the relationship between the fact table and the shadow table and the dimensions based on the obtained dimensions and metrics. The method comprises the steps of: defining a connection relationship between the degree table and the degree table; defining a pre-created index based on the connection relationship; and performing pre-calculation based on the pre-created index to obtain an index data segment; obtaining the pre-created index, wherein the pre-created index is divided into a batch index, a stream index and a fusion index; performing pre-calculation in a data source through a computing engine based on the pre-created index, wherein the computing engine includes a batch processing computing engine and a stream processing computing engine, and the data source includes a batch data source and a stream data source; obtaining a pre-calculated index data segment based on the pre-calculation, wherein the pre-calculated index data segment is divided into a batch index data segment and a stream index data segment; storing the statistical information of the pre-created index in a memory, and storing the pre-calculated index data segment in a storage medium; The update module is used to extract the newly calculated index data segment to update the query results.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: The computer program can be executed in the processor to implement the method according to any one of claims 1 to 3.

6. A storage medium storing a computer program, characterized in that: The computer program is executed in a processor to implement any method according to claims 1 to 3.

Images