A data report generation method and device, computer equipment and a storage medium

Abstract

The present application relates to the technical field of big data, and provides a data report generation method and device, computer equipment and a storage medium. According to the received report generation request information, the data source of the data to be queried is obtained, it is detected whether the database corresponding to the data source is a preset source database, when it is detected that the database corresponding to the data source is the preset source database, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage, according to the data amount of the data to be queried, the query mode in the preset engine is matched as a target query mode, the data to be queried is exported from the service node corresponding to the preset engine through the target query mode, the report corresponding to the data to be queried is generated, and by synchronizing the data to be queried to the preset engine, the query of a large amount of data is improved, thereby improving the generation efficiency of the report corresponding to the large amount of data.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a data report generation method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In enterprises, as the volume of business data increases, corresponding reports are typically generated to present the data. A well-designed business report can effectively reflect the performance and indicator trends over recent months, playing a crucial role in subsequent strategic development decisions. Currently, the business data used to generate reports is generally stored in relational databases. Reports are generated by searching and querying data within these databases. However, as the volume of business data increases, the complexity of the data structure grows. When performing large-scale data queries, the complexity leads to poor query performance, reducing report generation efficiency. The lengthy report generation process consumes significant resources. Therefore, improving report generation efficiency has become a pressing issue. Summary of the Invention

[0003] Therefore, it is necessary to provide a data report generation method, apparatus, computer equipment, and storage medium to address the aforementioned technical problems and solve the problem of low report generation efficiency.

[0004] Firstly, a method for generating data reports is provided, the method comprising:

[0005] Based on the request information generated from the received report, obtain the data source of the data to be queried in the request information, and detect whether the database corresponding to the data source is a preset source database;

[0006] When the database corresponding to the data source is detected to be the preset source database, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage;

[0007] Based on the amount of data to be queried in the request information, the query method in the preset engine is matched as the target query method. The data to be queried is exported from the service node corresponding to the preset engine through the target query method, and a report corresponding to the data to be queried is generated.

[0008] Secondly, a data report generation apparatus is provided, the apparatus comprising:

[0009] The detection module is used to obtain the data source of the data to be queried in the request information generated by the received report, and to detect whether the database corresponding to the data source is a preset source database.

[0010] The synchronization module is used to synchronize the data to be queried from the preset source database to the service node corresponding to the preset engine for storage when it is detected that the database corresponding to the data source is the preset source database;

[0011] The report generation module is used to match the query method in the preset engine as the target query method according to the amount of data to be queried in the request information, and export the data to be queried from the service node corresponding to the preset engine through the target query method to generate a report corresponding to the data to be queried.

[0012] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data report generation method as described in the first aspect.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data report generation method as described in the first aspect.

[0014] The advantages of this invention compared to the prior art are:

[0015] Based on the received report generation request information, the data source of the data to be queried in the request information is obtained. It is checked whether the database corresponding to the data source is a preset source database. If the database corresponding to the data source is detected as a preset source database, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage. According to the amount of data to be queried in the request information, the query method in the preset engine is matched as the target query method. Through the target query method, the data to be queried is exported from the service node corresponding to the preset engine, and the report corresponding to the data to be queried is generated. By synchronizing the query data to the preset engine, the query of large batches of data is improved, thereby improving the generation efficiency of reports corresponding to large batches of data. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1This is a schematic diagram of an application environment for a data report generation method provided in an embodiment of the present invention;

[0018] Figure 2 This is a flowchart illustrating a data report generation method according to an embodiment of the present invention;

[0019] Figure 3 This is a flowchart illustrating a data report generation method according to an embodiment of the present invention;

[0020] Figure 4 This is a flowchart illustrating a data report generation method according to an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of the structure of a data report generation device provided in an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0027] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0029] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0030] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0031] An embodiment of the present invention provides a data report generation method, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0032] See Figure 2 This is a flowchart illustrating a data report generation method according to an embodiment of the present invention. The data report generation method described above can be applied to... Figure 1 The server in the example connects to the corresponding client. For example... Figure 2 As shown, this data report generation method may include the following steps.

[0033] S201: Based on the request information generated from the received report, obtain the data source of the data to be queried in the request information, and check whether the database corresponding to the data source is the preset source database.

[0034] In step S201, based on the request information generated from the received report, the data source of the data to be queried in the request information is obtained. The request information includes the data source of the data to be queried, and it is detected whether the database corresponding to the data source is a preset source database.

[0035] In this embodiment, the received report generation request information includes the data source of the queried data, the database name, etc. The report generation request information can be user-triggered instructions for generating reports, or instructions automatically generated by the client according to a preset time or when preset conditions are met. The storage address of the queried data is obtained based on the data source, thereby detecting whether the database corresponding to the queried data is a preset source database. When detecting the data source, methods such as detecting the corresponding API interface, SDK interface, shell script, Python script, etc., can be used to determine whether the database corresponding to the queried data is a preset source database.

[0036] The default source database is a data warehouse tool based on Hadoop. It can map structured data files to a database table and provide simple SQL query functionality. It can convert SQL statements into MapReduce tasks for execution, quickly implementing simple MapReduce statistics through SQL-like statements without the need to develop dedicated MapReduce applications, making it ideal for statistical analysis in data warehouses. The default source database resides on a server, meaning a dedicated storage space is allocated on the server for storing the data.

[0037] S202: When the database corresponding to the data source is detected to be a preset source database, the data to be queried will be synchronized from the preset source database to the service node corresponding to the preset engine for storage.

[0038] In step S202, when the database corresponding to the data source is detected to be a preset source database, the query data is synchronized from the preset source database to the service node corresponding to the preset engine for storage. The preset engine is a distributed cluster search engine that includes multiple service nodes.

[0039] In this embodiment, when the database corresponding to the data source is detected to be a preset source database, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage. The preset engine is a distributed cluster including multiple service nodes. Unlike other databases, the preset engine can achieve fast information search. The preset engine can store, search, and analyze large amounts of data in a short time, and therefore is often used in complex search scenarios. Synchronizing the preset source database to the service node corresponding to the preset engine enables data...

[0040] It's important to note that the default engine cluster mode uses a master-slave model, meaning the cluster doesn't rely on external components (such as Zookeeper or HDFS). The master-slave mode simplifies system design. The master node maintains cluster information and uses a heartbeat mechanism to confirm the continued presence of cluster members. If the master node fails, an election is required. Clusters typically use an odd number of nodes to prevent split-brain scenarios during master election. Therefore, the default engine cluster has at least a master node and data nodes. If cluster node resources are sufficient, redundant nodes can be configured as coordinating nodes. To ensure availability in a distributed environment and to expand storage space, the default engine partitions the data. Each shard stores multiple replicas based on the initial index settings. The total number of shards equals the product of the number of nodes and the number of shards, and these shards are evenly distributed across the cluster nodes.

[0041] Optionally, when the database corresponding to the data source is detected to be a preset source database, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage, including:

[0042] Create a mapping table between the preset source database and the service nodes corresponding to the preset engine;

[0043] Based on the mapping table, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage.

[0044] In this embodiment, a mapping table is created between the preset source database and the service nodes corresponding to the preset engine. The mapping table shows the service node address, port name, and corresponding index information and document information of the preset engine cluster, thereby establishing a data transmission channel between the preset source database and the preset engine cluster.

[0045] It's important to note that the mapping table can also specify that primary key fields in the default source database are mapped to ID identifiers in the default engine cluster. The primary key field uniquely identifies each field in the mapping table. The ID identifier in the default engine cluster is a string. When combined with _index (where the document is stored) and _type (the type of object the document represents), it represents a specific document in the default engine cluster. In other words, documents in the default engine cluster can be uniquely identified by their ID identifiers; different documents have unique ID identifiers. Once the mapping table is created, a data synchronization channel with the default engine cluster can be established.

[0046] S203: Based on the amount of data to be queried in the request information, match the query method in the preset engine as the target query method, and export the data to be queried from the corresponding service node in the preset engine through the target query method to generate the report corresponding to the data to be queried.

[0047] In step S203, when the amount of data to be queried in the request information is different, the query method in the preset engine is different. The query is performed based on different query methods so that the corresponding data to be queried can be obtained more quickly and a data report corresponding to the data to be queried can be generated.

[0048] In this embodiment, after determining the target query method, the query request for the data to be queried is written in JSON format, and then the relevant query is performed. The preset engine provides a RESTful API to access and use the JSON format, supporting match queries, multi_match queries, term queries, range queries, and bool queries. The preset engine does not need to sort the data by column and row; instead, it stores the entire document. A document is a data entry or a row in a traditional database. It stores these documents and searches their content by building an index. It creates an inverted index, which is a sorted dictionary of terms mapped to the document. Scalability is achieved because documents can be distributed across different blocks. The preset engine organizes data for each index within a master node containing multiple data nodes. These data nodes facilitate scaling across multiple hardware devices, and the preset engine also provides resilience due to replica nodes.

[0049] It should be noted that when querying through the preset engine, the query includes DSL (Domain Specific Language) query statements, which can be portions of the query statements in the logical query information and can be executed independently within the Elasticsearch (Elasticsearch, the search and analytics engine) cluster. The query result merging strategy can be obtained from the logical query information. Query result merging strategies include intersection, union, and difference of query results, corresponding to keywords such as MUST, SHOULD, and MUST_NOT in the logical query information.

[0050] When querying based on the desired data, upon receiving a JSON-formatted query statement, a parser converts the conditional JSON strings within the JSON statement into logical query information for a DSL query statement containing an Elasticsearch query object (QueryBuilder). During query execution, the DSL query statement is further converted into physical query information for multiple DSL query statements. Query builder objects include Boolean query builder objects, value query builder objects, range query builder objects, and IN query builder objects, among others.

[0051] The parser performs MAP (map) processing on the JSON-formatted query statement to obtain the element-value pairs and their logical relationships. The first element is used as the ROOT element, and an Elasticsearch query builder object is generated based on the element-value pairs and logical relationships. It checks for logical index names; if present, a flexible alias Boolean query builder object with logical index information is constructed based on the logical relationships. Logical relationships include one or more of AND, OR, and non-logical relationships.

[0052] It's important to note that when querying indexes, you can configure the index manager. The index manager allows you to configure clustering strategies, partitioning strategies, JOIN field names, and logical index aliases. For example, a clustering strategy might be based on date, with indexes for odd-numbered days in cluster A and even-numbered days in cluster B. A partitioning strategy might involve creating an index weekly and only querying indexes from the past eight weeks. JOIN field names can be configured, for example, by joining the `pin` field of index A with the `user_pin` field of index B. The `fields` attribute is an array; typically, the first field is used as the join attribute, and the remaining fields are used as additional query attributes.

[0053] Optionally, based on the amount of data to be queried in the request information, a query method in the preset engine is matched as the target query method. Using the target query method, the data to be queried is exported from the corresponding service node in the preset engine, generating a report corresponding to the queried data, including:

[0054] When the amount of data to be queried is less than the preset threshold, the pagination query method in the preset engine is used to export the data to be queried and generate the corresponding report.

[0055] When the amount of data to be queried exceeds the preset threshold, the cursor query method in the preset engine is used to export the data to be queried and generate the corresponding report.

[0056] In this embodiment, the query methods in the preset engine include pagination query and cursor query. When the amount of data to be queried is less than a preset threshold, the pagination query method in the preset engine is used to export the data to be queried and generate a report corresponding to the data to be queried. When the amount of data to be queried is greater than the preset threshold, the cursor query method in the preset engine is used to export the data to be queried and generate a report corresponding to the data to be queried.

[0057] Optionally, when the amount of data to be queried is less than a preset threshold, the pagination query method in the preset engine is used to export the data to be queried and generate a report corresponding to the data to be queried, including:

[0058] Based on the received pagination query command for the data to be queried, obtain the starting row number of the pagination query and the amount of data in each page of the pagination query, and obtain the query result of the pagination query;

[0059] Based on the query results of the paginated query, the data to be queried is exported sequentially to generate a report corresponding to the data to be queried.

[0060] In this embodiment, based on the received pagination query command for the data to be queried, the starting row number of the pagination query and the amount of data in each page of the pagination query are obtained, and the query results of the pagination query are obtained. The starting row number of the pagination query and the amount of data in each page of the pagination query are two parameters required for the pagination query. The starting row number indicates from which row the data to be queried begins, and the amount of data in each page of the pagination query indicates the page size of the pagination query. The initial values ​​of these two parameters are generally passed from the front end.

[0061] It should be noted that the default engine can record the starting row number of the current pagination query and the amount of data in each page of the pagination query. When the current pagination query is completed, the starting row number of the current pagination query and the amount of data in each page of the pagination query are determined and recorded. Before performing the next pagination query, the starting row number of the current pagination query and the amount of data in each page of the pagination query are obtained from the recorded data. However, those skilled in the art should understand that the methods for obtaining the starting row number of the current pagination query and the amount of data in each page of the pagination query are not limited to this.

[0062] It should be noted that the starting row number is usually 0 or 1. For example, in MySQL databases, the row number index starts counting from 0, so the starting row number is 0.

[0063] Optionally, when the amount of data to be queried exceeds a preset threshold, the cursor query method in the preset engine is used to export the data to be queried and generate a report corresponding to the data to be queried, including:

[0064] Based on the preset export threshold, the cursor query method in the preset engine is used to write the data to be queried into the preset file in the amount of data of the export threshold each time, and the written file is obtained.

[0065] The file is uploaded to the file container platform, generating a report link corresponding to the data to be queried. Based on the download results of the report link, the report corresponding to the data to be queried is generated.

[0066] In this embodiment, based on a preset export threshold, the cursor query method in the preset engine is used to write the data to be queried to a preset file in increments of the export threshold amount. When querying via cursor, FETCH reads data from the preset engine, either one or N records at a time. When reading one record, the cursor offset increases by 1; when reading N records, the cursor offset increases by N. For example, the FETCH statement reads data from the preset engine, and for each record read, the cursor offset increases by 1 until the export threshold is reached. If the cursor has a flag of 1 indicating that not all the data to be queried has been read, the suspended query environment is found, the query statement continues execution, the next batch of data to be queried is obtained, and the FETCH statement is retried. Since the cursor registers the cache address of nsql_ctx, i.e., the entry point of the SQL query statement in the virtual machine, it can be restarted by FETCH for fast querying.

[0067] It should be noted that if the offset equals the export threshold after reading the query results of the data to be queried, the reading of the data to be queried will end. The export threshold can be the total number of data to be queried that needs to be read, and the export threshold can be set by the user.

[0068] In this embodiment, each time the cursor reads the data to be queried according to the index, the cursor offset and the export threshold are compared. If the cursor offset is equal to the export threshold of the data to be queried, the cursor can read a sufficient amount of the data to be queried, and the reading of the data to be queried can end. If the cursor offset is less than the export threshold of the data to be queried, the cursor has not yet read a sufficient amount of the data to be queried, and the flag is 1, then the cursor can continue to read the data to be queried according to the registered index. If the cursor offset is less than the export threshold of the data to be queried, the cursor has not yet read a sufficient amount of the data to be queried, and the flag is 0, then the cursor cannot obtain any new data to be queried, and the cursor ends the reading of the data to be queried after reading all the data to be queried.

[0069] It should be noted that after the query results for the desired data are retrieved using a cursor query, they are written to a preset file, resulting in a write file. This write file is then uploaded to the file container platform, generating a report link corresponding to the queried data. Based on the download results of the report link, the report corresponding to the queried data is generated. The file container refers to a data storage container that synchronizes data with the service nodes of the preset engine over the network, such as a local sync disk or a cloud server.

[0070] In another embodiment, when real-time data needs to be exported, a corresponding acquisition frequency can be preset. For example, if the acquisition is set to occur every 10 minutes, and the acquisition time is data generated between 11:10 and 11:20, then if the next acquisition time is 11:30, the data generated between 11:20 and 11:30 needs to be acquired. During acquisition, the data generated between 11:20 and 11:30 is first sorted in ascending order, that is, sorted according to the time the data was generated. The corresponding data is then acquired and synchronized to the service node of the preset engine.

[0071] It's important to note that when retrieving data in ascending order, if data retrieval fails, the failed data can be marked. The next time data is retrieved, it can be retrieved again directly from the time of the previous failure, preventing duplicate data retrieval. For example, if retrieving data at 10:12 fails, this failure time is recorded as 10:12. The next task will then retrieve data from 10:12 until the next task's scheduled time.

[0072] Based on the received report generation request information, the data source of the data to be queried in the request information is obtained. It is checked whether the database corresponding to the data source is a preset source database. If the database corresponding to the data source is detected as a preset source database, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage. According to the amount of data to be queried in the request information, the query method in the preset engine is matched as the target query method. Through the target query method, the data to be queried is exported from the service node corresponding to the preset engine, and the report corresponding to the data to be queried is generated. By synchronizing the data to be queried to the preset engine, the query of large batches of data is improved, thereby improving the generation efficiency of reports corresponding to large batches of data.

[0073] See Figure 3 This is a flowchart illustrating a data report generation method according to an embodiment of the present invention, as shown below. Figure 3 The data report generation method may include the following steps:

[0074] S301: Based on the request information generated from the received report, obtain the data source of the data to be queried in the request information, and check whether the database corresponding to the data source is the preset source database;

[0075] The content of step S301 is the same as that of step S201, and can be referred to the description of step S201 above, so it will not be repeated here.

[0076] S302: When the database corresponding to the detected data source is not the database stored in the preset source database, synchronize the data to be queried to the local database;

[0077] S303: Through the scheduling platform, the database corresponding to the data to be queried is synchronized from the local database to the preset source database for storage;

[0078] In this embodiment, when the database corresponding to the detected data source is not a preset source database, the data to be queried is synchronized to the local database. The scheduling platform then synchronizes the database corresponding to the data to be queried from the local database to the preset source database for storage. During synchronization to the local database, the data to be queried is tagged with a synchronization label. When saving the synchronized data, it is necessary to consider whether the label of the data to be queried is a failure label. Specifically, first, the synchronized data is saved in the local database. Then, it is determined whether the label of the saved synchronized data is a failure label. If not, the saving process ends; if so, after saving, a message can be sent to the user who wants to query the data.

[0079] For example, when the data to be queried needs to be synchronized to the local database, the corresponding data to be queried is extracted from the server and synchronized to the local database. If there is an abnormal situation (such as abnormal network communication) and the local database is not fully synchronized, the data to be queried can be marked with a synchronization failure label. When the data to be queried is fully synchronized to the local database, the failure label can be changed to a success label.

[0080] Data in the local database is synchronized to a preset source database. During synchronization, the mapping relationship between the preset source database and the data source is one-to-one; each data source has exactly one corresponding preset source database, and each preset source database receives data synchronized from only one data source. In this embodiment, the number of preset source databases in the server is greater than the number of data sources, which facilitates the direct establishment of a mapping relationship between the newly acquired data source and the added preset source database when the server acquires other data sources. The target preset source database can be named using the ID of the hardware device of the data source.

[0081] During data synchronization, data from the target preset source database is loaded into the preset engine service node. Multiple sub-databases within the target preset source database are associated with different data sources. During synchronization, data from the target HIVE database is synchronized to the preset engine service node. Since the data in each target preset source database exhibits certain patterns, the time spent searching for data during synchronization is reduced, resulting in very rapid synchronization to the preset engine service node.

[0082] It should be noted that, since the data to be queried may have different sources, different data formats, or different layout styles, it is preferable to preprocess the data after selecting the target data to ensure that it conforms to preset rules. Preprocessing the data may involve converting it into a specified data format. Specifically, preprocessing the target data may include the following steps: reading the target data line by line and determining whether each line conforms to preset rules; when lines that do not conform to preset rules are found, outputting data editing prompts; receiving editing feedback for lines that do not conform to preset rules, and repeating the determination steps until each line conforms to the preset rules. Preset rules can be formulated according to actual usage requirements, such as data format requirements and rules for each line of data (how many columns each line contains, whether each column is numeric or text, etc.).

[0083] It should be noted that the preset source database in this embodiment is a database built on a preset source database model. It uses concepts and methods such as set algebra to process data in the database. It is also a set of formally descriptive tables. The essence of these tables is to contain a special collection of data items. The data in these tables can be accessed or reassembled in many different ways without reorganizing the database tables. Each table contains one or more data types represented by rows or columns. The table structure of the database tables in this embodiment includes: table name, database name, fields included in the table, and descriptive information for each field. Sqoop is an open-source tool mainly used for transferring data between Hadoop and traditional databases (MySQL, PostgreSQL, etc.). It can import data from a relational database (e.g., MySQL, Oracle, Postgres, etc.) into Hadoop's HDFS, and vice versa. All preset source databases are standardized using the same Sqoop script. A pre-configured table structure is configured in each target preset source database. The server reads data from the target preset source database and, according to the preset script instructions, configures the data from each sub-database into the table structure. The data in the sub-databases is defined and tagged to facilitate subsequent querying and retrieval when synchronized to the preset engine service node. Standardization methods include storing data according to the same order rules or adding different tags based on different data types.

[0084] S304: Based on the amount of data to be queried in the request information, match the query method in the preset engine as the target query method, and export the data to be queried from the service node corresponding to the preset engine through the target query method to generate the report corresponding to the data to be queried.

[0085] The content of step S304 is the same as that of step S203, and can be referred to the description of step S203 above, so it will not be repeated here.

[0086] See Figure 4 This is a flowchart illustrating a data report generation method according to an embodiment of the present invention, as shown below. Figure 4 The data report generation method may include the following steps:

[0087] S401: Based on the request information generated from the received report, obtain the data source of the data to be queried in the request information, and check whether the database corresponding to the data source is the preset source database;

[0088] S402: When it is detected that the database corresponding to the data source is a preset source database, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage;

[0089] The contents of steps S401 to S402 are the same as those of steps S201 to S202, and can be referred to the description of steps S201 to S202, which will not be repeated here.

[0090] S403: Check whether the synchronized data to be queried is abnormal in the service node corresponding to the preset engine;

[0091] S404: When the data to be queried during synchronization is abnormal, the abnormal data in the data to be queried is deleted according to the preset instructions, and the normal data to be queried is stored in the service node corresponding to the preset engine.

[0092] In this embodiment, the service node corresponding to the preset engine checks whether the synchronized query data is abnormal. If it is abnormal, the query data in the preset engine can be compared with the query data in the preset source database. If the comparison results are different, it is considered that there is abnormal data in the preset engine, and the corresponding abnormal data is deleted. When deleting, it can be done according to the preset deletion command. For example, it can be set to delete all data in the data table corresponding to the abnormal data when abnormal data exists, or to delete the data related to the abnormal data when abnormal data exists, etc.

[0093] S405: Based on the amount of data to be queried in the request information, match the query method in the preset engine as the target query method, and export the data to be queried from the service node corresponding to the preset engine through the target query method to generate the report corresponding to the data to be queried.

[0094] The content of step S405 is the same as that of step S203, and can be referred to the description of step S203. It will not be repeated here.

[0095] Please see Figure 5 , Figure 5 This is a schematic diagram of a data report generation device provided in an embodiment of the present invention. In this embodiment, the computer device includes units used for execution... Figures 2 to 4 The steps in the corresponding embodiments. Please refer to the details. Figures 2 to 4 as well as Figures 2 to 4 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 5 The data report generation module 50 includes: detection module 51, synchronization module 52, and report generation module 53.

[0096] The detection module 51 is used to obtain the data source of the data to be queried in the request information generated from the received report, and to detect whether the database corresponding to the data source is a preset source database.

[0097] The synchronization module 52 is used to synchronize the data to be queried from the preset source database to the service node corresponding to the preset engine when the database corresponding to the data source is detected to be a preset source database.

[0098] The report generation module 53 is used to match the query method in the preset engine as the target query method according to the amount of data to be queried in the request information, and export the data to be queried from the service node corresponding to the preset engine through the target query method to generate the report corresponding to the data to be queried.

[0099] Optionally, the above synchronization module 52 includes:

[0100] The Create Mapping Table unit is used to create a mapping table between the preset source database and the service nodes corresponding to the preset engine.

[0101] The storage unit is used to synchronize the data to be queried from the preset source database to the service node corresponding to the preset engine, based on the mapping table.

[0102] Optionally, the report generation module 53 mentioned above includes:

[0103] The pagination query unit is used to export the data to be queried and generate a report corresponding to the data to be queried when the amount of data to be queried is less than a preset threshold, using the pagination query method in the preset engine.

[0104] The cursor query unit is used to export the data to be queried and generate a report corresponding to the data to be queried when the amount of data to be queried is greater than a preset threshold, using the cursor query method in the preset engine.

[0105] Optionally, the above pagination query unit includes:

[0106] The sub-unit is used to obtain the starting row number of the pagination query and the amount of data in each page of the pagination query based on the pagination query command received for the data to be queried, and to obtain the query result of the pagination query.

[0107] The export sub-cell is used to export the data to be queried sequentially based on the query results of the paginated query, and generate the corresponding report for the data to be queried.

[0108] Optionally, the above cursor query unit includes:

[0109] The write sub-unit is used to write the data to be queried into a preset file in the form of the export threshold data each time, using the cursor query method in the preset engine, according to the preset export threshold, to obtain the write file;

[0110] The upload sub-unit is used to upload the written file to the file container platform, generate the report link corresponding to the data to be queried, and generate the report corresponding to the data to be queried based on the download results of the report link.

[0111] Optionally, the above-mentioned generating apparatus further includes:

[0112] The local database synchronization module is used to synchronize the data to be queried to the local database when the database corresponding to the detected data source is not the preset source database.

[0113] The scheduling module is used to synchronize the database corresponding to the data to be queried from the local database to the preset source database for storage through the scheduling platform.

[0114] Optionally, the above-mentioned generating apparatus further includes:

[0115] The detection module is used to detect whether the synchronized data to be queried is abnormal in the service node corresponding to the preset engine.

[0116] The deletion module is used to delete abnormal data in the data to be queried according to preset instructions when the synchronized data is abnormal, and to store the normal data to be queried in the service node corresponding to the preset engine.

[0117] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0118] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 6 As shown, the computer device of this embodiment includes: at least one processor ( Figure 6 Only one is shown in the diagram), a memory, and a computer program stored in the memory and capable of running on at least one processor, wherein the processor executes the computer program to implement the steps in any of the above-described data report generation method embodiments.

[0119] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 6 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0120] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0121] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0122] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0123] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0124] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0125] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0126] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data report generation method characterized by, The generation method comprises: According to the received report generation request information, the data source of the data to be queried in the request information is obtained, and it is detected whether the database corresponding to the data source is a preset source database; When it is detected that the database corresponding to the data source is the preset source database, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage; According to the data amount of the data to be queried in the request information, the query mode in the preset engine is matched as a target query mode, the data to be queried is exported from the service node corresponding to the preset engine through the target query mode, and a report corresponding to the data to be queried is generated; According to the data amount of the data to be queried in the request information, the query mode in the preset engine is matched as a target query mode, the data to be queried is exported from the service node corresponding to the preset engine through the target query mode, and a report corresponding to the data to be queried is generated; When the data amount of the data to be queried is less than a preset threshold, the data to be queried is exported by using the paging query mode in the preset engine, and a report corresponding to the data to be queried is generated; When the data amount of the data to be queried is greater than a preset threshold, the data to be queried is exported by using the cursor query mode in the preset engine, and a report corresponding to the data to be queried is generated.

2. The data report generation method of claim 1, wherein, After the detection of whether the database corresponding to the data source is the preset source database according to the received report generation request information, the method further comprises: When it is detected that the database corresponding to the data source is not the preset source database, the data to be queried is synchronized to a local database; Through a scheduling platform, the database corresponding to the data to be queried is synchronized from the local database to the preset source database for storage.

3. The data report generation method of claim 1, wherein, When it is detected that the database corresponding to the data source is the preset source database, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage, which comprises: A mapping table of the preset source database and the service node corresponding to the preset engine is created; According to the mapping table, the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage.

4. The data report generation method of claim 1, wherein, After the data to be queried is synchronized from the preset source database to the service node corresponding to the preset engine for storage when it is detected that the database corresponding to the data source is the preset source database, the method further comprises: It is detected in the service node corresponding to the preset engine whether the synchronized data to be queried is abnormal; When the synchronized data to be queried is abnormal, according to a preset instruction, the abnormal data in the data to be queried is deleted, and the normal data to be queried is stored in the service node corresponding to the preset engine.

5. The data report generation method of claim 1, wherein, The data report generation method comprises the following steps: According to the received page query command of the data to be queried, the starting row number of the page query and the data amount of each page in the page query are obtained, and the query result of the page query is obtained; According to the query result of the page query, the data to be queried is exported in sequence, and the report corresponding to the data to be queried is generated.

6. The data report generation method of claim 1, wherein, When the data amount of the data to be queried is greater than the preset threshold, the data to be queried is exported by using the cursor query mode in the preset engine, and the report corresponding to the data to be queried is generated, comprising: According to the preset export threshold, the data to be queried is written into the preset file in the data amount of the export threshold each time by using the cursor query mode in the preset engine, and the written file is obtained; The written file is uploaded to the file container platform, the report link corresponding to the data to be queried is generated, and the report corresponding to the data to be queried is generated based on the download result of the report link.

7. A data report generation apparatus characterized by comprising: The device comprises: The detection module is configured to obtain the data source of the data to be queried according to the received request information of the report generation, and detect whether the database corresponding to the data source is a preset source database; The synchronization module is configured to synchronize the data to be queried from the preset source database to the service node corresponding to the preset engine for storage when it is detected that the database corresponding to the data source is the preset source database; The report generation module is configured to match the query mode in the preset engine as a target query mode according to the data amount of the data to be queried in the request information, export the data to be queried from the service node corresponding to the preset engine by using the target query mode, and generate the report corresponding to the data to be queried; The report generation module is configured to match the query mode in the preset engine as a target query mode according to the data amount of the data to be queried in the request information, export the data to be queried from the service node corresponding to the preset engine by using the target query mode, and generate the report corresponding to the data to be queried; When the data amount of the data to be queried is less than a preset threshold, the data to be queried is exported by using the page query mode in the preset engine, and the report corresponding to the data to be queried is generated; When the data amount of the data to be queried is greater than the preset threshold, the data to be queried is exported by using the cursor query mode in the preset engine, and the report corresponding to the data to be queried is generated.

8. A computer device, comprising: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the data report generation method in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the data report generation method in any one of claims 1 to 6.

Images